Fixed RD-15322: Report user-visible errors to caller instead of retrying

[bgaidioz]

This goes with raw-labs/multicorn-das#15.

In this changeset, we use StatusCode INVALID_ARGUMENT to flag an error that should be advertised to the user. If a call fails with that error, in multicorn we propagate the exception. That patch goes with a symmetric patch in multicorn.

• DAS not found is returned as NOT_FOUND.
• Table not found is returned to the user because that would be the Postgres schema contains a table the DAS doesn't contain, which would happen if restarting a DAS leads to a new schema.
• Calls to the backend are occurring after the DAS and the table are found. They're wrapped in a try/catch that eventually returns an INVALID_ARGUMENT error. This fails if an INSERT, UPDATE is given a forbidden value (e.g. a string that is too large), if the operation isn't supported by the backend. Also SELECT can fail as certain backends wouldn't offer read access to a table.

These code paths were tested using:

• a DASMock table that doesn't support CRUD,
• a custom mock DAS that exposes a table called events that does support all operations but fails if provided dates (qualifiers, values) are before year 2000,
• a custom mock DAS which exposes a table that isn't part of its catalog eventually.

Then all these queries are run with a DAS server up, or down then restarted during the retries.

Execute

SELECT * FROM test.events WHERE date > DATE '1922-03-02';

retries if needed and fails because the filter is rejected.

ERROR:  Error in python: _MultiThreadedRendezvous
DETAIL:  <_MultiThreadedRendezvous of RPC that terminated with:
        status = StatusCode.INVALID_ARGUMENT
        details = "Invalid date: year: 1922
month: 3
day: 2
 (only > 2000 supported)"
        debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2025-02-03T14:38:41.660197+01:00", grpc_status:3, grpc_message:"Invalid date: year: 1922\nmonth: 3\nday: 2\n (only > 2000 supported)"}"
>

SELECT * FROM test.events WHERE date > DATE '2022-03-02';

retries if needed and runs successfully.

WARNING:  Error in table_service.UniqueColumns for table name: "events"
: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.NOT_FOUND
        details = "DAS not found"
        debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2025-02-03T14:55:19.312844+01:00", grpc_status:5, grpc_message:"DAS not found"}"
>
 event  |    date    
--------+------------
 event3 | 2023-01-01
 event4 | 2024-01-01

SELECT * FROM test.vanished ;

retries if needed and fails because the table isn't found. (This can happen if restarting a DAS and it lost a table.)

ERROR:  Error in python: _InactiveRpcError
DETAIL:  <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.INVALID_ARGUMENT
        details = "Table vanished not found"
        debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2025-02-03T14:58:20.208856+01:00", grpc_status:3, grpc_message:"Table vanished not found"}"
>

Insert

INSERT INTO test.small VALUES (101, 'blah');

retries if needed and fails because the table doesn't support INSERT.

ERROR:  Error in python: _InactiveRpcError
DETAIL:  <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.INVALID_ARGUMENT
        details = "unsupported operation"
        debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"unsupported operation", grpc_status:3, created_time:"2025-02-03T14:52:52.834397+01:00"}"
>

INSERT INTO test.events VALUES ('event5', DATE '1922-03-02');

retries if needed and fails because the date is wrong.

ERROR:  Error in python: _InactiveRpcError
DETAIL:  <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.INVALID_ARGUMENT
        details = "Invalid date: year: 1922
month: 3
day: 2
 (only > 2000 supported)"
        debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Invalid date: year: 1922\nmonth: 3\nday: 2\n (only > 2000 supported)", grpc_status:3, created_time:"2025-02-03T14:41:09.959135+01:00"}"
>

INSERT INTO test.events VALUES ('event5', DATE '2022-03-02');

retries if needed and runs successfully.

WARNING:  Error in table_service.UniqueColumns for table name: "events"
: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.NOT_FOUND
        details = "DAS not found"
        debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2025-02-03T14:55:19.312844+01:00", grpc_status:5, grpc_message:"DAS not found"}"
>
INSERT 0 1

Update

UPDATE test.small SET column2 = 'tralala' WHERE column1 = 98;

retries if needed and fails because the table doesn't support UPDATE.

ERROR:  Error in python: _InactiveRpcError
DETAIL:  <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.INVALID_ARGUMENT
        details = "unsupported operation"
        debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"unsupported operation", grpc_status:3, created_time:"2025-02-03T14:53:18.871765+01:00"}"
>

UPDATE test.events SET date = DATE '1928-03-03' WHERE event = 'event1';

retries if needed and fails because the date is wrong.

ERROR:  Error in python: _InactiveRpcError
DETAIL:  <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.INVALID_ARGUMENT
        details = "Invalid date: year: 1928
month: 3
day: 3
 (only > 2000 supported)"
        debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Invalid date: year: 1928\nmonth: 3\nday: 3\n (only > 2000 supported)", grpc_status:3, created_time:"2025-02-03T14:50:38.411487+01:00"}"
>

UPDATE test.events SET date = DATE '2028-03-03' WHERE event = 'event1';

retries if needed and runs successfully.

WARNING:  Error in table_service.UniqueColumns for table name: "events"
: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.NOT_FOUND
        details = "DAS not found"
        debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2025-02-03T14:55:19.312844+01:00", grpc_status:5, grpc_message:"DAS not found"}"
>
UPDATE 1

Delete

DELETE FROM test.small WHERE column1 = 98;

retries if needed and fails because the table doesn't support DELETE.

ERROR:  Error in python: _InactiveRpcError
DETAIL:  <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.INVALID_ARGUMENT
        details = "unsupported operation"
        debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"unsupported operation", grpc_status:3, created_time:"2025-02-03T14:54:11.536554+01:00"}"
>

DELETE FROM test.events WHERE date = DATE '1921-01-01';

retries if needed and fails because the date is wrong.

ERROR:  Error in python: _MultiThreadedRendezvous
DETAIL:  <_MultiThreadedRendezvous of RPC that terminated with:
        status = StatusCode.INVALID_ARGUMENT
        details = "Invalid date: year: 1921
month: 1
day: 1
 (only > 2000 supported)"
        debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Invalid date: year: 1921\nmonth: 1\nday: 1\n (only > 2000 supported)", grpc_status:3, created_time:"2025-02-03T14:54:34.129372+01:00"}"
>

DELETE FROM test.events WHERE date = DATE '2021-01-01';

retries if needed and runs successfully.

WARNING:  Error in table_service.UniqueColumns for table name: "events"
: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.NOT_FOUND
        details = "DAS not found"
        debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2025-02-03T14:55:19.312844+01:00", grpc_status:5, grpc_message:"DAS not found"}"
>
DELETE 1