It doesn't work properly with Partitioned tables instead of Ingestion-time partitioned tables

[majidshokrolahi]

It doesn't work properly with Partitioned tables!
We have deleted a table and we recreated with the same name but as Partitioned tables which is different from Ingestion-time partitioned tables, but we don't see any record after that for that table. In log we see it search for "timepartitioning.field" which doesn't exist as it has been mentioned in documentation:

Partitioned tables do not need a _PARTITIONTIME pseudo column.
and we see in log the following error:
4: { logMessage: "Error during streaming metadata to BigQuery: [{"index": 0, "errors": [{"debugInfo": "", "reason": "invalid", "message": "no such field.", "location": "timepartitioning.field"}]}]" severity: "ERROR" sourceLocation: { file: "/base/data/home/apps/e~w3-dp-prod-monitor/v1.408280075425225715/gcp_census/bigquery/bigquery_client.py" functionName: "stream_stats" line: "153" } time: "2018-03-26T04:00:22.682840Z" }

[marcin-kolda]

Hi @magicomajid,

It looks like new partitioned tables are not supported. Could you please provide JSON response example from tables.get, it will be easier to reproduce the issue on our end. Feel free to redact project id and any other company specific names.

Thanks,
Marcin

[majidshokrolahi]

Hi @marcin-kolda
Here is the JSON:

{ "kind": "bigquery#table", "etag": "\"OhENgf-some-e-tag\"", "id": "my-project-id:MY_DATA_SET.MY_TABLE", "selfLink": "https://www.googleapis.com/bigquery/v2/projects/my-project-id/datasets/MY_DATA_SET/tables/MY_TABLE", "tableReference": { "projectId": "my-project-id", "datasetId": "MY_DATA_SET", "tableId": "MY_TABLE" }, "schema": { "fields": [ { "name": "RATING_TYPE", "type": "STRING", "mode": "NULLABLE" }, { "name": "LOCATION_IND", "type": "STRING", "mode": "NULLABLE" }, { "name": "PLMN_CODE", "type": "INTEGER", "mode": "NULLABLE" }, { "name": "VLR_ADDRESS", "type": "STRING", "mode": "NULLABLE" }, { "name": "TRAFFIC_CASE", "type": "INTEGER", "mode": "NULLABLE" }, { "name": "SOMETHING", "type": "STRING", "mode": "NULLABLE" }, { "name": "SWITCH", "type": "STRING", "mode": "NULLABLE" }, { "name": "CRM", "type": "STRING", "mode": "NULLABLE" }, { "name": "CONTRACT", "type": "STRING", "mode": "NULLABLE" }, { "name": "FIRST_CELL_CRYPT", "type": "STRING", "mode": "NULLABLE" }, { "name": "DESTINATION_CLEAN_CRYPT", "type": "STRING", "mode": "NULLABLE" }, { "name": "DATE_TO_PARTITION", "type": "DATE", "mode": "NULLABLE" } ] }, "timePartitioning": { "type": "DAY", "field": "DATE_TO_PARTITION" }, "numBytes": "11638002273766", "numLongTermBytes": "0", "numRows": "10218089354", "creationTime": "1520984873993", "lastModifiedTime": "1522076000565", "type": "TABLE", "location": "EU" }

[marcin-kolda]

Hi @magicomajid,

Thanks for sample.

I have prepared PR #32 with fix for this issue. While it's in the review, you might want to try code from partitioned_table_fix branch. I created also #33 for better support of the new field.

[majidshokrolahi]

@marcin-kolda Hi Marcin, thank you for your very fast answer and fix.
I have done new deploy and it seems working just fine :)
Great job 👍