Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(go/adbc/driver/snowflake): Add support for table constraints when calling GetObjects #1455

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
+163 −89
Draft

feat(go/adbc/driver/snowflake): Add support for table constraints when calling GetObjects #1455

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2afbbb7
Remove ineffectual assignment to xdbcType
vleslief-ms Jan 11, 2024
fd3df0e
Merge branch 'main' of https://github.com/apache/arrow-adbc into snow…
vleslief-ms Feb 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions .pre-commit-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@

repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v2.3.0
rev: v4.5.0
hooks:
- id: check-xml
- id: check-yaml
Expand Down Expand Up @@ -60,30 +60,30 @@ repos:
- "--linelength=90"
- "--verbose=2"
- repo: https://github.com/golangci/golangci-lint
rev: v1.49.0
rev: v1.55.2
hooks:
- id: golangci-lint
entry: bash -c 'cd go/adbc && golangci-lint run --fix --timeout 5m'
types_or: [go]
- repo: https://github.com/macisamuele/language-formatters-pre-commit-hooks
rev: v2.3.0
rev: v2.12.0
hooks:
- id: pretty-format-golang
- id: pretty-format-java
args: [--autofix]
types_or: [java]
- repo: https://github.com/psf/black
rev: 22.3.0
rev: 23.12.1
hooks:
- id: black
types_or: [pyi, python]
- repo: https://github.com/PyCQA/flake8
rev: 4.0.1
rev: 7.0.0
hooks:
- id: flake8
types_or: [python]
- repo: https://github.com/PyCQA/isort
rev: 5.12.0
rev: 5.13.2
hooks:
- id: isort
types_or: [python]
Expand Down
1 change: 1 addition & 0 deletions go/adbc/driver/internal/shared_utils.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ type Metadata struct {
NumericPrec, NumericPrecRadix, NumericScale, DatetimePrec sql.NullInt16
IsNullable, IsIdent bool
CharMaxLength, CharOctetLength sql.NullInt32
ConstraintName, ConstraintType sql.NullString
}

type GetObjDBSchemasFn func(ctx context.Context, depth adbc.ObjectDepth, catalog *string, schema *string, metadataRecords []Metadata) (map[string][]string, error)
Expand Down
28 changes: 21 additions & 7 deletions go/adbc/driver/snowflake/connection.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -408,7 +408,6 @@ func toField(name string, isnullable bool, dataType string, numPrec, numPrecRadi
}

func toXdbcDataType(dt arrow.DataType) (xdbcType internal.XdbcDataType) {
xdbcType = internal.XdbcDataType_XDBC_UNKNOWN_TYPE
switch dt.ID() {
case arrow.EXTENSION:
return toXdbcDataType(dt.(arrow.ExtensionType).StorageType())
Expand Down Expand Up @@ -640,7 +639,8 @@ func (c *cnxn) getColumnsMetadata(ctx context.Context, matchingCatalogNames []st
err = rows.Scan(&data.TblType, &data.Dbname, &data.Schema, &data.TblName, &data.ColName,
&data.OrdinalPos, &data.IsNullable, &data.DataType, &data.NumericPrec,
&data.NumericPrecRadix, &data.NumericScale, &data.IsIdent, &data.IdentGen,
&data.IdentIncrement, &data.CharMaxLength, &data.CharOctetLength, &data.DatetimePrec, &data.Comment)
&data.IdentIncrement, &data.CharMaxLength, &data.CharOctetLength, &data.DatetimePrec, &data.Comment,
&data.ConstraintName, &data.ConstraintType)
Copy link
Contributor

@ryan-syed ryan-syed Jan 12, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TableInfo struct doesn't currently capture table constraints.

I couldn't locate them being appended to tableConstraintsBuilder which seems to be marked as unimplemented.

if err != nil {
return nil, errToAdbcErr(adbc.StatusIO, err)
}
Expand Down Expand Up @@ -701,10 +701,16 @@ func prepareTablesSQL(matchingCatalogNames []string, catalog *string, dbSchema *
if query != "" {
query += " UNION ALL "
}
query += `SELECT * FROM "` + strings.ReplaceAll(catalog_name, "\"", "\"\"") + `".INFORMATION_SCHEMA.TABLES`
query += `SELECT T.*, K.constraint_name, K.constraint_type FROM "` + strings.ReplaceAll(catalog_name, "\"", "\"\"") + `".INFORMATION_SCHEMA.TABLES AS T
Copy link
Contributor

@ryan-syed ryan-syed Jan 12, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As per the schema, constraint_column_names and, constraint_column_usage need to be included, however, I couldn't locate them in the queries.
[
constraint_column_names | list not null | (2)
constraint_column_usage | list<USAGE_SCHEMA> | (3)
](https://github.com/apache/arrow-adbc/blob/2afbbb7d4d471d21fe20eed976bf4c0b20151732/go/adbc/driver/snowflake/connection.go#L210C1-L217C56)

LEFT OUTER JOIN
"` + strings.ReplaceAll(catalog_name, "\"", "\"\"") + `".INFORMATION_SCHEMA.TABLE_CONSTRAINTS AS K
ON
T.table_catalog = K.table_catalog
AND T.table_schema = K.table_schema
AND t.table_name = K.table_name`
}

query = `SELECT table_catalog, table_schema, table_name, table_type FROM (` + query + `)`
query = `SELECT table_catalog, table_schema, table_name, table_type, constraint_name, constraint_type FROM (` + query + `)`
conditions, queryArgs := prepareFilterConditions(adbc.ObjectDepthTables, catalog, dbSchema, tableName, nil, tableType)
if conditions != "" {
query += " WHERE " + conditions
Expand All @@ -719,22 +725,30 @@ func prepareColumnsSQL(matchingCatalogNames []string, catalog *string, dbSchema
prefixQuery += " UNION ALL "
}
prefixQuery += `SELECT T.table_type,
C.*
C.*,
K.constraint_name, K.constraint_type
FROM
"` + strings.ReplaceAll(catalog_name, "\"", "\"\"") + `".INFORMATION_SCHEMA.TABLES AS T
JOIN
"` + strings.ReplaceAll(catalog_name, "\"", "\"\"") + `".INFORMATION_SCHEMA.COLUMNS AS C
ON
T.table_catalog = C.table_catalog
AND T.table_schema = C.table_schema
AND t.table_name = C.table_name`
AND T.table_name = C.table_name
LEFT OUTER JOIN
"` + strings.ReplaceAll(catalog_name, "\"", "\"\"") + `".INFORMATION_SCHEMA.TABLE_CONSTRAINTS AS K
ON
T.table_catalog = K.table_catalog
AND T.table_schema = K.table_schema
AND T.table_name = K.table_name`
}

prefixQuery = `SELECT table_type, table_catalog, table_schema, table_name, column_name,
ordinal_position, is_nullable::boolean, data_type, numeric_precision,
numeric_precision_radix, numeric_scale, is_identity::boolean,
identity_generation, identity_increment,
character_maximum_length, character_octet_length, datetime_precision, comment FROM (` + prefixQuery + `)`
character_maximum_length, character_octet_length, datetime_precision, comment,
constraint_name, constraint_type FROM (` + prefixQuery + `)`
ordering := ` ORDER BY table_catalog, table_schema, table_name, ordinal_position`
conditions, queryArgs := prepareFilterConditions(adbc.ObjectDepthColumns, catalog, dbSchema, tableName, columnName, tableType)
query := prefixQuery
Expand Down
211 changes: 135 additions & 76 deletions go/adbc/driver/snowflake/connection_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,15 +158,25 @@ func TestPrepareTablesSQLWithNoFilter(t *testing.T) {
tableNamePattern := ""
tableType := make([]string, 0)

expected := `SELECT table_catalog, table_schema, table_name, table_type
FROM
(
SELECT * FROM "DEMO_DB".INFORMATION_SCHEMA.TABLES
UNION ALL
SELECT * FROM "DEMOADB".INFORMATION_SCHEMA.TABLES
UNION ALL
SELECT * FROM "DEMO'DB".INFORMATION_SCHEMA.TABLES
)`
expected := `SELECT table_catalog, table_schema, table_name, table_type, constraint_name, constraint_type FROM (SELECT T.*, K.constraint_name, K.constraint_type FROM "DEMO_DB".INFORMATION_SCHEMA.TABLES AS T
LEFT OUTER JOIN
"DEMO_DB".INFORMATION_SCHEMA.TABLE_CONSTRAINTS AS K
ON
T.table_catalog = K.table_catalog
AND T.table_schema = K.table_schema
AND t.table_name = K.table_name UNION ALL SELECT T.*, K.constraint_name, K.constraint_type FROM "DEMOADB".INFORMATION_SCHEMA.TABLES AS T
LEFT OUTER JOIN
"DEMOADB".INFORMATION_SCHEMA.TABLE_CONSTRAINTS AS K
ON
T.table_catalog = K.table_catalog
AND T.table_schema = K.table_schema
AND t.table_name = K.table_name UNION ALL SELECT T.*, K.constraint_name, K.constraint_type FROM "DEMO'DB".INFORMATION_SCHEMA.TABLES AS T
LEFT OUTER JOIN
"DEMO'DB".INFORMATION_SCHEMA.TABLE_CONSTRAINTS AS K
ON
T.table_catalog = K.table_catalog
AND T.table_schema = K.table_schema
AND t.table_name = K.table_name)`
actual, queryArgs := prepareTablesSQL(catalogNames[:], &catalogPattern, &schemaPattern, &tableNamePattern, tableType[:])

println("Query Args", queryArgs)
Expand All @@ -180,16 +190,25 @@ func TestPrepareTablesSQLWithNoTableTypeFilter(t *testing.T) {
tableNamePattern := "ADBC-TABLE"
tableType := make([]string, 0)

expected := `SELECT table_catalog, table_schema, table_name, table_type
FROM
(
SELECT * FROM "DEMO_DB".INFORMATION_SCHEMA.TABLES
UNION ALL
SELECT * FROM "DEMOADB".INFORMATION_SCHEMA.TABLES
UNION ALL
SELECT * FROM "DEMO'DB".INFORMATION_SCHEMA.TABLES
)
WHERE TABLE_CATALOG ILIKE ? AND TABLE_SCHEMA ILIKE ? AND TABLE_NAME ILIKE ? `
expected := `SELECT table_catalog, table_schema, table_name, table_type, constraint_name, constraint_type FROM (SELECT T.*, K.constraint_name, K.constraint_type FROM "DEMO_DB".INFORMATION_SCHEMA.TABLES AS T
LEFT OUTER JOIN
"DEMO_DB".INFORMATION_SCHEMA.TABLE_CONSTRAINTS AS K
ON
T.table_catalog = K.table_catalog
AND T.table_schema = K.table_schema
AND t.table_name = K.table_name UNION ALL SELECT T.*, K.constraint_name, K.constraint_type FROM "DEMOADB".INFORMATION_SCHEMA.TABLES AS T
LEFT OUTER JOIN
"DEMOADB".INFORMATION_SCHEMA.TABLE_CONSTRAINTS AS K
ON
T.table_catalog = K.table_catalog
AND T.table_schema = K.table_schema
AND t.table_name = K.table_name UNION ALL SELECT T.*, K.constraint_name, K.constraint_type FROM "DEMO'DB".INFORMATION_SCHEMA.TABLES AS T
LEFT OUTER JOIN
"DEMO'DB".INFORMATION_SCHEMA.TABLE_CONSTRAINTS AS K
ON
T.table_catalog = K.table_catalog
AND T.table_schema = K.table_schema
AND t.table_name = K.table_name) WHERE TABLE_CATALOG ILIKE ? AND TABLE_SCHEMA ILIKE ? AND TABLE_NAME ILIKE ? `
actual, queryArgs := prepareTablesSQL(catalogNames[:], &catalogPattern, &schemaPattern, &tableNamePattern, tableType[:])

stringqueryArgs := make([]string, len(queryArgs)) // Pre-allocate the right size
Expand All @@ -208,16 +227,26 @@ func TestPrepareTablesSQL(t *testing.T) {
tableNamePattern := "ADBC-TABLE"
tableType := [2]string{"BASE TABLE", "VIEW"}

expected := `SELECT table_catalog, table_schema, table_name, table_type
FROM
(
SELECT * FROM "DEMO_DB".INFORMATION_SCHEMA.TABLES
UNION ALL
SELECT * FROM "DEMOADB".INFORMATION_SCHEMA.TABLES
UNION ALL
SELECT * FROM "DEMO'DB".INFORMATION_SCHEMA.TABLES
)
WHERE TABLE_CATALOG ILIKE ? AND TABLE_SCHEMA ILIKE ? AND TABLE_NAME ILIKE ? AND TABLE_TYPE IN ('BASE TABLE','VIEW')`
expected := `SELECT table_catalog, table_schema, table_name, table_type, constraint_name, constraint_type FROM (SELECT T.*, K.constraint_name, K.constraint_type FROM "DEMO_DB".INFORMATION_SCHEMA.TABLES AS T
LEFT OUTER JOIN
"DEMO_DB".INFORMATION_SCHEMA.TABLE_CONSTRAINTS AS K
ON
T.table_catalog = K.table_catalog
AND T.table_schema = K.table_schema
AND t.table_name = K.table_name UNION ALL SELECT T.*, K.constraint_name, K.constraint_type FROM "DEMOADB".INFORMATION_SCHEMA.TABLES AS T
LEFT OUTER JOIN
"DEMOADB".INFORMATION_SCHEMA.TABLE_CONSTRAINTS AS K
ON
T.table_catalog = K.table_catalog
AND T.table_schema = K.table_schema
AND t.table_name = K.table_name UNION ALL SELECT T.*, K.constraint_name, K.constraint_type FROM "DEMO'DB".INFORMATION_SCHEMA.TABLES AS T
LEFT OUTER JOIN
"DEMO'DB".INFORMATION_SCHEMA.TABLE_CONSTRAINTS AS K
ON
T.table_catalog = K.table_catalog
AND T.table_schema = K.table_schema
AND t.table_name = K.table_name)
WHERE TABLE_CATALOG ILIKE ? AND TABLE_SCHEMA ILIKE ? AND TABLE_NAME ILIKE ? AND TABLE_TYPE IN ('BASE TABLE','VIEW')`
actual, queryArgs := prepareTablesSQL(catalogNames[:], &catalogPattern, &schemaPattern, &tableNamePattern, tableType[:])

stringqueryArgs := make([]string, len(queryArgs)) // Pre-allocate the right size
Expand All @@ -238,29 +267,44 @@ func TestPrepareColumnsSQLNoFilter(t *testing.T) {
tableType := make([]string, 0)

expected := `SELECT table_type, table_catalog, table_schema, table_name, column_name,
ordinal_position, is_nullable::boolean, data_type, numeric_precision,
numeric_precision_radix, numeric_scale, is_identity::boolean,
identity_generation, identity_increment,
character_maximum_length, character_octet_length, datetime_precision, comment
FROM
(
SELECT T.table_type, C.*
FROM
"DEMO_DB".INFORMATION_SCHEMA.TABLES AS T
JOIN
"DEMO_DB".INFORMATION_SCHEMA.COLUMNS AS C
ON
T.table_catalog = C.table_catalog AND T.table_schema = C.table_schema AND t.table_name = C.table_name
UNION ALL
SELECT T.table_type, C.*
FROM
"DEMOADB".INFORMATION_SCHEMA.TABLES AS T
JOIN
"DEMOADB".INFORMATION_SCHEMA.COLUMNS AS C
ON
T.table_catalog = C.table_catalog AND T.table_schema = C.table_schema AND t.table_name = C.table_name
)
ORDER BY table_catalog, table_schema, table_name, ordinal_position`
ordinal_position, is_nullable::boolean, data_type, numeric_precision,
numeric_precision_radix, numeric_scale, is_identity::boolean,
identity_generation, identity_increment,
character_maximum_length, character_octet_length, datetime_precision, comment,
constraint_name, constraint_type FROM (SELECT T.table_type,
C.*,
K.constraint_name, K.constraint_type
FROM
"DEMO_DB".INFORMATION_SCHEMA.TABLES AS T
JOIN
"DEMO_DB".INFORMATION_SCHEMA.COLUMNS AS C
ON
T.table_catalog = C.table_catalog
AND T.table_schema = C.table_schema
AND T.table_name = C.table_name
LEFT OUTER JOIN
"DEMO_DB".INFORMATION_SCHEMA.TABLE_CONSTRAINTS AS K
ON
T.table_catalog = K.table_catalog
AND T.table_schema = K.table_schema
AND T.table_name = K.table_name UNION ALL SELECT T.table_type,
C.*,
K.constraint_name, K.constraint_type
FROM
"DEMOADB".INFORMATION_SCHEMA.TABLES AS T
JOIN
"DEMOADB".INFORMATION_SCHEMA.COLUMNS AS C
ON
T.table_catalog = C.table_catalog
AND T.table_schema = C.table_schema
AND T.table_name = C.table_name
LEFT OUTER JOIN
"DEMOADB".INFORMATION_SCHEMA.TABLE_CONSTRAINTS AS K
ON
T.table_catalog = K.table_catalog
AND T.table_schema = K.table_schema
AND T.table_name = K.table_name)
ORDER BY table_catalog, table_schema, table_name, ordinal_position`
actual, queryArgs := prepareColumnsSQL(catalogNames[:], &catalogPattern, &schemaPattern, &tableNamePattern, &columnNamePattern, tableType[:])

println("Query Args", queryArgs)
Expand All @@ -276,30 +320,45 @@ func TestPrepareColumnsSQL(t *testing.T) {
tableType := [2]string{"BASE TABLE", "VIEW"}

expected := `SELECT table_type, table_catalog, table_schema, table_name, column_name,
ordinal_position, is_nullable::boolean, data_type, numeric_precision,
numeric_precision_radix, numeric_scale, is_identity::boolean,
identity_generation, identity_increment,
character_maximum_length, character_octet_length, datetime_precision, comment
FROM
(
SELECT T.table_type, C.*
FROM
"DEMO_DB".INFORMATION_SCHEMA.TABLES AS T
JOIN
"DEMO_DB".INFORMATION_SCHEMA.COLUMNS AS C
ON
T.table_catalog = C.table_catalog AND T.table_schema = C.table_schema AND t.table_name = C.table_name
UNION ALL
SELECT T.table_type, C.*
FROM
"DEMOADB".INFORMATION_SCHEMA.TABLES AS T
JOIN
"DEMOADB".INFORMATION_SCHEMA.COLUMNS AS C
ON
T.table_catalog = C.table_catalog AND T.table_schema = C.table_schema AND t.table_name = C.table_name
)
WHERE TABLE_CATALOG ILIKE ? AND TABLE_SCHEMA ILIKE ? AND TABLE_NAME ILIKE ? AND COLUMN_NAME ILIKE ? AND TABLE_TYPE IN ('BASE TABLE','VIEW')
ORDER BY table_catalog, table_schema, table_name, ordinal_position`
ordinal_position, is_nullable::boolean, data_type, numeric_precision,
numeric_precision_radix, numeric_scale, is_identity::boolean,
identity_generation, identity_increment,
character_maximum_length, character_octet_length, datetime_precision, comment,
constraint_name, constraint_type FROM (SELECT T.table_type,
C.*,
K.constraint_name, K.constraint_type
FROM
"DEMO_DB".INFORMATION_SCHEMA.TABLES AS T
JOIN
"DEMO_DB".INFORMATION_SCHEMA.COLUMNS AS C
ON
T.table_catalog = C.table_catalog
AND T.table_schema = C.table_schema
AND T.table_name = C.table_name
LEFT OUTER JOIN
"DEMO_DB".INFORMATION_SCHEMA.TABLE_CONSTRAINTS AS K
ON
T.table_catalog = K.table_catalog
AND T.table_schema = K.table_schema
AND T.table_name = K.table_name UNION ALL SELECT T.table_type,
C.*,
K.constraint_name, K.constraint_type
Copy link
Contributor

@ryan-syed ryan-syed Jan 12, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding a test similar to TestMetadataGetObjectsColumnsXdbc will be helpful to verify the table constraints are returned correctly. Adding tests in the CSharp layer would also work.

FROM
"DEMOADB".INFORMATION_SCHEMA.TABLES AS T
JOIN
"DEMOADB".INFORMATION_SCHEMA.COLUMNS AS C
ON
T.table_catalog = C.table_catalog
AND T.table_schema = C.table_schema
AND T.table_name = C.table_name
LEFT OUTER JOIN
"DEMOADB".INFORMATION_SCHEMA.TABLE_CONSTRAINTS AS K
ON
T.table_catalog = K.table_catalog
AND T.table_schema = K.table_schema
AND T.table_name = K.table_name)
WHERE TABLE_CATALOG ILIKE ? AND TABLE_SCHEMA ILIKE ? AND TABLE_NAME ILIKE ? AND COLUMN_NAME ILIKE ? AND TABLE_TYPE IN ('BASE TABLE','VIEW')
ORDER BY table_catalog, table_schema, table_name, ordinal_position`
actual, queryArgs := prepareColumnsSQL(catalogNames[:], &catalogPattern, &schemaPattern, &tableNamePattern, &columnNamePattern, tableType[:])

stringqueryArgs := make([]string, len(queryArgs)) // Pre-allocate the right size
Expand Down
Loading