Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Neptune Analytics Support & Logging #23

Closed
Closed
Show file tree
Hide file tree
Changes from 42 commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
3395a29
Update getEdgesDirections queries
Cole-Greer Mar 28, 2024
6f1afeb
Neptune Analytics - Logger
AndreaNassisi May 21, 2024
c102a84
Merge remote-tracking branch 'origin/updateGetEdgesDirections' into N…
AndreaNassisi May 21, 2024
5e54131
Update getEdgesDirections queries
Cole-Greer Mar 28, 2024
83307b6
cherry pick 'Neptune Analytics - Logger' from #19
AndreaNassisi May 21, 2024
9ac1bfa
cherry pick 'Merge remote-tracking branch...' from #19
sophiadt Sep 17, 2024
936614f
Test milestone
AndreaNassisi Jul 10, 2024
ebb2875
Test package
AndreaNassisi Jul 10, 2024
615569c
change consoleOut.error to console.error
sophiadt Sep 18, 2024
b95c4d3
Skipped retrieving neptune cluster info for analytics.
andreachild Sep 18, 2024
4254af8
added in AndreaC's fixes to todo schema, commas and incorrect orderin…
sophiadt Sep 19, 2024
90cffc8
reverted endpoint config in unit tests
sophiadt Sep 19, 2024
8573166
Revert "Update getEdgesDirections queries"
sophiadt Sep 19, 2024
b5e80c2
fixed error message typo
sophiadt Sep 19, 2024
bbbd7b6
Modified logger method to accept flag which dictates if given text sh…
andreachild Sep 19, 2024
71791fa
Reverted a change which would have too verbose output to console.
andreachild Sep 19, 2024
d59383d
change if statement to check if endpoint has exactly 2 parts
sophiadt Sep 20, 2024
63ee6a4
Merge branch 'sophiadt/Neptune-Analytics---Logger' into achild/Neptun…
sophiadt Sep 20, 2024
6c27318
Merge pull request #2 from Bit-Quill/achild/Neptune-Analytics---Logger
sophiadt Sep 20, 2024
a5dbb3a
Removed default region and changed parsing of region from URL to diff…
andreachild Sep 20, 2024
8623a70
removed code for dual lamba option and read-only endpoints
sophiadt Sep 23, 2024
12691dd
added host and port placeholders
sophiadt Sep 24, 2024
d60e36b
Changed queryNeptune and queryNeptuneSdk to take optional parameters …
andreachild Sep 26, 2024
f1326f5
Merge branch 'main' into sophiadt/Neptune-Analytics---Logger
sophiadt Sep 26, 2024
f4b7335
use parameterized query for getEdgeProperties
sophiadt Sep 27, 2024
c162b62
use parameterized queries for the rest of the functions and added typ…
sophiadt Sep 27, 2024
16c431a
merge 'Update getEdgesDirections queries' from #16
Cole-Greer Mar 28, 2024
77794f0
add sanitize placeholder function and reverted queries to use labels
sophiadt Oct 3, 2024
dbecfb8
remove params in function call for queries that aren't parameterized
sophiadt Oct 3, 2024
dc987c0
Replaced file and console logging with the pino logger library for ea…
andreachild Oct 4, 2024
8f63bb9
Remove duplicated sanitize function.
andreachild Oct 4, 2024
88385cb
Integrate usage of neptune graph SDK for schema queries and lambda lo…
andreachild Oct 8, 2024
a6e89ac
Fixed pino dependencies and updated tarball with latest.
andreachild Oct 9, 2024
121a8ae
Fixed packaging to include the new analytics sdk template.
andreachild Oct 9, 2024
ee28939
Fixed lambda environment variables to include neptune type which is r…
andreachild Oct 9, 2024
70293fc
fixed typo in README.md
sophiadt Oct 9, 2024
38ad71c
fixed typos
sophiadt Oct 9, 2024
477d819
Refactor console output and introduce file log levels (#6)
andreachild Oct 11, 2024
98f4a05
fixed typos in README.md
sophiadt Oct 11, 2024
20bd7e7
fixed typos and added some more details in README.md
sophiadt Oct 11, 2024
cda89f1
Merge branch 'main' into sophiadt/Neptune-Analytics---Logger
andreachild Oct 11, 2024
563c2da
Merge branch 'main' into sophiadt/Neptune-Analytics---Logger
andreachild Oct 11, 2024
4c06de9
Merge branch 'main' into sophiadt/Neptune-Analytics---Logger
andreachild Oct 15, 2024
79069b3
Merge branch 'main' into sophiadt/Neptune-Analytics---Logger
andreachild Oct 15, 2024
d737463
Cleanup of changes that shouldn't have been included when merging fro…
andreachild Oct 16, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ package-lock.json
node_modules/**
templates/Lambda4AppSyncHTTP/node_modules/**
templates/Lambda4AppSyncSDK/node_modules/**
templates/Lambda4AppSyncGraphSDK/node_modules/**
coverage/**
test/node_modules/**
test/TestCases/Case01/output/**
Expand All @@ -13,4 +14,5 @@ test/TestCases/Case03/output/**
test/TestCases/Case04/output/**
test/TestCases/Case05/output/**
test/TestCases/Case06/output/**
test/TestCases/Case07/output/**
*.iml
16 changes: 8 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ To rollback, removing all the AWS resources run:
`neptune-for-graphql --remove-aws-pipeline-name` <*your-new-GraphQL-API-name*>

#### References:
- [here](/doc/routesExample.md) an example using the Air Routes data on Amazon Neptune, showing the outputs of the utility.
- [here](/doc/routesExample.md) is an example using the Air Routes data on Amazon Neptune, showing the outputs of the utility.
- If you are wondering which AWS resources the utility is creating, look at the section below.
- To customize the GraphQL schema, look at the section below.

Expand All @@ -89,7 +89,7 @@ You can start from an empty Neptune database and use a GraphQL API to create the
`neptune-for-graphql --input-schema-file `<*your-graphql-schema-file*>` --create-update-aws-pipeline --create-update-aws-pipeline-name` <*your-new-GraphQL-API-name*>` --create-update-aws-pipeline-neptune-endpoint` <*your-neptune-database-endpoint:port*> ` --output-resolver-query-https`

#### References:
- [here](https://github.com/aws/amazon-neptune-for-graphql/blob/main/doc/todoExample.md) an example using a TODO GraphQL schema, showing the outputs of the utility.
- [here](https://github.com/aws/amazon-neptune-for-graphql/blob/main/doc/todoExample.md) is an example using a TODO GraphQL schema, showing the outputs of the utility.
- If you are wondering which AWS resources the utility is creating, look at the section below.
- To customize the GraphQL schema, look at the section below.

Expand All @@ -113,7 +113,7 @@ In case you don't want your Graph API having the option of updating your databas


### @alias
This directive can be applied to GraphQL schema types or fields. It maps different names between the graph database and the GraphQL schema. The syntax is *@alias(property: your-name)*. In the example below *airport* is the graph database node label mapped to the *Airport* GraphQL type, and *desc* is the property of the graph database node mapped to the field *description*. See the [Air Routes Example](https://github.com/aws/amazon-neptune-for-graphql/blob/main/doc/routesExample.md). The GraphQL guidence is pascal case for types and camel case for fields.
This directive can be applied to GraphQL schema types or fields. It maps different names between the graph database and the GraphQL schema. The syntax is *@alias(property: your-name)*. In the example below *airport* is the graph database node label mapped to the *Airport* GraphQL type, and *desc* is the property of the graph database node mapped to the field *description*. See the [Air Routes Example](https://github.com/aws/amazon-neptune-for-graphql/blob/main/doc/routesExample.md). The GraphQL guidance is pascal case for types and camel case for fields.

```graphql
type Airport @alias(property: "airport") {
Expand Down Expand Up @@ -160,7 +160,7 @@ type Mutation {
}
```

You can add query or mututation using a Gremlin query. At this time Gremlin queries are limited to return *scalar* values, or *elementMap()* for a single node, or *elementMap().fold()* for a list of nodes.
You can add a query or mutation using a Gremlin query. At this time Gremlin queries are limited to return *scalar* values, or *elementMap()* for a single node, or *elementMap().fold()* for a list of nodes.
```graphql
type Query {
getAirportWithGremlin(code:String): Airport @graphQuery(statement: "g.V().has('airport', 'code', '$code').elementMap()") # single node
Expand Down Expand Up @@ -215,7 +215,7 @@ type Mutation {
```

### Re-apply your changes with --input-schema-changes-file
You might want to modify the GraphQL source schema and run the utility again getting the latest schema from your Neptune database. Every time the utility discover the a new graphdb schema it generates a new GraphQL schema. To inject your changes, you can manually edit the GraphQL source schema, and run the utility again using it as input instead of the Neptune database endpoint, or write your changes the file format below. As you run the utility with the option `--input-schema-changes-file <value>`, your changes will be applied at once.
You might want to modify the GraphQL source schema and run the utility again to get the latest schema from your Neptune database. Every time the utility discovers a new graphdb schema, it generates a new GraphQL schema. To inject your changes, you can manually edit the GraphQL source schema and run the utility again, using it as input instead of the Neptune database endpoint, or write your changes in the format below. As you run the utility with the added option `--input-schema-changes-file <your-changes-file>`, your changes will be applied at once.
```json
[
{ "type": "graphQLTypeName",
Expand Down Expand Up @@ -260,15 +260,15 @@ Independently of the method you or the utility will need to create the following
With the CLI option `--create-update-aws-pipeline`, and its accesory options ([see the commands reference](https://github.com/aws/amazon-neptune-for-graphql/blob/main/doc/cliReference.md)), the utility automatically creates the resources.<br>
You need to run the utility from a shell in which you installed the AWS CLI, and it is configured for a user with the permission of creating AWS resources. <br>
The utility creates a file with the list of resources named *pipeline-name.resources.json*. Then it uses the file to modify the existing resources when the user runs the command again, or when the user wants to delete the AWS resources with the command option `--remove-aws-pipeline-name <value>`.
The code of the utiliy uses the JavaScript AWS sdk v3, if you'd like to review the code is only in [this file](https://github.com/aws/amazon-neptune-for-graphql/blob/main/src/pipelineResources.js).
The code of the utility uses the JavaScript AWS sdk v3, if you'd like to review the code is only in [this file](https://github.com/aws/amazon-neptune-for-graphql/blob/main/src/pipelineResources.js).

### I prefer a CDK file
To option to trigger the creation of the CDK file starts with `--output-aws-pipeline-cdk`, and its accessory options ([see the commands reference](https://github.com/aws/amazon-neptune-for-graphql/blob/main/doc/cliReference.md)). <br>
After you ran it, you will find in the *output* folder the file *CDK-pipeline-name-cdk.js* and *CDK-pipeline-name.zip*. The ZIP file is the code for the Lambda function.
See CDK end to end example [here](https://github.com/aws/amazon-neptune-for-graphql/blob/main/doc/cdk.md).

### Let me setup the resources manually or with my favorite DevOps toolchain
[Here](https://github.com/aws/amazon-neptune-for-graphql/blob/main/doc/resources.md) the detailed list of resorces needed to configure the GraphQL API pipeline.
[Here](https://github.com/aws/amazon-neptune-for-graphql/blob/main/doc/resources.md) the detailed list of resources needed to configure the GraphQL API pipeline.
<br>

# Known limitations
Expand All @@ -290,7 +290,7 @@ A copy of the License is located at http://www.apache.org/licenses/LICENSE-2.0
or in the "license" file accompanying this file. This file is distributed
on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
express or implied. See the License for the specific language governing
permissions and limitations under the License. [Full licence page](/LICENSE.txt).
permissions and limitations under the License. [Full license page](/LICENSE.txt).
<br>

# Contributing
Expand Down
Binary file added aws-neptune-for-graphql-1.1.0.tgz
Binary file not shown.
2 changes: 1 addition & 1 deletion doc/cdk.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Example starting from an Amazon Neptune database endpoint. In this case the util

Example starting from a GraphQL schema.

`neptune-for-graphql --input-schema-file `<*your-graphql-schema-file*>` --output-aws-pipeline-cdk --output-aws-pipeline-cdk-name` <*your-new-GraphQL-API-name*>` --output-aws-pipeline-cdk-neptume-endpoint` <*your-neptune-database-endpoint:port*>` --output-resolver-query-https`
`neptune-for-graphql --input-schema-file `<*your-graphql-schema-file*>` --output-aws-pipeline-cdk --output-aws-pipeline-cdk-name` <*your-new-GraphQL-API-name*>` --output-aws-pipeline-cdk-neptune-endpoint` <*your-neptune-database-endpoint:port*>` --output-resolver-query-https`



Expand Down
8 changes: 4 additions & 4 deletions doc/cliReference.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ The GraphQL schema with or without directives as data or file name.

`--input-schema-changes-file <value>, -isc <value>`
<br>
The file that contains your GraphQL schema changes to be applied when you run the utility multiple time. When you run the utility poiniting to a database multiple times, and you manually changed the GraphQL source schema, maybe adding a custom query, your chnages will be lost. To avoid it, add your changes to the a json file and pass it to this option. See section [*Re-apply your changes*](#re-apply-your-changes-with---input-schema-changes-file) for the file format.
The file that contains your GraphQL schema changes to be applied when you run the utility multiple time. When you run the utility pointing to a database multiple times, and you manually changed the GraphQL source schema, maybe adding a custom query, your changes will be lost. To avoid it, add your changes to the a json file and pass it to this option. See section [*Re-apply your changes*](#re-apply-your-changes-with---input-schema-changes-file) for the file format.

```json
[
Expand Down Expand Up @@ -64,7 +64,7 @@ The graphdb schema as data or file name. Instead of pointing to a Neptune databa

`--input-graphdb-schema-neptune-endpoint <value>, -ie <value>`
<br>
The Neptune database enpoint from which the utility extract the graphdb schema. Format: `host:port`
The Neptune database endpoint from which the utility extract the graphdb schema. Format: `host:port`

## Output options
`--output-folder-path <value>, -o <value>`
Expand All @@ -81,7 +81,7 @@ The file name output for the GraphQL schema with directives. If not specified th

`--output-schema-no-mutations, -onm`
<br>
The inferred GraphQL schema will not have mutuations, just queries.
The inferred GraphQL schema will not have mutations, just queries.

`--output-neptune-schema-file <value>, -og <value>`
<br>
Expand Down Expand Up @@ -135,7 +135,7 @@ It removes the pipeline created with `--create-update-aws-pipeline`. The resourc
<br>
This trigger the creation of a CDK file to be use to create the AWS resources for the GraphQL API, including the AppSync GraphQL API and the Lambda that run the resolver.

`--output-aws-pipeline-cdk-neptume-endpoint <value>, -ce <value>`
`--output-aws-pipeline-cdk-neptune-endpoint <value>, -ce <value>`
<br>
This set the Neptune database endpoint used by the Lambda function to query the Neptune database. If not set it used the endpoint set with `--input-graphdb-schema-neptune-endpoint`.

Expand Down
12 changes: 9 additions & 3 deletions package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@aws/neptune-for-graphql",
"version": "1.0.0",
"version": "1.1.0",
"description": "CLI utility to create and maintain a GraphQL API for Amazon Neptune",
"keywords": [
"Amazon Neptune",
Expand All @@ -19,9 +19,10 @@
"test": "test"
},
"scripts": {
"postinstall": "cd templates/Lambda4AppSyncHTTP && npm install && cd ../Lambda4AppSyncSDK && npm install",
"postinstall": "cd templates/Lambda4AppSyncHTTP && npm install && cd ../Lambda4AppSyncSDK && npm install && cd ../Lambda4AppSyncGraphSDK && npm install",
"lint": "eslint neptune-for-graphql.mjs ./src",
"test": "node --experimental-vm-modules node_modules/jest/bin/jest.js --coverage --runInBand --detectOpenHandles --config .jest.js",
"test:sdk": "node --experimental-vm-modules node_modules/jest/bin/jest.js --coverage --runInBand --detectOpenHandles --config .jest.js --testPathPattern=test/TestCases/Case07",
"test:resolver": "node --experimental-vm-modules node_modules/jest/bin/jest.js --coverage --runInBand --detectOpenHandles --config .jest.js --testPathPattern=test/TestCases/Case01"
},
"jest": {
Expand All @@ -42,6 +43,8 @@
"./templates/Lambda4AppSyncHTTP/package.json",
"./templates/Lambda4AppSyncSDK/index.mjs",
"./templates/Lambda4AppSyncSDK/package.json",
"./templates/Lambda4AppSyncGraphSDK/index.mjs",
"./templates/Lambda4AppSyncGraphSDK/package.json",
"./src/**"
],
"author": "AWS",
Expand All @@ -53,14 +56,17 @@
"@aws-sdk/client-lambda": "3.387.0",
"@aws-sdk/client-neptune": "3.387.0",
"@aws-sdk/client-neptunedata": "3.403.0",
"@aws-sdk/client-neptune-graph": "3.662.0",
"@aws-sdk/credential-providers": "3.414.0",
"archiver": "5.3.1",
"aws4-axios": "3.3.0",
"axios": "1.7.4",
"graphql": "^16.8.1",
"graphql-tag": "2.12.6",
"ora": "7.0.1",
"semver": "7.5.4"
"semver": "7.5.4",
"pino": "9.4.0",
"pino-pretty": "11.2.2"
},
"devDependencies": {
"@jest/test-sequencer": "^29.7.0",
Expand Down
4 changes: 0 additions & 4 deletions samples/todo.schema.graphql
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,4 @@ type Todo {

type Comment {
content: String
}

input Options {
limit: Int
}
Loading