Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change ci_* prefix to z_ci_* in Terraform resources #109

Merged
merged 3 commits into from
Mar 14, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -314,7 +314,7 @@ This repository manages the configurations, scripts, and details for an AWS Glue
2. All changes to these components should originate from this repository. This ensures that every modification is tracked and version-controlled.
3. The **only** advisable actions in the AWS Console concerning this Glue job are:
- Running the job
4. To test a change to the Glue job script or the flagging script, make an edit on a branch and open a pull request. Our GitHub Actions configuration will deploy a staging version of your job, named `ci_<your-branch-name>_sales_val_flagging`, that you can run to test your changes. See the [Modifying the Glue job](#modifying-the-glue-job-its-flagging-script-or-its-settings) section below for more details.
4. To test a change to the Glue job script or the flagging script, make an edit on a branch and open a pull request. Our GitHub Actions configuration will deploy a staging version of your job, named `z_ci_<your-branch-name>_sales_val_flagging`, that you can run to test your changes. See the [Modifying the Glue job](#modifying-the-glue-job-its-flagging-script-or-its-settings) section below for more details.

### Modifying the Glue job, its flagging script, or its settings

Expand All @@ -327,7 +327,7 @@ The Glue job and its flagging script are written in Python, while the job detail
2. Any changes to these files should be made in the following sequence:
- Make a new git branch for your changes.
- Edit the files as necessary.
- Open a pull request for your changes against the `main` branch. A GitHub Actions workflow called `deploy-terraform` will deploy a staging version of your job named `ci_<your-branch-name>_sales_val_flagging` that you can run to test your changes.
- Open a pull request for your changes against the `main` branch. A GitHub Actions workflow called `deploy-terraform` will deploy a staging version of your job named `z_ci_<your-branch-name>_sales_val_flagging` that you can run to test your changes.
- By default, this configuration will deploy an empty version of the `sale.flag` table, which simulates an environment in which there are no preexisting flags prior to a run.
- If you would like to test your job against a subset of the production data, copy your data subset from the production job bucket to the bucket created by Terraform for your job (or leave the new bucket empty to simulate running the job when no flags exist). Then, run the crawler created by Terraform for your PR in order to populate the staging version of the `sale.flag` database that your staging job uses. If you're having trouble finding your staging bucket, job, or crawler, check the GitHub Actions output for the first successful run of your PR and look for the Terraform output displaying the IDs of these resources.
- If you need to make further changes, push commits to your branch and GitHub Actions will deploy the changes to the staging job and its associated resources.
Expand Down
10 changes: 5 additions & 5 deletions main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,8 @@ locals {
s3_prefix = "scripts/sales-val"
s3_bucket_data_warehouse = terraform.workspace == "prod" ? "ccao-data-warehouse-us-east-1" : aws_s3_bucket.data_warehouse[0].id
s3_bucket_glue_assets = terraform.workspace == "prod" ? "ccao-glue-assets-us-east-1" : aws_s3_bucket.glue_assets[0].id
glue_job_name = terraform.workspace == "prod" ? "sales_val_flagging" : "ci_${terraform.workspace}_sales_val_flagging"
glue_crawler_name = terraform.workspace == "prod" ? "ccao-data-warehouse-sale-crawler" : "ci_${terraform.workspace}_ccao-data-warehouse-sale-crawler"
glue_job_name = terraform.workspace == "prod" ? "sales_val_flagging" : "z_ci_${terraform.workspace}_sales_val_flagging"
glue_crawler_name = terraform.workspace == "prod" ? "ccao-data-warehouse-sale-crawler" : "z_ci_${terraform.workspace}_ccao-data-warehouse-sale-crawler"
glue_table_sale_flag_parameters = {
CrawlerSchemaDeserializerVersion = "1.0"
CrawlerSchemaSerializerVersion = "1.0"
Expand All @@ -42,7 +42,7 @@ locals {
# (Note that this is not always true -- notably, dbt-athena is able to
# create Athena tables with hyphens -- but it's a rule that Terraform
# enforces, so we follow it here)
athena_database_name = terraform.workspace == "prod" ? "sale" : "ci_model_sales_val_${replace(terraform.workspace, "-", "_")}_sale"
athena_database_name = terraform.workspace == "prod" ? "sale" : "z_ci_model_sales_val_${replace(terraform.workspace, "-", "_")}_sale"
}

variable "iam_role_arn" {
Expand All @@ -66,7 +66,7 @@ resource "aws_s3_bucket" "glue_assets" {
count = terraform.workspace == "prod" ? 0 : 1
# Buckets can only be a max of 63 characters long:
# https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html
bucket = "ccao-ci-${substr(terraform.workspace, 0, 33)}-glue-assets-us-east-1"
bucket = "z-ci-ccao-${substr(terraform.workspace, 0, 31)}-glue-assets-us-east-1"
force_destroy = true
}

Expand All @@ -91,7 +91,7 @@ resource "aws_s3_bucket_versioning" "glue_assets" {

resource "aws_s3_bucket" "data_warehouse" {
count = terraform.workspace == "prod" ? 0 : 1
bucket = "ccao-ci-${substr(terraform.workspace, 0, 30)}-data-warehouse-us-east-1"
bucket = "z-ci-ccao-${substr(terraform.workspace, 0, 28)}-data-warehouse-us-east-1"
force_destroy = true
}

Expand Down
Loading