Skip to content

Commit

Permalink
Change ci_* prefix to z_ci_* in Terraform resources (#109)
Browse files Browse the repository at this point in the history
* Change `ci_*` prefix to `z_ci_*` in Terraform resources

* Adjust substr calls in main.tf to reflect longer CI prefixes

* Change s3 bucket prefix from `ccao-z-ci` to `z-ci-ccao` to better match normal pattern
  • Loading branch information
jeancochrane authored Mar 14, 2024
1 parent 8eec146 commit aa2fd56
Show file tree
Hide file tree
Showing 2 changed files with 7 additions and 7 deletions.
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -314,7 +314,7 @@ This repository manages the configurations, scripts, and details for an AWS Glue
2. All changes to these components should originate from this repository. This ensures that every modification is tracked and version-controlled.
3. The **only** advisable actions in the AWS Console concerning this Glue job are:
- Running the job
4. To test a change to the Glue job script or the flagging script, make an edit on a branch and open a pull request. Our GitHub Actions configuration will deploy a staging version of your job, named `ci_<your-branch-name>_sales_val_flagging`, that you can run to test your changes. See the [Modifying the Glue job](#modifying-the-glue-job-its-flagging-script-or-its-settings) section below for more details.
4. To test a change to the Glue job script or the flagging script, make an edit on a branch and open a pull request. Our GitHub Actions configuration will deploy a staging version of your job, named `z_ci_<your-branch-name>_sales_val_flagging`, that you can run to test your changes. See the [Modifying the Glue job](#modifying-the-glue-job-its-flagging-script-or-its-settings) section below for more details.

### Modifying the Glue job, its flagging script, or its settings

Expand All @@ -327,7 +327,7 @@ The Glue job and its flagging script are written in Python, while the job detail
2. Any changes to these files should be made in the following sequence:
- Make a new git branch for your changes.
- Edit the files as necessary.
- Open a pull request for your changes against the `main` branch. A GitHub Actions workflow called `deploy-terraform` will deploy a staging version of your job named `ci_<your-branch-name>_sales_val_flagging` that you can run to test your changes.
- Open a pull request for your changes against the `main` branch. A GitHub Actions workflow called `deploy-terraform` will deploy a staging version of your job named `z_ci_<your-branch-name>_sales_val_flagging` that you can run to test your changes.
- By default, this configuration will deploy an empty version of the `sale.flag` table, which simulates an environment in which there are no preexisting flags prior to a run.
- If you would like to test your job against a subset of the production data, copy your data subset from the production job bucket to the bucket created by Terraform for your job (or leave the new bucket empty to simulate running the job when no flags exist). Then, run the crawler created by Terraform for your PR in order to populate the staging version of the `sale.flag` database that your staging job uses. If you're having trouble finding your staging bucket, job, or crawler, check the GitHub Actions output for the first successful run of your PR and look for the Terraform output displaying the IDs of these resources.
- If you need to make further changes, push commits to your branch and GitHub Actions will deploy the changes to the staging job and its associated resources.
Expand Down
10 changes: 5 additions & 5 deletions main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,8 @@ locals {
s3_prefix = "scripts/sales-val"
s3_bucket_data_warehouse = terraform.workspace == "prod" ? "ccao-data-warehouse-us-east-1" : aws_s3_bucket.data_warehouse[0].id
s3_bucket_glue_assets = terraform.workspace == "prod" ? "ccao-glue-assets-us-east-1" : aws_s3_bucket.glue_assets[0].id
glue_job_name = terraform.workspace == "prod" ? "sales_val_flagging" : "ci_${terraform.workspace}_sales_val_flagging"
glue_crawler_name = terraform.workspace == "prod" ? "ccao-data-warehouse-sale-crawler" : "ci_${terraform.workspace}_ccao-data-warehouse-sale-crawler"
glue_job_name = terraform.workspace == "prod" ? "sales_val_flagging" : "z_ci_${terraform.workspace}_sales_val_flagging"
glue_crawler_name = terraform.workspace == "prod" ? "ccao-data-warehouse-sale-crawler" : "z_ci_${terraform.workspace}_ccao-data-warehouse-sale-crawler"
glue_table_sale_flag_parameters = {
CrawlerSchemaDeserializerVersion = "1.0"
CrawlerSchemaSerializerVersion = "1.0"
Expand All @@ -42,7 +42,7 @@ locals {
# (Note that this is not always true -- notably, dbt-athena is able to
# create Athena tables with hyphens -- but it's a rule that Terraform
# enforces, so we follow it here)
athena_database_name = terraform.workspace == "prod" ? "sale" : "ci_model_sales_val_${replace(terraform.workspace, "-", "_")}_sale"
athena_database_name = terraform.workspace == "prod" ? "sale" : "z_ci_model_sales_val_${replace(terraform.workspace, "-", "_")}_sale"
}

variable "iam_role_arn" {
Expand All @@ -66,7 +66,7 @@ resource "aws_s3_bucket" "glue_assets" {
count = terraform.workspace == "prod" ? 0 : 1
# Buckets can only be a max of 63 characters long:
# https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html
bucket = "ccao-ci-${substr(terraform.workspace, 0, 33)}-glue-assets-us-east-1"
bucket = "z-ci-ccao-${substr(terraform.workspace, 0, 31)}-glue-assets-us-east-1"
force_destroy = true
}

Expand All @@ -91,7 +91,7 @@ resource "aws_s3_bucket_versioning" "glue_assets" {

resource "aws_s3_bucket" "data_warehouse" {
count = terraform.workspace == "prod" ? 0 : 1
bucket = "ccao-ci-${substr(terraform.workspace, 0, 30)}-data-warehouse-us-east-1"
bucket = "z-ci-ccao-${substr(terraform.workspace, 0, 28)}-data-warehouse-us-east-1"
force_destroy = true
}

Expand Down

0 comments on commit aa2fd56

Please sign in to comment.