Merge pull request #170 from tinybirdco/feat/cleaning-delete-a-resource

Moving "delete a resource" use case to the root
tinybirdco · Jan 10, 2024 · 9697ed3 · 9697ed3
2 parents b13e8e3 + 57a4fbc
commit 9697ed3
Show file tree

Hide file tree

Showing 39 changed files with 122 additions and 656 deletions.
diff --git a/.github/workflows/delete_simple_resource_cd.yml b/.github/workflows/delete_simple_resource_cd.yml
@@ -17,7 +17,7 @@
       cd:
         uses: tinybirdco/ci/.github/workflows/cd.yml@main
         with:
-          tb_deploy: false
+          tb_deploy: true
           data_project_dir: ./delete_simple_resource
         secrets:
           tb_admin_token: ${{ secrets.TB_ADMIN_TOKEN_DELETE_SIMPLE_RESOURCE }}

diff --git a/.github/workflows/delete_simple_resource_ci.yml b/.github/workflows/delete_simple_resource_ci.yml
@@ -21,6 +21,7 @@
         ci:
           uses: tinybirdco/ci/.github/workflows/ci.yml@main
           with:
+            tb_deploy: true
             data_project_dir: ./delete_simple_resource
           secrets:
             tb_admin_token: ${{ secrets.TB_ADMIN_TOKEN_DELETE_SIMPLE_RESOURCE }}

diff --git a/.github/workflows/v3_delete_simple_resource_cd.yml b/.github/workflows/v3_delete_simple_resource_cd.yml
diff --git a/.github/workflows/v3_delete_simple_resource_ci.yml b/.github/workflows/v3_delete_simple_resource_ci.yml
diff --git a/README.md b/README.md
@@ -11,27 +11,7 @@ Either you want to reproduce a use case or create a new one, please follow the [
 This repository contains all the use cases you can iterate with Versions:
 
 - [Delete simple resource](delete_simple_resource)
-- [Delete simple resource using Releases](v3/delete_simple_resource)
-- [Change Data Source sorting key](change_sorting_key)
-- [Change S3 Data Source sorting key](change_sorting_key_to_s3_data_source)
-- [Change S3 Data Source sorting key with reingestion](change_sorting_key_to_s3_data_source_with_reingestion)
-- [Change Kafka Data Source sorting key](change_sorting_key_to_kafka_data_source)
-- [Recover data from quarantine](v3/Quarentine_V3) using a copy Pipe
-- [Add a new column to a Landing Data Source](add_column_landing_ds)
-- [Add a new column to a Landing Data Source using V3](v3/add_column_landing_ds)
-- [Add a new column to a Materialized View](add_column_materialized_view)
-- [Add a new column to a Materialized View using Releases](v3/add_nullable_column_to_landing_datas_source)
-- [Add a new column in a BigQuery Data Source](add_column_BQ_ds)
-- [Add a new column to a Snowflake Data Source](add_column_snowflake_ds)
-- [Add a new column to an S3 Data Source](add_column_to_s3_ds)
-- [Add a new column to a Kafka Data Source](add_column_kafka_ds)
-- [Change a Data Source's TTL](change_ttl)
-- [Change column type from String to LowCardinality](change_column_type_to_lowcardinality)
-- [Remove Column from a Landing Data Source using Releases](v3/delete_column_from_landing_data_source_releases)
-- [Remove Column from a Materialized View](delete_column_materialized_view)
-- [Remove Column from a Snowflake Data Source](delete_column_snowflake_ds)
-- [Change Copy pipe and its target Data Source](change_copy_pipe)
-- [Use shared Data Source](use_new_columns_from_shared_datasource)
+
 
 ## Caveats
 

diff --git a/delete_simple_resource/.gitignore b/delete_simple_resource/.gitignore
diff --git a/delete_simple_resource/.tinyenv b/delete_simple_resource/.tinyenv
@@ -1 +1,2 @@
+TB_VERSION_WARNING=0
 VERSION=0.0.1
diff --git a/delete_simple_resource/README.md b/delete_simple_resource/README.md
@@ -1,21 +1,11 @@
 # Delete simple resource
-> Important! Resources Deletion has been improved and it's much easier using Releases: [Delete simple resource using Releases](v3/delete_simple_resource) 
 
-Delete simple Data Sources or Pipes easily. Create a [Pull Request](https://github.com/tinybirdco/use-case-examples/pull/5) following these steps:
+Delete simple Data Sources or Pipes easily. Create a [Pull Request](https://github.com/tinybirdco/use-case-examples/pull/143) following these steps:
 
 > Remember to follow the [instructions](../README.md) to setup your Tinybird Data Project before jumping into the use-case steps
 
-- Create a new branch
-- Remove the Pipes or Data Sources
-- Generate a new CI/CD version `tb release generate --semver 0.0.1`
-- Include your delete operations in both CI/CD files
-- Push your changes
+- Remove the Pipes or Data Sources datafiles you want to delete
+- Bump .tinyenv
+- CI/CD will protect you from removing resources with depedencies and doing in proper order
 
-Go to your Workspace and check the new Environment created. These changes will be displayed:
-
-![Changes in environment](./images/delete-resources.png)
-
-> For the moment, you need to specify the commands for the CI and CD, but in future versions it won't be needed
-
-
-[Internal workspace](https://ui.tinybird.co/128be410-8de1-4b1c-805c-145fdcf2566a/dashboard)
+[workspace](https://ui.tinybird.co/128be410-8de1-4b1c-805c-145fdcf2566a/dashboard)
diff --git a/delete_simple_resource/datasources/analytics_pages_mv.datasource b/delete_simple_resource/datasources/analytics_pages_mv.datasource
@@ -1,4 +1,3 @@
-# Data Source created from Pipe 'analytics_pages'
 
 SCHEMA >
     `date` Date,

diff --git a/delete_simple_resource/datasources/analytics_sessions_mv.datasource b/delete_simple_resource/datasources/analytics_sessions_mv.datasource
diff --git a/delete_simple_resource/datasources/analytics_sources_mv.datasource b/delete_simple_resource/datasources/analytics_sources_mv.datasource
@@ -1,4 +1,3 @@
-# Data Source created from Pipe 'analytics_sources'
 
 SCHEMA >
     `date` Date,

diff --git a/...asources/fixtures/analytics_events.ndjson → ...asources/fixtures/analytics_events.ndjson b/...asources/fixtures/analytics_events.ndjson → ...asources/fixtures/analytics_events.ndjson
diff --git a/delete_simple_resource/deploy/0.0.1/cd-deploy.sh b/delete_simple_resource/deploy/0.0.1/cd-deploy.sh
diff --git a/delete_simple_resource/deploy/0.0.1/ci-deploy.sh b/delete_simple_resource/deploy/0.0.1/ci-deploy.sh
diff --git a/delete_simple_resource/images/delete-resources.png b/delete_simple_resource/images/delete-resources.png
diff --git a/delete_simple_resource/pipes/analytics_pages.pipe b/delete_simple_resource/pipes/analytics_pages.pipe
@@ -22,7 +22,5 @@ SQL >
 
 TYPE materialized
 DATASOURCE analytics_pages_mv
-ENGINE "AggregatingMergeTree"
-ENGINE_PARTITION_KEY "toYYYYMM(date)"
-ENGINE_SORTING_KEY "date, device, browser, location, pathname"
+
 
diff --git a/delete_simple_resource/pipes/analytics_sessions.pipe b/delete_simple_resource/pipes/analytics_sessions.pipe
diff --git a/delete_simple_resource/pipes/analytics_sources.pipe b/delete_simple_resource/pipes/analytics_sources.pipe
@@ -7,6 +7,7 @@ SQL >
     WITH (
             SELECT domainWithoutWWW(href)
             FROM analytics_hits
+            WHERE href is not null and href != ''
             LIMIT 1
         ) AS currenct_domain
     SELECT
@@ -28,7 +29,5 @@ SQL >
 
 TYPE materialized
 DATASOURCE analytics_sources_mv
-ENGINE "AggregatingMergeTree"
-ENGINE_PARTITION_KEY "toYYYYMM(date)"
-ENGINE_SORTING_KEY "date, device, browser, location, referrer"
+
 
diff --git a/delete_simple_resource/pipes/top_browsers.pipe b/delete_simple_resource/pipes/top_browsers.pipe
@@ -13,27 +13,27 @@ DESCRIPTION >
 SQL >
 
     %
-        select
-          browser,
-          uniqMerge(visits) as visits,
-          countMerge(hits) as hits
-        from
-          analytics_sources_mv
-        where
-          {% if defined(date_from) %}
-            date >= {{Date(date_from, description="Starting day for filtering a date range", required=False)}}
-          {% else %}
-            date >= timestampAdd(today(), interval -7 day)
-          {% end %}
-          {% if defined(date_to) %}
-            and date <= {{Date(date_to, description="Finishing day for filtering a date range", required=False)}}
-          {% else %}
-            and date <= today()
-          {% end %}
-        group by
-          browser
-        order by
-          visits desc
-        limit {{Int32(skip, 0)}},{{Int32(limit, 50)}}
+            select
+              browser,
+              uniqMerge(visits) as visits,
+              countMerge(hits) as hits
+            from
+              analytics_sources_mv
+            where
+              {% if defined(date_from) %}
+                date >= {{Date(date_from, description="Starting day for filtering a date range", required=False)}}
+              {% else %}
+                date >= timestampAdd(today(), interval -7 day)
+              {% end %}
+              {% if defined(date_to) %}
+                and date <= {{Date(date_to, description="Finishing day for filtering a date range", required=False)}}
+              {% else %}
+                and date <= today()
+              {% end %}
+            group by
+              browser
+            order by
+              visits desc
+            limit {{Int32(skip, 0)}},{{Int32(limit, 50)}}
 
 
diff --git a/delete_simple_resource/pipes/top_devices.pipe b/delete_simple_resource/pipes/top_devices.pipe
@@ -13,27 +13,27 @@ DESCRIPTION >
 SQL >
 
     %
-        select
-          device,
-          uniqMerge(visits) as visits,
-          countMerge(hits) as hits
-        from
-          analytics_sources_mv
-        where
-          {% if defined(date_from) %}
-            date >= {{Date(date_from, description="Starting day for filtering a date range", required=False)}}
-          {% else %}
-            date >= timestampAdd(today(), interval -7 day)
-          {% end %}
-          {% if defined(date_to) %}
-            and date <= {{Date(date_to, description="Finishing day for filtering a date range", required=False)}}
-          {% else %}
-            and date <= today()
-          {% end %}
-        group by
-          device
-        order by
-          visits desc
-        limit {{Int32(skip, 0)}},{{Int32(limit, 50)}}
+            select
+              device,
+              uniqMerge(visits) as visits,
+              countMerge(hits) as hits
+            from
+              analytics_sources_mv
+            where
+              {% if defined(date_from) %}
+                date >= {{Date(date_from, description="Starting day for filtering a date range", required=False)}}
+              {% else %}
+                date >= timestampAdd(today(), interval -7 day)
+              {% end %}
+              {% if defined(date_to) %}
+                and date <= {{Date(date_to, description="Finishing day for filtering a date range", required=False)}}
+              {% else %}
+                and date <= today()
+              {% end %}
+            group by
+              device
+            order by
+              visits desc
+            limit {{Int32(skip, 0)}},{{Int32(limit, 50)}}
 
 
diff --git a/delete_simple_resource/pipes/top_locations.pipe b/delete_simple_resource/pipes/top_locations.pipe
@@ -13,27 +13,27 @@ DESCRIPTION >
 SQL >
 
     %
-        select
-          location,
-          uniqMerge(visits) as visits,
-          countMerge(hits) as hits
-        from
-          analytics_pages_mv
-        where
-          {% if defined(date_from) %}
-            date >= {{Date(date_from, description="Starting day for filtering a date range", required=False)}}
-          {% else %}
-            date >= timestampAdd(today(), interval -7 day)
-          {% end %}
-          {% if defined(date_to) %}
-            and date <= {{Date(date_to, description="Finishing day for filtering a date range", required=False)}}
-          {% else %}
-            and date <= today()
-          {% end %}
-        group by
-          location
-        order by
-          visits desc
-        limit {{Int32(skip, 0)}},{{Int32(limit, 50)}}
+            select
+              location,
+              uniqMerge(visits) as visits,
+              countMerge(hits) as hits
+            from
+              analytics_pages_mv
+            where
+              {% if defined(date_from) %}
+                date >= {{Date(date_from, description="Starting day for filtering a date range", required=False)}}
+              {% else %}
+                date >= timestampAdd(today(), interval -7 day)
+              {% end %}
+              {% if defined(date_to) %}
+                and date <= {{Date(date_to, description="Finishing day for filtering a date range", required=False)}}
+              {% else %}
+                and date <= today()
+              {% end %}
+            group by
+              location
+            order by
+              visits desc
+            limit {{Int32(skip, 0)}},{{Int32(limit, 50)}}