Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Moving "delete a resource" use case to the root #170

Merged
merged 5 commits into from
Jan 10, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/delete_simple_resource_cd.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
cd:
uses: tinybirdco/ci/.github/workflows/cd.yml@main
with:
tb_deploy: false
tb_deploy: true
data_project_dir: ./delete_simple_resource
secrets:
tb_admin_token: ${{ secrets.TB_ADMIN_TOKEN_DELETE_SIMPLE_RESOURCE }}
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/delete_simple_resource_ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
ci:
uses: tinybirdco/ci/.github/workflows/ci.yml@main
with:
tb_deploy: true
data_project_dir: ./delete_simple_resource
secrets:
tb_admin_token: ${{ secrets.TB_ADMIN_TOKEN_DELETE_SIMPLE_RESOURCE }}
Expand Down
25 changes: 0 additions & 25 deletions .github/workflows/v3_delete_simple_resource_cd.yml

This file was deleted.

29 changes: 0 additions & 29 deletions .github/workflows/v3_delete_simple_resource_ci.yml

This file was deleted.

22 changes: 1 addition & 21 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,27 +11,7 @@ Either you want to reproduce a use case or create a new one, please follow the [
This repository contains all the use cases you can iterate with Versions:

- [Delete simple resource](delete_simple_resource)
- [Delete simple resource using Releases](v3/delete_simple_resource)
- [Change Data Source sorting key](change_sorting_key)
- [Change S3 Data Source sorting key](change_sorting_key_to_s3_data_source)
- [Change S3 Data Source sorting key with reingestion](change_sorting_key_to_s3_data_source_with_reingestion)
- [Change Kafka Data Source sorting key](change_sorting_key_to_kafka_data_source)
- [Recover data from quarantine](v3/Quarentine_V3) using a copy Pipe
- [Add a new column to a Landing Data Source](add_column_landing_ds)
- [Add a new column to a Landing Data Source using V3](v3/add_column_landing_ds)
- [Add a new column to a Materialized View](add_column_materialized_view)
- [Add a new column to a Materialized View using Releases](v3/add_nullable_column_to_landing_datas_source)
- [Add a new column in a BigQuery Data Source](add_column_BQ_ds)
- [Add a new column to a Snowflake Data Source](add_column_snowflake_ds)
- [Add a new column to an S3 Data Source](add_column_to_s3_ds)
- [Add a new column to a Kafka Data Source](add_column_kafka_ds)
- [Change a Data Source's TTL](change_ttl)
- [Change column type from String to LowCardinality](change_column_type_to_lowcardinality)
- [Remove Column from a Landing Data Source using Releases](v3/delete_column_from_landing_data_source_releases)
- [Remove Column from a Materialized View](delete_column_materialized_view)
- [Remove Column from a Snowflake Data Source](delete_column_snowflake_ds)
- [Change Copy pipe and its target Data Source](change_copy_pipe)
- [Use shared Data Source](use_new_columns_from_shared_datasource)


## Caveats

Expand Down
7 changes: 0 additions & 7 deletions delete_simple_resource/.gitignore

This file was deleted.

1 change: 1 addition & 0 deletions delete_simple_resource/.tinyenv
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
TB_VERSION_WARNING=0
VERSION=0.0.1
20 changes: 5 additions & 15 deletions delete_simple_resource/README.md
Original file line number Diff line number Diff line change
@@ -1,21 +1,11 @@
# Delete simple resource
> Important! Resources Deletion has been improved and it's much easier using Releases: [Delete simple resource using Releases](v3/delete_simple_resource)

Delete simple Data Sources or Pipes easily. Create a [Pull Request](https://github.com/tinybirdco/use-case-examples/pull/5) following these steps:
Delete simple Data Sources or Pipes easily. Create a [Pull Request](https://github.com/tinybirdco/use-case-examples/pull/143) following these steps:

> Remember to follow the [instructions](../README.md) to setup your Tinybird Data Project before jumping into the use-case steps

- Create a new branch
- Remove the Pipes or Data Sources
- Generate a new CI/CD version `tb release generate --semver 0.0.1`
- Include your delete operations in both CI/CD files
- Push your changes
- Remove the Pipes or Data Sources datafiles you want to delete
- Bump .tinyenv
- CI/CD will protect you from removing resources with depedencies and doing in proper order

Go to your Workspace and check the new Environment created. These changes will be displayed:

![Changes in environment](./images/delete-resources.png)

> For the moment, you need to specify the commands for the CI and CD, but in future versions it won't be needed


[Internal workspace](https://ui.tinybird.co/128be410-8de1-4b1c-805c-145fdcf2566a/dashboard)
[workspace](https://ui.tinybird.co/128be410-8de1-4b1c-805c-145fdcf2566a/dashboard)
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
# Data Source created from Pipe 'analytics_pages'

SCHEMA >
`date` Date,
Expand Down

This file was deleted.

Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
# Data Source created from Pipe 'analytics_sources'

SCHEMA >
`date` Date,
Expand Down
5 changes: 0 additions & 5 deletions delete_simple_resource/deploy/0.0.1/cd-deploy.sh

This file was deleted.

5 changes: 0 additions & 5 deletions delete_simple_resource/deploy/0.0.1/ci-deploy.sh

This file was deleted.

Binary file removed delete_simple_resource/images/delete-resources.png
Binary file not shown.
4 changes: 1 addition & 3 deletions delete_simple_resource/pipes/analytics_pages.pipe
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,5 @@ SQL >

TYPE materialized
DATASOURCE analytics_pages_mv
ENGINE "AggregatingMergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(date)"
ENGINE_SORTING_KEY "date, device, browser, location, pathname"


26 changes: 0 additions & 26 deletions delete_simple_resource/pipes/analytics_sessions.pipe

This file was deleted.

5 changes: 2 additions & 3 deletions delete_simple_resource/pipes/analytics_sources.pipe
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ SQL >
WITH (
SELECT domainWithoutWWW(href)
FROM analytics_hits
WHERE href is not null and href != ''
LIMIT 1
) AS currenct_domain
SELECT
Expand All @@ -28,7 +29,5 @@ SQL >

TYPE materialized
DATASOURCE analytics_sources_mv
ENGINE "AggregatingMergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(date)"
ENGINE_SORTING_KEY "date, device, browser, location, referrer"


44 changes: 22 additions & 22 deletions delete_simple_resource/pipes/top_browsers.pipe
Original file line number Diff line number Diff line change
Expand Up @@ -13,27 +13,27 @@ DESCRIPTION >
SQL >

%
select
browser,
uniqMerge(visits) as visits,
countMerge(hits) as hits
from
analytics_sources_mv
where
{% if defined(date_from) %}
date >= {{Date(date_from, description="Starting day for filtering a date range", required=False)}}
{% else %}
date >= timestampAdd(today(), interval -7 day)
{% end %}
{% if defined(date_to) %}
and date <= {{Date(date_to, description="Finishing day for filtering a date range", required=False)}}
{% else %}
and date <= today()
{% end %}
group by
browser
order by
visits desc
limit {{Int32(skip, 0)}},{{Int32(limit, 50)}}
select
browser,
uniqMerge(visits) as visits,
countMerge(hits) as hits
from
analytics_sources_mv
where
{% if defined(date_from) %}
date >= {{Date(date_from, description="Starting day for filtering a date range", required=False)}}
{% else %}
date >= timestampAdd(today(), interval -7 day)
{% end %}
{% if defined(date_to) %}
and date <= {{Date(date_to, description="Finishing day for filtering a date range", required=False)}}
{% else %}
and date <= today()
{% end %}
group by
browser
order by
visits desc
limit {{Int32(skip, 0)}},{{Int32(limit, 50)}}


44 changes: 22 additions & 22 deletions delete_simple_resource/pipes/top_devices.pipe
Original file line number Diff line number Diff line change
Expand Up @@ -13,27 +13,27 @@ DESCRIPTION >
SQL >

%
select
device,
uniqMerge(visits) as visits,
countMerge(hits) as hits
from
analytics_sources_mv
where
{% if defined(date_from) %}
date >= {{Date(date_from, description="Starting day for filtering a date range", required=False)}}
{% else %}
date >= timestampAdd(today(), interval -7 day)
{% end %}
{% if defined(date_to) %}
and date <= {{Date(date_to, description="Finishing day for filtering a date range", required=False)}}
{% else %}
and date <= today()
{% end %}
group by
device
order by
visits desc
limit {{Int32(skip, 0)}},{{Int32(limit, 50)}}
select
device,
uniqMerge(visits) as visits,
countMerge(hits) as hits
from
analytics_sources_mv
where
{% if defined(date_from) %}
date >= {{Date(date_from, description="Starting day for filtering a date range", required=False)}}
{% else %}
date >= timestampAdd(today(), interval -7 day)
{% end %}
{% if defined(date_to) %}
and date <= {{Date(date_to, description="Finishing day for filtering a date range", required=False)}}
{% else %}
and date <= today()
{% end %}
group by
device
order by
visits desc
limit {{Int32(skip, 0)}},{{Int32(limit, 50)}}


44 changes: 22 additions & 22 deletions delete_simple_resource/pipes/top_locations.pipe
Original file line number Diff line number Diff line change
Expand Up @@ -13,27 +13,27 @@ DESCRIPTION >
SQL >

%
select
location,
uniqMerge(visits) as visits,
countMerge(hits) as hits
from
analytics_pages_mv
where
{% if defined(date_from) %}
date >= {{Date(date_from, description="Starting day for filtering a date range", required=False)}}
{% else %}
date >= timestampAdd(today(), interval -7 day)
{% end %}
{% if defined(date_to) %}
and date <= {{Date(date_to, description="Finishing day for filtering a date range", required=False)}}
{% else %}
and date <= today()
{% end %}
group by
location
order by
visits desc
limit {{Int32(skip, 0)}},{{Int32(limit, 50)}}
select
location,
uniqMerge(visits) as visits,
countMerge(hits) as hits
from
analytics_pages_mv
where
{% if defined(date_from) %}
date >= {{Date(date_from, description="Starting day for filtering a date range", required=False)}}
{% else %}
date >= timestampAdd(today(), interval -7 day)
{% end %}
{% if defined(date_to) %}
and date <= {{Date(date_to, description="Finishing day for filtering a date range", required=False)}}
{% else %}
and date <= today()
{% end %}
group by
location
order by
visits desc
limit {{Int32(skip, 0)}},{{Int32(limit, 50)}}


Loading
Loading