Recover Quarantine using releases

README.md

@@ -16,7 +16,7 @@ This repository contains all the use cases you can iterate with Versions:
 - [Change S3 Data Source sorting key](change_sorting_key_to_s3_data_source)
 - [Change S3 Data Source sorting key with reingestion](change_sorting_key_to_s3_data_source_with_reingestion)
 - [Change Kafka Data Source sorting key](change_sorting_key_to_kafka_data_source)
-- [Recover data from quarantine](recover_data_from_quarantine) using a copy Pipe
+- [Recover data from quarantine](v3/Quarentine_V3) using a copy Pipe
 - [Add a new column to a Landing Data Source](add_column_landing_ds)
 - [Add a new column to a Materialized View](add_column_materialized_view)
 - [Add a new column to a Materialized View using Releases](v3/add_new_column_to_a_materialized_view_v3)

v3/Quarentine_V3/.tinyenv

@@ -1,2 +1,2 @@
 TB_VERSION_WARNING=0
-VERSION=0.0.0
+VERSION=0.0.1

v3/Quarentine_V3/README.md

@@ -1,3 +1,16 @@
-# Tinybird Versions - {{ YOUR USE CASE NAME HERE }}
+# Recover data from quarantine V3
 
-Work in progress ...
+When data ends up in quarantine, it is possible to re-ingest it using a Copy Pipe. Create a [Pull Request](https://github.com/tinybirdco/use-case-examples/pull/152) following these steps:
+
+> Remember to follow the [instructions](../README.md) to setup your Tinybird Data Project before jumping into the use-case steps
+
+- Bump a new CI/CD version and generate deployment scripts `tb release generate --semver 0.0.1`
+- In the CI file:
+    - Let's append incorrect data to `analytics_events` using a fixture (that's required to create the quarantine tables)
+    - Use `set +e` command when the incorrect data is being appended, if not the pipeline will finish with error.
+    - Create a copy Pipe to fix the incorrect data and re-ingest it into `analytics_events`
+- In the CD file, it is only needed run the copy Pipe after creation`
+- The temporary copy pipe will be created inside a Release (0.0.1), once the data is migrated is safe to remove the release:
+  ```
+  tb release rm --semver 0.0.1 --force --yes
+  ```

v3/Quarentine_V3/deploy/0.0.1/cd-deploy.sh

@@ -0,0 +1,5 @@
+#!/bin/bash
+set -e
+
+tb --semver 0.0.1 deploy --v3
+tb --semver 0.0.1 pipe copy run analytics_events_quarantine_to_final --wait --yes

v3/Quarentine_V3/deploy/0.0.1/ci-deploy.sh

@@ -0,0 +1,11 @@
+#!/bin/bash
+set +e
+
+# This line is only for demo purposes, It's adding wrong data to the analytics_events datasource
+tb datasource append analytics_events datasources/fixtures/analytics_events_errors.ndjson
+
+set -e
+
+tb --semver 0.0.1 deploy --fixtures --v3
+tb --semver 0.0.1 pipe copy run analytics_events_quarantine_to_final --wait --yes
+

v3/Quarentine_V3/pipes/analytics_events_quarantine_to_final.pipe

@@ -0,0 +1,14 @@
+NODE copy_quarantine
+SQL >
+    SELECT
+        toDateTime(
+            fromUnixTimestamp64Milli(toUInt64(replaceAll(assumeNotNull(timestamp), '"', '')) * 1000)
+        ) timestamp,
+        replaceAll(assumeNotNull(session_id), '"', '') session_id,
+        replaceAll(assumeNotNull(action), '"', '') action,
+        replaceAll(assumeNotNull(version), '"', '') version,
+        replaceAll(assumeNotNull(payload), '"', '') payload
+    FROM analytics_events_quarantine
+
+TYPE COPY
+TARGET_DATASOURCE analytics_events