Reorder and improve documentation.

Author: krisgeus

README.md

@@ -24,7 +24,7 @@ brew install jq
 The current implementation was developed on macOS but is intended to work with any platform supported by Docker. In our experience, Linux and macOS are fine. You can run it on native Windows 10 using [WSL](https://docs.microsoft.com/en-us/windows/wsl/about). Unfortunately, Docker on Windows 10 (version 1809) is hamstrung because it relies on Windows File Sharing (CIFS) to establish the volume mounts. Airflow hammers the volume a little harder than CIFS can handle, and you'll see intermittent FileNotFound errors in the volume mount. This may improve in the future. For now, running _whirl_ inside a Linux VM in Hyper-V gives more reliable results.
 
 ### Airflow Versions
-As of January 2021, Whirl uses Airflow 2.0.0 as the default version. A specific tag was made for Airflow 1.10.x, which can be found [here](https://github.com/godatadriven/whirl/tree/airflow-1.10.x)
+As of January 2021, Whirl uses Airflow 2.x.x as the default version. A specific tag was made for Airflow 1.10.x, which can be found [here](https://github.com/godatadriven/whirl/tree/airflow-1.10.x)
 
 ## Getting Started
 
@@ -68,23 +68,25 @@ Stops the configured environment.
 
 If you want to stop all containers from a specific environment you can add the `-e` or `--environment` commandline argument with the name of the environment. This name corresponds with a directory in the `envs` directory.
 
-#### Usage in a CI Pipeline _(work in progress)_
+#### Usage in a CI Pipeline
 
-We do not currently have a complete example of how to usage _whirl_ as part of a CI pipeline. However the first step in doing this is involves starting while in `ci` mode. This will:
+We run most of the examples from within our own CI (github actions, see for implementation details our [github workflow](.github/workflows/whirl-ci.yml).
+
+You are able to run an example in `ci` mode on your local system by useing the `whirl ci` command. This will:
 
   - run the Docker containers daemonized in the background;
   - ensure the DAG(s) are unpaused; and
   - wait for the pipeline to either succeed or fail.
 
 Upon success the containers will be stopped and exit successfully.
 
-At present we don't exit upon failure because it can be useful to be able to inspect the environment to see what happened. In the future we plan to print out the logs of the failed task and cleanup before indicating the pipeline has failed.
+In case of failure (or success if failure is expected)we print out the logs of the failed task and cleanup before indicating the pipeline has failed.
 
 #### Configuring Environment Variables
 
 Instead of using the environment option each time you run _whirl_, you can also configure your environment in a `.whirl.env` file. This can be in three places. They are applied in order:
 
-- A `.whirl.env` file in the root this repository. This can also specify a default environment to be used when starting _whirl_. You do this by setting the `WHIRL_ENVIRONMENT` which references a directory in the [`envs`](./envs) folder. This repository contains an example you can modify. It specifies the default `PYTHON_VERSION` to be used in any environment.
+- A `.whirl.env` file in the root of this repository. This can also specify a default environment to be used when starting _whirl_. You do this by setting the `WHIRL_ENVIRONMENT` which references a directory in the [`envs`](./envs) folder. This repository contains an example you can modify. It specifies the default `PYTHON_VERSION` to be used in any environment.
 - A `.whirl.env` file in your [`envs/{your-env}`](./envs) subdirectory. The environment directory to use can be set by any of the other `.whirl.env` files or specified on the commandline. This is helpful to set environment specific variables. Of course it doesn't make much sense to set the `WHIRL_ENVIRONMENT` here.
 - A `.whirl.env` in your DAG directory to override any environment variables. This can be useful for example to overwrite the (default) `WHIRL_ENVIRONMENT`.
 
@@ -128,145 +130,22 @@ This is also a location for installing and configuring extra client libraries th
 
 This repository contains some example environments and workflows. The components used might serve as a starting point for your own environment. If you have a good example you'd like to add, please submit a merge request!
 
-#### SSH to Localhost
-
-The first example environment only involves one component, the Apache Airflow docker container itself. The environment contains one preparation script called `01_enable_local_ssh.sh` which makes it possible in that container to SSH to `localhost`. The script also adds a new connection called `ssh_local` to the Airflow connections. The directory `example/localhost-ssh-example/` contains a single file, the Airflow DAG, so we have to pass the whirl environment as an argument. 
-
-To run this example:
-
-```bash
-$ cd ./examples/localhost-ssh-example
-# Note: here we pass the whirl environment 'local-ssh' as a command-line argument.
-$ whirl -e local-ssh
-```
-
-Open your browser to [http://localhost:5000](http://localhost:5000) to access the Airflow UI. Manually enable the DAG and watch the pipeline run to successful completion.
+Each example contains it's own README file to explain the specifics of that example.
 
-#### Rest API to S3 Storage Example
+#### Generic running of examples
 
-In this example we are going to:
+From within the example directory the `whirl` command can be executed. 
 
-1. Consume a REST API;
-2. Convert the JSON data to Parquet;
-3. Store the result in a S3 bucket.
-
-The environment includes containers for:
-
- - A S3 server;
- - A MockServer instance
- - The core Airflow component.
- 
-The environment contains a setup script in the `whirl.setup.d/` folder:
-
- - `01_add_connection_api.sh` which:
-
-   -  Adds a S3 connection to Airflow;
-   -  Installs the `awscli` Python libraries and configures them to connect to the S3 server;
-   -  Creates a bucket (with a `/etc/hosts` entry to support the [virtual host style method](https://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html)).
-
-To run this example:
+To run a example:
 
 ```bash
-$ cd ./examples/api-to-s3
-$ whirl
+$ cd ./examples/<example-dag-directory>
+# Note: here we pass the whirl environment as a command-line argument. It can also be configured with the WHIRL_ENVIRONMENT variable
+$ whirl -e <environment to use>
 ```
 
 Open your browser to [http://localhost:5000](http://localhost:5000) to access the Airflow UI. Manually enable the DAG and watch the pipeline run to successful completion.
 
-This example includes a `.whirl.env` configuration file in the DAG directory. In the environment folder there is also a `.whirl.env` which specifies S3-specific variables. The example folder also contains a `whirl.setup.d/` directory which contains an initialization script (`01_add_connection_api_and_mockdata.sh`). This script is executed in the container after the environment-specific scripts have run and will:
-
- - Add a connection to the API endpoint;
- - Add an [expectation](http://www.mock-server.com/mock_server/creating_expectations.html) for the MockServer to know which response needs to be sent for which requested path;
- - Install Pandas and PyArrow to support transforming the JSON into a Parquet file;
- - Create a local directory where the intermediate file is stored before being uploaded to S3.
-
-#### SFTPOperator + PythonOperator + MySQL Example
-
-This example includes containers for:
-
- - A SFTP server;
- - A MySQL instance;
- - The core Airflow component.
- 
-The environment contains two startup scripts in the `whirl.setup.d/` folder:
-
- - `01_prepare_sftp.sh` which adds a SFTP connection to Airflow;
- - `02_prepare_mysql.sh` which adds a MySQL connection to Airflow.
-
-To run this example:
-
-```bash
-$ cd ./examples/sftp-mysql-example
-$ whirl
-```
-
-Open your browser to [http://localhost:5000](http://localhost:5000) to access the Airflow UI. Manually enable the DAG and watch the pipeline run to successful completion.
-
-The environment to be used is set in the `.whirl.env` in the DAG directory. In the environment folder there is also a `.whirl.env` which specifies how `MOCK_DATA_FOLDER` is set. The DAG folder also contains a `whirl.setup.d/` directory which contains the script `01_cp_mock_data_to_sftp.sh`. This script gets executed in the container after the environment specific scripts have run and will do a couple of things:
-
-1. It will rename the file `mocked-data-#ds_nodash#.csv` that is in the `./mock-data/` folder. It will replace `#ds_nodash#` with the same value that Apache Airflow will use when templating `ds_nodash` in the Python files. This means we have a file available for our specific DAG run. (The logic to rename these files is located in `/etc/airflow/functions/date_replacement.sh` in the Airflow container.)
-2. It will copy this file to the SFTP server, where the DAG expects to find it. When the DAG starts it will try to copy that file from the SFTP server to the local filesystem.
-
-#### Remote logging for Airflow
-
-In this example the dag is not the most important part. This example is all about how to configure airflow to log to S3.
-We have created an environment that spins up an S3 server together with the Airflow one. The environment contains a setup script in the `whirl.setup.d` folder:
-
-- `01_add_connection_s3.sh` which:
-    -  adds an S3 connection to Airflow
-    -  Installs awscli Python libraries and configures them to connect to the S3 server
-    -  Creates a bucket (with adding a `/etc/hosts` entry to support the [virtual host style method](https://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html))
-- `02_configue_logging_to_s3.sh` which:
-    -  exports environment varibles which airflow uses to override the default config. For example: `export AIRFLOW__CORE__REMOTE_LOGGING=True`
-
-
-To run the corresponding example DAG, perform the following (assuming you have put _whirl_ to your `PATH`)
-
-```bash
-$ cd ./examples/logging-to-s3
-$ whirl
-```
-
-Open your browser to [http://localhost:5000](http://localhost:5000) to see the Airflow UI appear. Manually enable the DAG and see the pipeline get marked success. If you open one of the logs, the first line shows that the log is retrieved from S3.
-
-The environment to be used is set in the `.whirl.env` in the DAG directory. In the environment folder there is also a `.whirl.env` which specifies S3 specific variables.
-
-
-#### Having an external database for Airflow
-
-In this example the dag is not the most important part. This example is all about how to configure airflow to use a external database.
-We have created an environment that spins up an postgres database server together with the Airflow one.
-
-
-To run the corresponding example DAG, perform the following (assuming you have put _whirl_ to your `PATH`)
-
-```bash
-$ cd ./examples/external-airflow-db
-$ whirl
-```
-
-Open your browser to [http://localhost:5000](http://localhost:5000) to see the Airflow UI appear. Manually enable the DAG and see the pipeline get marked success.
-
-The environment to be used is set in the `.whirl.env` in the DAG directory. In the environment folder there is also a `.whirl.env` which specifies Postgres specific variables.
-
-
-#### Testing failure email
-
-In this example the dag is set to fail. This example is all about how to configure airflow to use a external smtp server for sending the failure emails.
-We have created an environment that spins up an smtp server together with the Airflow one.
-
-
-To run the corresponding example DAG, perform the following (assuming you have put _whirl_ to your `PATH`)
-
-```bash
-$ cd ./examples/external-smtp-for-failure-emails
-$ whirl
-```
-
-Open your browser to [http://localhost:5000](http://localhost:5000) to see the Airflow UI appear. Manually enable the DAG and see the pipeline get marked failed.
-Also open your browser at [http://localhost:1080](http://localhost:1080) for the email client where the emails should show up.
-
-The environment to be used is set in the `.whirl.env` in the DAG directory. In the environment folder there is also a `.whirl.env` which specifies specific Airflow configuration variables.
 
 ## References
 

examples/api-to-s3/.airflowignore

@@ -1,2 +1,3 @@
 whirl.setup.d
-.whirl.env
+.whirl.env
+README.md

examples/api-to-s3/README.md

@@ -0,0 +1,50 @@
+#### Rest API to S3 Storage Example
+
+In this example we are going to:
+
+1. Consume a REST API;
+2. Convert the JSON data to Parquet;
+3. Store the result in a S3 bucket.
+
+The default environment (`api-python-s3`) includes containers for:
+
+ - A S3 server;
+ - A MockServer instance
+ - The core Airflow component.
+ 
+The environment contains a setup script in the `whirl.setup.d/` folder:
+
+ - `01_add_connection_api.sh` which:
+
+   -  Adds a S3 connection to Airflow;
+   -  Installs the `awscli` Python libraries and configures them to connect to the S3 server;
+   -  Creates a bucket (with a `/etc/hosts` entry to support the [virtual host style method](https://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html)).
+
+It is also possible to use a more complex environment (`api-python-s3-k8s`) that adds a Kubernetes cluster and the use of Airflows KubernetesExecutor to run this example. This environment is explained in depth in the [environment README](../../envs/api-python-s3-k8s/README.md)
+
+To run this example with the default environment:
+
+```bash
+$ cd ./examples/api-to-s3
+$ whirl
+```
+
+To run this example with the k8s based environment:
+
+```bash
+$ cd ./examples/api-to-s3
+$ whirl -e api-python-s3-k8s
+```
+
+Open your browser to [http://localhost:5000](http://localhost:5000) to access the Airflow UI. Manually enable the DAG and watch the pipeline run to successful completion.
+
+This example includes a `.whirl.env` configuration file in the DAG directory. In the environment folder there is also a `.whirl.env` which specifies S3-specific variables. The example folder also contains a `whirl.setup.d/` directory which contains an initialization script (`01_add_connection_api_and_mockdata.sh`). This script is executed in the container after the environment-specific scripts have run and will:
+
+ - Add a connection to the API endpoint;
+ - Add an [expectation](http://www.mock-server.com/mock_server/creating_expectations.html) for the MockServer to know which response needs to be sent for which requested path;
+ - Install Pandas and PyArrow to support transforming the JSON into a Parquet file;
+ - Create a local directory where the intermediate file is stored before being uploaded to S3.
+
+The DAG contains 2 tasks which both use the PythonOperator to:
+- call the mock api to retrieve the JSON data and convert it to a local parquet file
+- use the S3Hook to copy the local file to the S3 bucket

examples/external-airflow-db/.airflowignore

@@ -1,2 +1,3 @@
 whirl.setup.d
-.whirl.env
+.whirl.env
+README.md

examples/external-airflow-db/README.md

@@ -0,0 +1,18 @@
+#### Having an external database for Airflow
+
+In this example the dag is not the most important part. This example is all about how to configure airflow to use a external database.
+We have created an environment (`external-airflow-db`) that spins up an postgres database server together with the Airflow one.
+
+
+To run the corresponding example DAG, perform the following (assuming you have put _whirl_ to your `PATH`)
+
+```bash
+$ cd ./examples/external-airflow-db
+$ whirl
+```
+
+Open your browser to [http://localhost:5000](http://localhost:5000) to see the Airflow UI appear. Manually enable the DAG and see the pipeline get marked success.
+
+The environment to be used is set in the `.whirl.env` in the DAG directory. In the environment folder there is also a `.whirl.env` which specifies Postgres specific variables.
+
+

examples/external-smtp-for-failure-emails/README.md

@@ -0,0 +1,17 @@
+#### Testing failure email
+
+In this example the dag is set to fail. This example is all about how to configure airflow to use a external smtp server for sending the failure emails.
+We have created an environment that spins up an smtp server together with the Airflow one.
+
+
+To run the corresponding example DAG, perform the following (assuming you have put _whirl_ to your `PATH`)
+
+```bash
+$ cd ./examples/external-smtp-for-failure-emails
+$ whirl
+```
+
+Open your browser to [http://localhost:5000](http://localhost:5000) to see the Airflow UI appear. Manually enable the DAG and see the pipeline get marked failed.
+Also open your browser at [http://localhost:1080](http://localhost:1080) for the email client where the emails should show up.
+
+The environment to be used is set in the `.whirl.env` in the DAG directory. In the environment folder there is also a `.whirl.env` which specifies specific Airflow configuration variables.

examples/localhost-ssh-example/.airflowignore

@@ -1,2 +1,3 @@
 whirl.setup.d
-.whirl.env
+.whirl.env
+README.md

examples/localhost-ssh-example/README.md

@@ -0,0 +1,18 @@
+#### SSH to Localhost
+
+The directory `example/localhost-ssh-example/` contains  the Airflow DAG and the environment to be used is configured inside the `.whirl.env` file. (it uses the `local-ssh` environment)
+
+The local-ssh environment only involves one component, the Apache Airflow docker container itself. The environment contains one preparation script called `01_enable_local_ssh.sh` which makes it possible in that container to SSH to `localhost`. The script also adds a new connection called `ssh_local` to the Airflow connections.
+
+The DAG has 1 task that simply uses the SSHOperator to copy a file. A succesfull run proves that we are able to use a airflow connection to execute commands through an ssh connection.
+
+To run this example:
+
+```bash
+$ cd ./examples/localhost-ssh-example
+# Note: here we pass the whirl environment 'local-ssh' as a command-line argument.
+$ whirl -e local-ssh
+```
+
+Open your browser to [http://localhost:5000](http://localhost:5000) to access the Airflow UI. Manually enable the DAG and watch the pipeline run to successful completion.
+

examples/logging-to-s3/.airflowignore

@@ -1,2 +1,3 @@
 whirl.setup.d
-.whirl.env
+.whirl.env
+README.md

examples/logging-to-s3/README.md

@@ -0,0 +1,27 @@
+#### Remote logging for Airflow
+
+In this example the dag is not the most important part. This example is all about how to configure airflow to log to S3.
+
+The environment to be used (`airflow-s3-logging`) is set in the `.whirl.env` in the DAG directory. In the environment folder there is also a `.whirl.env` which specifies S3 specific variables.
+
+The docker compose of the environment that spins up an S3 server together with the Airflow one. The environment contains a setup script in the `whirl.setup.d` folder:
+
+- `01_add_connection_s3.sh` which:
+    -  adds an S3 connection to Airflow
+    -  Installs awscli Python libraries and configures them to connect to the S3 server
+    -  Creates a bucket (with adding a `/etc/hosts` entry to support the [virtual host style method](https://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html))
+- `02_configue_logging_to_s3.sh` which:
+    -  exports environment varibles which airflow uses to override the default config. For example: `export AIRFLOW__CORE__REMOTE_LOGGING=True`
+
+
+To run the corresponding example DAG, perform the following (assuming you have put _whirl_ to your `PATH`)
+
+```bash
+$ cd ./examples/logging-to-s3
+$ whirl
+```
+
+Open your browser to [http://localhost:5000](http://localhost:5000) to see the Airflow UI appear. Manually enable the DAG and see the pipeline get marked success. If you open one of the logs, the first line shows that the log is retrieved from S3.
+
+
+