Merge branch 'feature/vault-updates' into 'master'

some updates to vault and postgres and some readme cleanups

See merge request all-staff/hashiqube!117

Author: Riaan Nolan

README.md

@@ -8,6 +8,7 @@ HashiCorp blog post about HashiQube: https://www.hashicorp.com/resources/hashiqu
 HashiQube website: https://servian.github.io/hashiqube <br />
 HashiQube github: https://github.com/servian/hashiqube <br />
 HashiQube youtube: https://www.youtube.com/watch?v=6jGDAGWaFiw
+HashiQube medium: https://medium.com/search?q=hashiqube
 
 ## HashiQube runs all HashiCorp's products 
 ![HashiQube](images/thestack.png?raw=true "HashiQube")
@@ -30,11 +31,13 @@ Thanks to the flexibility of the HashiCorp products there is no need wonder how
 | mac apple | ✓ | ✘ | ✘ |
 
 ## Instructions
-* Vagrant - Please download __Vagrant__ from https://www.vagrantup.com/downloads.html and install
+:bulb: Docker is the Default and preferred way to run Hashiqube
+
 * Docker - Please download __Docker__ from https://www.docker.com/products/docker-desktop and install
+* Vagrant - Please download __Vagrant__ from https://www.vagrantup.com/downloads.html and install
 * Virtualbox (Optional) - Please download __Virtualbox__ from https://www.virtualbox.org/wiki/Downloads and install
 * Using `git` - clone this repo `git clone $repo .` [__What is Git?__](git/#git)
-* Inside the local repo folder, do `vagrant up --provision --provider docker` - This will setup, Vault, Nomad, Consul, Terraform, Localstack and Docker
+* Inside the local repo folder, do `vagrant up --provision` - This will setup, Vault, Nomad, Consul, Terraform, Localstack and Docker
 * Documentation locally available at http://localhost:3333
 
 ## Components
@@ -73,29 +76,7 @@ Please create the following file: __/etc/vbox/networks.conf__ with the following
 * 2001::/64
 ``` 
 
-and re-run `vagrant up --provision --provider docker`
-
-## Dependencies
-To get started we are now going to install some core dependencies to get the Lab started, you need to install
-below dependencies before you can do anything
-
-__Mac Users only, Windows Users can skip this step__
-Let's first check if we have an __M1 Mac__, if that is the case the __virtualbox__ provider will not work.
-
-Click on the Apple Icon top left
-![About this Mac](images/mac_apple_icon.png?raw=true "About this Mac")
-
-and click on __About this Mac__
-
-If you see an __Intel__ chip, you can proceed with the `virtualbox` provider.
-![About this Mac Intel](images/mac_intel.png?raw=true "About this Mac Intel")
-
-If you see an __Apple M1__ chip, please ensure you specify the environment variable and the provider to be docker.
-```
-vagrant plugin uninstall vagrant-hostsupdater # the hostsupdator plugin does not work with the docker provider
-```
-
-![About this Mac M1](images/mac_m1.png?raw=true "About this Mac M1")
+and re-run `vagrant up --provision`
 
 #### Docker Desktop
 Docker Desktop is an easy-to-install application for your Mac or Windows environment that enables you to build and share containerized applications and microservices. It's a graphical user interface for the docker service.
@@ -115,7 +96,7 @@ Now that docker has been installed we need to ensure that your docker environmen
 
 ![Docker Desktop Resources](images/docker_installed_resources.png?raw=true "Docker Desktop Resources")
 
-* Please ensure that you give your docker daemon at least __12G of RAM__ and sufficient disk space
+* Please ensure that you give your docker daemon at least __8G of RAM__ and sufficient disk space
 
 ## Consul DNS
 __Local DNS via Consul__ <br />
@@ -128,11 +109,11 @@ Now you can use DNS like nomad.service.consul:9999 vault.service.consul:9999 via
 
 ## Pre-requisites
 * 10GB of disk space
-* 4GB RAM
+* 8GB RAM
 * Admin rights / sudo (you will be asked to update ETC Host file)
 * Virtualbox
 * Vagrant
-* `vagrant up --provision --provider docker`
+* `vagrant up --provision`
 
 ## Additional Information
 * [__Multi Cloud__](multi-cloud/#terraform-hashicorp-hashiqube) - Hashiqube on AWS, GCP and Azure (Clustered) https://registry.terraform.io/modules/star3am/hashiqube/hashicorp/latest
@@ -166,7 +147,7 @@ For Documentation please open http://localhost:3333 in your browser
 * Vault http://localhost:8200
 * Nomad http://localhost:4646
 * Consul http://localhost:8500
-* Localstack http://localhost:8080
+* Docsify http://localhost:3333
 
 ## HashiQube runs all HashiCorp's products
 ![HashiQube](images/hashicorp_products.png?raw=true "HashiQube")
@@ -183,7 +164,7 @@ For Documentation please open http://localhost:3333 in your browser
 * Fabio http://localhost:9999
 
 ### Vagrant Basic Usage
-* vagrant up --provision OR vagrant up --provision-with bootstrap|nomad|consul|vault|docker|ldap --provider docker
+* vagrant up --provision OR vagrant up --provision-with bootstrap|nomad|consul|vault|docker|ldap
 * vagrant global-status # to see which VMs are active
 * vagrant global-status --prune # to remove stale VMs from Vagrant cache
 * vagrant status # vagrant status
@@ -255,6 +236,7 @@ __Solution__ Ensure the following contents are present in `/etc/vbox/networks.co
 For suggestions, feedback and queries please branch or and submit a Pull Request or directly contact the architects of the HashiQube via email:
 
 Lead Automation Architect [riaan.nolan@servian.com](mailto:riaan.nolan@servian.com)
+https://www.linkedin.com/in/riaannolan/
 
 ## Contributors and Special mentions
 A Very special mention to HashiQube's contributors, Thank You All for your help, suggestions and contributions no matter how small <3
@@ -264,6 +246,8 @@ A Very special mention to HashiQube's contributors, Thank You All for your help,
  - Ringo Chan
  - Ehsan Mirzaei
  - Greg Luxford
+ - Byron Tuckett
+ - Lane Birmingham
 
 ## Videos
 Videos were made with asciinema https://asciinema.org/

database/postgresql.sh

@@ -1,45 +1,104 @@
 #!/bin/bash
 # https://hub.docker.com/_/postgres
-# https://www.vaultproject.io/docs/secrets/databases/postgresql 
+# https://www.vaultproject.io/docs/secrets/databases/postgresql
+
+echo -e '\e[38;5;198m'"++++ "
+echo -e '\e[38;5;198m'"++++ Ensure postgres docker container is running"
+echo -e '\e[38;5;198m'"++++ "
 sudo docker stop postgres
 sudo docker rm postgres
 yes | sudo docker system prune -a
 yes | sudo docker system prune --volumes
 sudo docker run --name postgres -e POSTGRES_USER=root \
          -e POSTGRES_PASSWORD=rootpassword \
          -d -p 5432:5432 postgres
+
 sleep 15;
 
-// Vagrant install as a dependency
-bash /vagrant/hashicorp/vault.sh
+echo -e '\e[38;5;198m'"++++ "
+echo -e '\e[38;5;198m'"++++ Ensure postgresql-client is installed"
+echo -e '\e[38;5;198m'"++++ "
+sudo apt-get install -y postgresql-client libpq-dev python3.9-dev
 
+if pgrep -x "vault" >/dev/null
+then
+  echo -e '\e[38;5;198m'"++++ "
+  echo -e '\e[38;5;198m'"++++ Vault is running"
+  echo -e '\e[38;5;198m'"++++ "
+  echo -e '\e[38;5;198m'""
+  echo -e '\e[38;5;198m'"++++ "
+  echo -e '\e[38;5;198m'"++++ Vault status"
+  echo -e '\e[38;5;198m'"++++ "
+  vault status
+else
+  echo -e '\e[38;5;198m'"++++ "
+  echo -e '\e[38;5;198m'"++++ Ensure Vault is running.."
+  echo -e '\e[38;5;198m'"++++ "
+  sudo bash /vagrant/hashicorp/vault.sh
+  echo -e '\e[38;5;198m'"++++ "
+  echo -e '\e[38;5;198m'"++++ Vault status"
+  echo -e '\e[38;5;198m'"++++ "
+  vault status
+fi
+
+echo -e '\e[38;5;198m'"++++ "
 echo -e '\e[38;5;198m'"++++ Show users in database"
+echo -e '\e[38;5;198m'"++++ "
 sudo docker exec postgres psql -U root -c '\du'
+
+sleep 15;
+
+echo -e '\e[38;5;198m'"++++ "
+echo -e '\e[38;5;198m'"++++ Vault token lookup"
+echo -e '\e[38;5;198m'"++++ "
+vault token lookup
+
+echo -e '\e[38;5;198m'"++++ "
 echo -e '\e[38;5;198m'"++++ Enable Vault Database PostgreSQL secret engine"
+echo -e '\e[38;5;198m'"++++ "
 vault secrets enable database
+
+echo -e '\e[38;5;198m'"++++ "
 echo -e '\e[38;5;198m'"++++ Configure PostgreSQL "
+echo -e '\e[38;5;198m'"++++ "
 vault write database/config/postgresql \
     plugin_name=postgresql-database-plugin \
     allowed_roles=postgresql-role \
-    connection_url='postgresql://root:rootpassword@localhost:5432/postgres?sslmode=disable'   
+    connection_url='postgresql://root:rootpassword@localhost:5432/postgres?sslmode=disable'
+
+echo -e '\e[38;5;198m'"++++ "
 echo -e '\e[38;5;198m'"++++ Create a role"
+echo -e '\e[38;5;198m'"++++ "
 vault write database/roles/postgresql-role db_name=postgresql \
         creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
                              GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
-        default_ttl=1h max_ttl=24h     
+        default_ttl=1h max_ttl=24h
+
+echo -e '\e[38;5;198m'"++++ "  
 echo -e '\e[38;5;198m'"++++ Create policy"
+echo -e '\e[38;5;198m'"++++ "
 vault policy write apps  -<<EOF
 # Get credentials from the database secrets engine
 path "database/creds/postgresql-role" {
   capabilities = [ "read" ]
 }
 EOF
-echo -e '\e[38;5;198m'"++++ Create new token"
-# Create a new token with apps policy attached
-# vault token create -policy="apps"
+
+echo -e '\e[38;5;198m'"++++ "
+echo -e '\e[38;5;198m'"++++ Create a new token with apps policy attached"
+echo -e '\e[38;5;198m'"++++ "
 VAULT_TOKEN_APPS=$(vault token create -policy="apps" -field token)
+
+echo -e '\e[38;5;198m'"++++ "
 echo -e '\e[38;5;198m'"++++ New Token: $VAULT_TOKEN_APPS"
+echo -e '\e[38;5;198m'"++++ "
+
+echo -e '\e[38;5;198m'"++++ "
 echo -e '\e[38;5;198m'"++++ Create new connection with token"
+echo -e '\e[38;5;198m'"++++ "
 VAULT_TOKEN=$VAULT_TOKEN_APPS vault read database/creds/postgresql-role
+
+echo -e '\e[38;5;198m'"++++ "
 echo -e '\e[38;5;198m'"++++ Now show users in database again with new user created"
+echo -e '\e[38;5;198m'"++++ "
 sudo docker exec postgres psql -U root -c '\du'

hashicorp/vault.sh

@@ -30,15 +30,19 @@ echo -e '\e[38;5;198m'"CPU is $ARCH"
 sudo DEBIAN_FRONTEND=noninteractive apt-get --assume-yes install curl unzip jq
 # only do if vault is not found
 if [ ! -f /usr/local/bin/vault ]; then
-
+  
+  echo -e '\e[38;5;198m'"++++ "
   echo -e '\e[38;5;198m'"++++ Vault not installed, installing.."
+  echo -e '\e[38;5;198m'"++++ "
 
   LATEST_URL=$(curl -sL https://releases.hashicorp.com/vault/index.json | jq -r '.versions[].builds[].url' | sort -t. -k 1,1n -k 2,2n -k 3,3n -k 4,4n | egrep -v 'rc|ent|beta' | egrep "linux.*$ARCH" | sort -V | tail -n 1)
   wget -q $LATEST_URL -O /tmp/vault.zip
 
   mkdir -p /usr/local/bin
   (cd /usr/local/bin && unzip /tmp/vault.zip)
+  echo -e '\e[38;5;198m'"++++ "
   echo -e '\e[38;5;198m'"++++ Installed `/usr/local/bin/vault --version`"
+  echo -e '\e[38;5;198m'"++++ "
 
   # enable command autocompletion
   vault -autocomplete-install
@@ -130,8 +134,11 @@ EOF
   sleep 20
   vault operator init > /etc/vault/init.file
 
+  echo -e '\e[38;5;198m'"++++ "
   echo -e '\e[38;5;198m'"++++ Vault http://localhost:8200/ui and enter the following codes displayed below"
+  echo -e '\e[38;5;198m'"++++ "
   echo -e '\e[38;5;198m'"++++ Auto unseal vault"
+  echo -e '\e[38;5;198m'"++++ "
   for i in $(cat /etc/vault/init.file | grep Unseal | cut -d " " -f4 | head -n 3); do vault operator unseal $i; done
   vault status
   cat /etc/vault/init.file
@@ -164,11 +171,15 @@ else
   else
     sed -i "s%VAULT_ADDR=.*%VAULT_ADDR=http://127.0.0.1:8200%g" /etc/environment
   fi
+  echo -e '\e[38;5;198m'"++++ "
   echo -e '\e[38;5;198m'"++++ Vault already installed and running"
   echo -e '\e[38;5;198m'"++++ Vault http://localhost:8200/ui and enter the following codes displayed below"
+  echo -e '\e[38;5;198m'"++++ "
   # check vault status
   # vault status
+  echo -e '\e[38;5;198m'"++++ "
   echo -e '\e[38;5;198m'"++++ Auto unseal vault"
+  echo -e '\e[38;5;198m'"++++ "
   for i in `cat /etc/vault/init.file | grep Unseal | cut -d " " -f4 | head -n 3`; do vault operator unseal $i; done
   vault status
   cat /etc/vault/init.file