Upgrade all (#295)

Multiple changes:

* Migrate marathon for tar (deb are missing)
* add users -> start fabio/consul/marathon as separate users
* fix examples
* Upgrade supervisord
* Workaround for bug in new supervisord

.travis.yml

@@ -41,10 +41,10 @@ script:
 - curl -s http://${IP}:8080/v2/info
 - curl -I -s http://${IP}:5050/master/health
 - curl -s http://${IP}:5050/system/stats.json
-- docker exec -ti panteras_panteras_1 supervisorctl status
+- docker exec -ti panteras_panteras_1 supervisorctl status || true
 - cd examples/SimpleWebappPython
 - "./test_service.sh"
 - cd -
 - cd examples/SmoothWebappPython
 - "./test_service.sh"
-- docker exec -ti panteras_panteras_1 supervisorctl status
+- docker exec -ti panteras_panteras_1 supervisorctl status || true

README.md

@@ -1,6 +1,6 @@
 [![Build Status](https://travis-ci.org/eBayClassifiedsGroup/PanteraS.svg?branch=master)](https://travis-ci.org/eBayClassifiedsGroup/PanteraS)
 [![Docker Hub](https://img.shields.io/badge/docker-ready-blue.svg)](https://hub.docker.com/r/panteras/paas-in-a-box/)
-[![Current Release](https://img.shields.io/badge/release-0.3.9-blue.svg)](https://github.com/eBayClassifiedsGroup/PanteraS/releases/tag/v0.3.9)
+[![Current Release](https://img.shields.io/badge/release-0.4.0-blue.svg)](https://github.com/eBayClassifiedsGroup/PanteraS/releases/tag/v0.4.0)
 
 # PanteraS <br> _entire_ Platform as a Service, in a box
 _"One container to rule them all"_
@@ -16,10 +16,9 @@ _"You shall ~~not~~ PaaS"_
 ## Architecture
 
 ### Components
-- Mesos + Marathon + ZooKeeper + Chronos (orchestration components)
-- Consul (K/V store, monitoring, service directory and registry)  + Registrator (automating register/ deregister)
-- Fabio or  
-  HAproxy + consul-template (load balancer with dynamic config generation)
+- Mesos + Marathon + ZooKeeper (orchestration components)
+- Consul (K/V store, monitoring, service directory and registry) + Registrator (automating register/ deregister)
+- Fabio (load balancer with dynamic config generation)
 
 ![PanteraS Architecture](http://s3.amazonaws.com/easel.ly/all_easels/19186/panteras/image.jpg#)
 
@@ -54,13 +53,11 @@ Depending on `MASTER` and `SLAVE` you can define role of the container
    Mesos Master   | x | x | -
    Marathon       | x | x | -
    Zookeeper      | x | x | -
-   Chronos        | x | x | -
    Consul-template| x | - | x
-   Haproxy        | x | - | x
    Mesos Slave    | x | - | x
    Registrator    | x | - | x
-   dnsmasq        | x | x | x
-   Fabio          | - | - | -
+   Fabio          | x | - | x
+   Dnsmasq        | - | - | -
    Netdata        | - | - | -
 
 Last two require manual override like `START_FABIO=true`
@@ -74,7 +71,7 @@ Enabling `Fabio` require stop the other concurent service `START_CONSUL_TEMPLATE
 ## Usage:
 Clone it
 ```
-git clone -b 0.3.8 https://github.com/eBayClassifiedsGroup/PanteraS.git
+git clone -b 0.4.0 https://github.com/eBayClassifiedsGroup/PanteraS.git
 cd PanteraS
 ```
 #### Default: Stand alone mode
@@ -129,9 +126,8 @@ slavehost-n# docker-compose up -d
 You can reach the PaaS components
 on the following ports:
 
-- HAproxy / Fabio: http://hostname:81
+- Fabio: http://hostname:81
 - Consul: http://hostname:8500
-- Chronos: http://hostname:4400
 - Marathon: http://hostname:8080
 - Mesos: http://hostname:5050
 - Supervisord: http://hostname:9000
@@ -144,7 +140,7 @@ which might be dangerous if you want to expose the PaaS.
 Use ENV `LISTEN_IP` if you want to listen on specific IP address.  
 for example:  
 `echo LISTEN_IP=192.168.10.10 >> restricted/host`  
-This might not work for all services like Marathon or Chronos that has some additional random ports.
+This might not work for all services like Marathon that has some additional random ports.
 
 ## Services Accessibility
 
@@ -164,24 +160,16 @@ If you have direct access to the docker host DNS,
 then just modify your /etc/resolv.conf adding its IP address.
 
 If you do NOT have direct access to docker host DNS,
-then you have two options:
-
-A. use OpenVPN client
-an example server we have created for you (in optional),
-but you need to provide certificates and config file,
-it might be little bit complex for the beginners,
-so you might to try second option first.
-
-B. SSHuttle - use https://github.com/apenwarr/sshuttle project so you can tunnel DNS traffic over ssh
-but you have to have ssh daemon running in some container.
+you can use [SSHuttle](https://github.com/apenwarr/sshuttle) project  
+so you can tunnel DNS traffic over ssh
 
 ## Running an example application
 
 There are two examples available:  
 `SimpleWebappPython` - basic example - spawn 2x2 containers  
 `SmoothWebappPython` - similar to previous one, but with smooth scaling down  
 
-HAproxy will balance the ports which where mapped and assigned by marathon. 
+Fabio will balance the ports which where mapped and assigned by marathon. 
 
 For non human access like services intercommunication, you can use direct access 
 using DNS consul SRV abilities, to verify answers:
@@ -196,44 +184,6 @@ or ask consul DNS directly:
 $ dig @$CONSUL_IP -p8600  python.service.consul +tcp SRV
 ```
 
-Remember to disable DNS caching in your future services.
-
-## Put service into HAproxy HTTP load-balancer
-
-In order to put a service `my_service` into the `HTTP` load-balancer (`HAproxy`), you need to add a `consul` tag `haproxy` 
-(ENV `SERVICE_TAGS="haproxy"`) to the JSON deployment plan for `my_service` (see examples). `my_service` is then accessible
-on port `80` via `my_service.service.consul:80` and/or `my_service.service.<my_dc>.consul:80`.
-
-If you provide an additional environment variable `HAPROXY_ADD_DOMAIN` during the configuration phase you can access the
-service with that domain appended to the service name as well, e.g., with `HAPROXY_ADD_DOMAIN=".my.own.domain.com"` you
-can access the service `my_service` via `my_service.my.own.domain.com:80` (if the IP address returned by a DNS query for
-`*.my.own.domain.com` is pointing to one of the nodes running an `HAProxy` instance).
-
-You can also provide the additional `consul` tag `haproxy_route` with a corresponding value in order to dispatch the
-service based on the beginning of the `URL`; e.g., if you add the additional tag `haproxy_route=/minions` to the service
-definition for service `gru`, all `HTTP` requests against any of the cluster nodes on port `80` starting with `/minions/`
-will be re-routed to and load-balanced for the service `gru` (e.g., `http://cluster_node.my_company.com/minions/say/banana`).
-Note that no `URL` rewrite happens, so the service gets the full `URL` (`/minions/say/banana`) passed in.
-
-## Put service into HAproxy TCP load-balancer
-
-In order to put a service `my_service` into the `TCP` load-balancer (`HAproxy`), you need to add a `consul` tag `haproxy_tcp` specifying
-the specific `<port>` (ENV `SERVICE_TAGS="haproxy_tcp=<port>"`) to the JSON deployment plan for `my_service`. It is also recommended
-to set the same `<port>` as the `servicePort` in the `docker` part of the JSON deployment plan. `my_service` is then accessible on
-the specific `<port>` on all cluster nodes, e.g., `my_service.service.consul:<port>` and/or `my_service.service.<my_dc>.consul:<port>`.
-
-## Create A/B test services (AKA canaries services)
-
-1. You need to create services with the same consul name (ENV `SERVICE_NAME="consul_service"`), but different marathon `id` in every JSON deployment plan (see examples)
-2. You need to set different [weights](http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#weight) for those services. You can propagate weight value using consul tag  
-(ENV `SERVICE_TAGS="haproxy,haproxy_weight=1"`)
-3. We set the default weight value for `100` (max is `256`).
-
-## Add http health checks to HAproxy
-
-Use tag haproxy_httpchk (`SERVICE_TAGS="haproxy,haproxy_httpchk=GET /"`). You can also specify more complex tag like
-`SERVICE_TAGS="haproxy,haproxy_httpchk=GET /check HTTP/1.0\\r\\nHost:\\ www.domain.com"`
-but keep in mind to espace special characters
 
 ## Deploy using marathon_deploy
 
@@ -246,37 +196,6 @@ set with `ENV` variables, specified with `%%MACROS%%` in deployment plan.
 
 more info: https://github.com/eBayClassifiedsGroup/marathon_deploy
 
-## Enabling SSL on HAProxy
-
-By default, HAProxy will proxy all of your HTTP services via port `80`.  If you would like to enable ssl on HAProxy and proxy all of your HTTP services on port `443`, set the following `ENV` variable before running `generate_yml.sh`:
-
-`HAPROXY_SSL=true`
-
-By default, HA proxy will use a default certificate called `haproxy.pem` in the [infrastructure](/infrastructure) folder.  You can extract the cert public from that `pem` file to import into your other reverse proxies.  
-
-If you would like to use your own cert, create a new `pem` by running:
-
-```
-openssl genrsa -out haproxy.key 2048
-openssl req -new -key haproxy.key 2048 -out haproxy.csr
-```
-...complete the CSR details and then get it signed by a trusted CA, or sign it yourself:
-
-```
-openssl x509 -req -days 9999 -in haproxy.csr -signkey haproxy.key -out haproxy.crt
-```
-
-Create the pem:
-
-```cat haproxy.crt haproxy.key | tee haproxy.pem```
-
-Replace the `haproxy.pem` in the infrastructure folder before you build the PanteraS image.
-
-Alternatively, you could map your new `pem` to the container by adding this to `docker-compose.yml`
-
-```- "/path/to/your/haproxy.pem:/etc/haproxy/haproxy.pem" ```
-
-**Note**: Currently HAProxy supports `http` or `https`, but not both.  
 
 ## References
 

docker-compose.yml.tpl

@@ -9,7 +9,6 @@ panteras:
      ${CONSUL_UI_PORTS} 
      ${MARATHON_PORTS}
      ${MESOS_PORTS}
-     ${CHRONOS_PORTS}
      ${NETDATA_PORTS}
 
   environment:
@@ -21,7 +20,7 @@ panteras:
 
     SERVICE_81_NAME: fabio-ui
     SERVICE_81_TAGS: paas-fabio.ui.service.consul/
-    SERVICE_81_CHECK_HTTP: /v1/status/leader
+    SERVICE_81_CHECK_HTTP: /routes
 
     SERVICE_8500_NAME: consul-ui
     SERVICE_8500_TAGS: paas-consul.ui.service.consul/
@@ -35,10 +34,6 @@ panteras:
     SERVICE_5050_TAGS: paas-mesos.service.consul/
     SERVICE_5050_CHECK_HTTP: /master/health
 
-    SERVICE_4400_NAME: chronos
-    SERVICE_4400_TAGS: paas-chronos.service.consul/
-    SERVICE_4400_CHECK_HTTP: /ping
-
     SERVICE_19999_NAME: netdata
     SERVICE_19999_TAGS: paas-netdata.service.consul/
     SERVICE_19999_CHECK_HTTP: /version.txt
@@ -50,7 +45,6 @@ panteras:
     START_MESOS_SLAVE:       "${START_MESOS_SLAVE}"
     START_REGISTRATOR:       "${START_REGISTRATOR}"
     START_ZOOKEEPER:         "${START_ZOOKEEPER}"
-    START_CHRONOS:           "${START_CHRONOS}"
     START_FABIO:             "${START_FABIO}"
     START_NETDATA:           "${START_NETDATA}"
 
@@ -65,8 +59,6 @@ panteras:
     ZOOKEEPER_APP_PARAMS:       "${ZOOKEEPER_APP_PARAMS}"
     ZOOKEEPER_HOSTS:            "${ZOOKEEPER_HOSTS}"
     ZOOKEEPER_ID:               "${ZOOKEEPER_ID}"
-    CHRONOS_APP_PARAMS:         "${CHRONOS_APP_PARAMS}"
-    JAVA_OPTS:                  "${CHRONOS_JAVA_OPTS}"
     FABIO_APP_PARAMS:           "${FABIO_APP_PARAMS}"
     NETDATA_APP_PARAMS:         "${NETDATA_APP_PARAMS}"
 

examples/SimpleWebappPython/deploy0_marathon.json

@@ -24,7 +24,7 @@
    {
      "path": "/",
      "portIndex": 0,
-     "protocol": "HTTP",
+     "protocol": "MESOS_HTTP",
      "gracePeriodSeconds": 30,
      "intervalSeconds": 10,
      "timeoutSeconds": 30,

examples/SmoothWebappPython/deploy0_marathon.json

@@ -19,14 +19,14 @@
   "env": {
     "SERVICE_TAGS" : "paas-python-smooth.service.consul/",
     "SERVICE_NAME" : "python-smooth",
-    "SERVICE_8000_CHECK_HTTP" : "/index.html"
+    "SERVICE_8000_CHECK_HTTP" : "/cgi-bin/index"
   },
   "healthChecks": [
     {
       "gracePeriodSeconds": 30,
       "intervalSeconds": 10,
       "maxConsecutiveFailures": 3,
-      "path": "/index.html",
+      "path": "/cgi-bin/index",
       "portIndex": 0,
       "protocol": "MESOS_HTTP",
       "timeoutSeconds": 30

examples/SmoothWebappPython/deploy1_marathon.json

@@ -19,7 +19,7 @@
   "env": {
     "SERVICE_TAGS" : "paas-python-smooth.service.consul/",
     "SERVICE_NAME" : "python-smooth",
-    "SERVICE_8000_CHECK_HTTP" : "/index.html"
+    "SERVICE_8000_CHECK_HTTP" : "/cgi-bin/index"
   },
   "healthChecks": [
     {
@@ -34,7 +34,7 @@
       "gracePeriodSeconds": 30,
       "intervalSeconds": 10,
       "maxConsecutiveFailures": 3,
-      "path": "/index.html",
+      "path": "/cgi-bin/index",
       "portIndex": 0,
       "protocol": "MESOS_HTTP",
       "timeoutSeconds": 30

generate_yml.sh

@@ -16,7 +16,7 @@ touch ./restricted/env
 [ -f ./restricted/overwrite ] && . ./restricted/overwrite
 
 echo "Keep in mind, to set free these ports on DOCKER HOST:"
-echo "53, 80, 81, 2181, 2888, 3888, 4400, 5050, 5151, 8080, 8300 - 8302, 8400, 8500, 8600, 9000, 31000 - 32000"
+echo "53, 80, 81, 2181, 2888, 3888, 5050, 5151, 8080, 8300 - 8302, 8400, 8500, 8600, 9000, 31000 - 32000"
 echo "and be sure that your hostname is resolvable, if not, configure dns in /etc/resolv.conf or add entry in /etc/hosts"
 
 # Try to detect IP
@@ -49,7 +49,6 @@ START_FABIO=${START_FABIO:-"true"}
 START_MESOS_MASTER=${START_MESOS_MASTER:-${MASTER}}
 START_MARATHON=${START_MARATHON:-${MASTER}}
 START_ZOOKEEPER=${START_ZOOKEEPER:-${MASTER}}
-START_CHRONOS=${START_CHRONOS:-${MASTER}}
 
 #SLAVE
 START_CONSUL_TEMPLATE=${START_CONSUL_TEMPLATE:-${SLAVE}}
@@ -87,9 +86,8 @@ FQDN=${FQDN:-${HOSTNAME}}
 
 # Memory settings
 ZOOKEEPER_JAVA_OPTS=${ZOOKEEPER_JAVA_OPTS:-"-Xmx512m"}
-CHRONOS_JAVA_OPTS=${CHRONOS_JAVA_OPTS:-"-Xmx512m"}
 
-# Disable dnsmasq address re-mapping on non slaves - no HAProxy there
+# Disable dnsmasq address re-mapping on non slaves
 [ "${SLAVE}" == "false" ] && DNSMASQ_ADDRESS=${DNSMASQ_ADDRESS:-' '}
 # dnsmaq cannot be set to listen on 0.0.0.0 - it causes lot of issues
 # and by default it works on all addresses
@@ -101,7 +99,6 @@ DNSMASQ_ADDRESS=${DNSMASQ_ADDRESS:-"--address=/consul/${CONSUL_IP}"}
 [ "${START_CONSUL}"        == "true" ] && PORTS="ports:" && CONSUL_UI_PORTS='- "8500:8500"'
 [ "${START_MARATHON}"      == "true" ] && PORTS="ports:" && MARATHON_PORTS='- "8080:8080"'
 [ "${START_MESOS_MASTER}"  == "true" ] && PORTS="ports:" && MESOS_PORTS='- "5050:5050"'
-[ "${START_CHRONOS}"       == "true" ] && PORTS="ports:" && CHRONOS_PORTS='- "4400:4400"'
 [ "${START_NETDATA}"       == "true" ] && PORTS="ports:" && NETDATA_PORTS='- "19999:19999"'
 
 # Override docker with local binary
@@ -165,12 +162,6 @@ REGISTRATOR_PARAMS="-cleanup -ip=${HOST_IP} consul://${CONSUL_IP}:8500 \
 #
 ZOOKEEPER_PARAMS="start-foreground"
 #
-CHRONOS_PARAMS="--master zk://${ZOOKEEPER_HOSTS}/mesos \
- --zk_hosts ${ZOOKEEPER_HOSTS} \
- --http_address ${LISTEN_IP} \
- --http_port 4400 \
- ${CHRONOS_PARAMS}"
-#
 FABIO_PARAMS="-cfg ./fabio.properties"
 #
 NETDATA_PARAMS="-nd -ch /host"
@@ -182,7 +173,6 @@ MESOS_MASTER_APP_PARAMS=${MESOS_MASTER_APP_PARAMS:-$MESOS_MASTER_PARAMS}
 MESOS_SLAVE_APP_PARAMS=${MESOS_SLAVE_APP_PARAMS:-$MESOS_SLAVE_PARAMS}
 REGISTRATOR_APP_PARAMS=${REGISTRATOR_APP_PARAMS:-$REGISTRATOR_PARAMS}
 ZOOKEEPER_APP_PARAMS=${ZOOKEEPER_APP_PARAMS:-$ZOOKEEPER_PARAMS}
-CHRONOS_APP_PARAMS=${CHRONOS_APP_PARAMS:-$CHRONOS_PARAMS}
 FABIO_APP_PARAMS=${FABIO_APP_PARAMS:-$FABIO_PARAMS}
 NETDATA_APP_PARAMS=${NETDATA_APP_PARAMS:-$NETDATA_PARAMS}