error-handle tracebacks from main function

[durd]

when bulkdl cant access an ISE node it crashes with a traceback that is logged to /var/log/syslog. This was just before a patch upgrade. ISE primary had failed a backup, the backup had been reset. Not sure by button or by reboot of node.

Perhaps place the traceback in our syslog? At least try to figure out why it would have crashed. We hadn't started upgrading yet.

May  8 08:05:00 host systemd[1]: Starting pxGrid <-> Checkpoint service that downloads last X hours of DB-changes...
May  8 08:05:01 host CRON[36198]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
May  8 08:05:32 host python3[36196]: Traceback (most recent call last):
May  8 08:05:32 host python3[36196]:   File "/usr/lib/python3.7/urllib/request.py", line 1317, in do_open
May  8 08:05:32 host python3[36196]:     encode_chunked=req.has_header('Transfer-encoding'))
May  8 08:05:32 host python3[36196]:   File "/usr/lib/python3.7/http/client.py", line 1244, in request
May  8 08:05:32 host python3[36196]:     self._send_request(method, url, body, headers, encode_chunked)
May  8 08:05:32 host python3[36196]:   File "/usr/lib/python3.7/http/client.py", line 1290, in _send_request
May  8 08:05:32 host python3[36196]:     self.endheaders(body, encode_chunked=encode_chunked)
May  8 08:05:32 host python3[36196]:   File "/usr/lib/python3.7/http/client.py", line 1239, in endheaders
May  8 08:05:32 host python3[36196]:     self._send_output(message_body, encode_chunked=encode_chunked)
May  8 08:05:32 host python3[36196]:   File "/usr/lib/python3.7/http/client.py", line 1026, in _send_output
May  8 08:05:32 host python3[36196]:     self.send(msg)
May  8 08:05:32 host python3[36196]:   File "/usr/lib/python3.7/http/client.py", line 966, in send
May  8 08:05:32 host python3[36196]:     self.connect()
May  8 08:05:32 host python3[36196]:   File "/usr/lib/python3.7/http/client.py", line 1399, in connect
May  8 08:05:32 host python3[36196]:     super().connect()
May  8 08:05:32 host python3[36196]:   File "/usr/lib/python3.7/http/client.py", line 938, in connect
May  8 08:05:32 host python3[36196]:     (self.host,self.port), self.timeout, self.source_address)
May  8 08:05:32 host python3[36196]:   File "/usr/lib/python3.7/socket.py", line 727, in create_connection
May  8 08:05:32 host python3[36196]:     raise err
May  8 08:05:32 host python3[36196]:   File "/usr/lib/python3.7/socket.py", line 716, in create_connection
May  8 08:05:32 host python3[36196]:     sock.connect(sa)
May  8 08:05:32 host python3[36196]: OSError: [Errno 101] Network is unreachable
May  8 08:05:32 host python3[36196]: During handling of the above exception, another exception occurred:
May  8 08:05:32 host python3[36196]: Traceback (most recent call last):
May  8 08:05:32 host python3[36196]:   File "/usr/local/cp-pxgrid/session_query_all.cp.py", line 175, in <module>
May  8 08:05:32 host python3[36196]:     while pxgrid.account_activate()['accountState'] != 'ENABLED':
May  8 08:05:32 host python3[36196]:   File "/usr/local/cp-pxgrid/pxgrid.py", line 38, in account_activate
May  8 08:05:32 host python3[36196]:     return self.send_rest_request('AccountActivate', payload)
May  8 08:05:32 host python3[36196]:   File "/usr/local/cp-pxgrid/pxgrid.py", line 29, in send_rest_request
May  8 08:05:32 host python3[36196]:     rest_response = opener.open(rest_request)
May  8 08:05:32 host python3[36196]:   File "/usr/lib/python3.7/urllib/request.py", line 525, in open
May  8 08:05:32 host python3[36196]:     response = self._open(req, data)
May  8 08:05:32 host python3[36196]:   File "/usr/lib/python3.7/urllib/request.py", line 543, in _open
May  8 08:05:32 host python3[36196]:     '_open', req)
May  8 08:05:32 host python3[36196]:   File "/usr/lib/python3.7/urllib/request.py", line 503, in _call_chain
May  8 08:05:32 host python3[36196]:     result = func(*args)
May  8 08:05:32 host python3[36196]:   File "/usr/lib/python3.7/urllib/request.py", line 1360, in https_open
May  8 08:05:32 host python3[36196]:     context=self._context, check_hostname=self._check_hostname)
May  8 08:05:32 host python3[36196]:   File "/usr/lib/python3.7/urllib/request.py", line 1319, in do_open
May  8 08:05:32 host python3[36196]:     raise URLError(err)
May  8 08:05:32 host python3[36196]: urllib.error.URLError: <urlopen error [Errno 101] Network is unreachable>
May  8 08:05:32 host systemd[1]: cp-pxgrid-bulkdl.service: Main process exited, code=exited, status=1/FAILURE
May  8 08:05:32 host systemd[1]: cp-pxgrid-bulkdl.service: Failed with result 'exit-code'.
May  8 08:05:32 host systemd[1]: Failed to start pxGrid <-> Checkpoint service that downloads last X hours of DB-changes.

[durd]

Odd, we were having trouble talking to eachother just after 8am, I saw him start the upgrade after I called him at 08:11...