The bigIQCollect.sh script must be copied and run on a BIG-IQ Centralized Manager instance. It will collect raw JSON files and package them into a single .tgz file:
the .tgz file can then be processed offline by Second Sight to build all target JSON files
sampledata.tgz is provided for testing purposes
- Copy (scp)
bigIQCollect.shfrom your local host to your BIG-IQ CM instance, under/tmp/
$ scp bigIQCollect.sh root@bigiq.f5:/tmp/
(root@bigiq.f5) Password:
bigIQCollect.sh 100% 1611 789.7KB/s 00:00
$
- SSH to your BIG-IQ CM instance and run the collection script with no parameters to display the help banner
$ ssh root@bigiq.f5
(root@bigiq.f5) Password:
Last login: Fri Nov 19 00:00:05 2021 from 192.168.1.18
[root@bigiq:Active:Standalone] config # chmod +x /tmp/bigIQCollect.sh
[root@bigiq:Active:Standalone] config # /tmp/bigIQCollect.sh
FCP Usage Script - 20250207 - https://github.com/F5Networks/SecondSight
This tool collects usage tracking data from BIG-IQ for offline postprocessing.
=== Usage:
./bigIQCollect.sh [options]
=== Options:
-h - This help
-v - Show version
-i - Interactive mode
-u [username] - BIG-IQ username (batch mode)
-p [password] - BIG-IQ password (batch mode)
-c [customer_name] - Customer name (batch mode)
-s [http(s)://address] - Upload data to Second Sight (optional)
-t [seconds] - BIG-IQ timeout (optional, 90 seconds by default)
-d - Collect data for troubleshooting purposes
=== Examples:
Interactive mode: ./bigIQCollect.sh -i
Interactive mode + upload: ./bigIQCollect.sh -i -s https://<SECOND_SIGHT_FQDN_OR_IP>
Batch mode: ./bigIQCollect.sh -u [username] -p [password] -c [customer_name]
Batch mode: ./bigIQCollect.sh -u [username] -p [password] -c [customer_name] -s https://<SECOND_SIGHT_FQDN_OR_IP>
Collect debug data: ./bigIQCollect.sh -i -d
- On BIG-IQ CM run the collection script using "admin" as the authentication username and its password
$ ssh root@bigiq.f5
(root@bigiq.f5) Password:
Last login: Fri Nov 19 00:00:05 2021 from 192.168.1.18
[root@bigiq:Active:Standalone] config # . /tmp/bigIQCollect.sh -i
Username: admin
Password:
Customer name: Test
-> Authentication successful
-> Reading device list
-> Reading system provisioning
-> Removing old inventories
-> Reading device inventory details
... 0 inventories found: refresh requested
... waiting for inventory refresh, sleeping for 5 seconds
-> Found 1 inventories
-> Inventories summary
1 FINISHED
-> Using inventory [3a892dde-f61d-412c-a243-42fcd13b8945]
-> Reading device info for [1346a1f5-49aa-422f-8c4d-19b0119b3927]
-> Reading device info for [aabcaca7-6986-4d0b-b9fe-b72e3473144e]
-> Reading device telemetry
-> Reading device telemetry datapoints for [bigip1.f5.ff.lan]
-> Collecting utility billing for regkey [XXXXX-XXXXX-XXXXX-XXXXX-XXXXXXX]
-> Collecting utility billing for regkey [XXXXX-XXXXX-XXXXX-XXXXX-XXXXXXX]
-> Data collection completed, building tarfile
-> Adding summary
... Timestamp [20241003-1708]"
... Customer Name [Test]"
... Total devices [2]
-> All done, copy /tmp/Test-20241003-1708-bigIQCollect.tgz to your local host using scp
[root@bigiq:Active:Standalone] config # exit
logout
Connection to bigiq.f5 closed.
$
- Retrieve the tgz file generated by
bigIQCollect.sh
$ scp root@bigiq.f5:/tmp/Test-20241003-1708-bigIQCollect.tgz .
(root@bigiq.f5) Password:
/tmp/Test-20241003-1708-bigIQCollect.tgz 100% 14KB 5.4MB/s 00:00
$
- On BIG-IQ CM run the collection script using "admin" as the authentication username and its password, add the
-sswitch to specify Second Sight GUI URL
$ ssh root@bigiq.f5
(root@bigiq.f5) Password:
Last login: Fri Mar 10 18:30:03 2023 from 192.168.1.18
[root@bigiq:Active:Standalone] tmp # ./bigIQCollect.sh -i -s https://secondsight.acme.lan
Username: admin
Password:
Customer name: Test
-> Reading device list
-> Reading system provisioning
-> Reading device inventory details
-> Found 68 inventories
-> Inventories summary
1 FINISHED
-> Using inventory [3a892dde-f61d-412c-a243-42fcd13b8945]
-> Reading device info for [1346a1f5-49aa-422f-8c4d-19b0119b3927]
-> Reading device info for [aabcaca7-6986-4d0b-b9fe-b72e3473144e]
-> Reading device telemetry
-> Collecting utility billing for regkey [XXXXX-XXXXX-XXXXX-XXXXX-XXXXXXX]
-> Collecting utility billing for regkey [XXXXX-XXXXX-XXXXX-XXXXX-XXXXXXX]
-> Data collection completed, building tarfile
-> Adding summary
... Timestamp [20241003-1708]"
... Customer Name [Test]"
... Total devices [2]
-> Uploading /tmp/Test-20241003-1708-bigIQCollect.tgz to Second Sight at https://secondsight.acme.lan
{"success":true,"status":"File upload completed","filename":"Test-20241003-1708-bigIQCollect.tgz","content-type":"application/octet-stream","description":"Test-20241003-1708-bigIQCollect.tgz","uid":"23b8fdd3-ee3c-4cf3-89ea-5396ca467915"}
-> Upload complete
[root@bigiq:Active:Standalone] config # exit
logout
Connection to bigiq.f5 closed.
$
- Start Second Sight and the additional fileserver script to process the tgz file and generate the target JSON file
$ ./f5ttFsStart.sh 20220113-0005-bigIQCollect.tgz
Using work directory /tmp/SecondSight_5ap6afcs
Decompressing sampledata.tgz
* Serving Flask app 'f5ttfs' (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: off
* Running on all addresses.
WARNING: This is a development server. Do not use it in a production deployment.
* Running on http://192.168.1.26:5001/ (Press CTRL+C to quit)
Basic CVE Tracking - for full tracking get a NIST API key at https://nvd.nist.gov/developers/request-an-api-key
Running BIG-IQ inventory refresh thread
2022-07-10 22:58:34.262641 Requesting BIG-IQ inventory refresh
Running REST API/Prometheus metrics server
127.0.0.1 - - [10/Jul/2022 22:58:34] "POST /mgmt/shared/authn/login HTTP/1.1" 200 -
INFO: Started server process [898181]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:5000 (Press CTRL+C to quit)
- Query Second Sight from another terminal session to get the target JSON file
Uncompressed output:
$ curl http://127.0.0.1:5000/instances
{"instances":[{"bigip":2,"hwTotals":[{"F5-VE":2}],"swTotals":[{"H-VE-AFM":1,"H-VE-AWF":1,"H-VE-LTM":2,"H-VE-APM":1,"H-VE-DNS":1}]}],"details":[{"inventoryTimestamp":1641986071513,"inventoryStatus":"full"[...]
Compressed output:
$ curl -s -H "Accept-Encoding: gzip" http://127.0.0.1:5000/instances --output output-json.gz
- Start Second Sight and the additional fileserver script to process the tgz file and generate the target JSON file
$ ./f5ttFsStart.sh
* Serving Flask app 'f5ttfs' (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: off
* Running on all addresses.
WARNING: This is a development server. Do not use it in a production deployment.
* Running on http://192.168.1.26:5001/ (Press CTRL+C to quit)
Basic CVE Tracking - for full tracking get a NIST API key at https://nvd.nist.gov/developers/request-an-api-key
Running BIG-IQ inventory refresh thread
2022-07-10 23:02:45.320908 Requesting BIG-IQ inventory refresh
Running REST API/Prometheus metrics server
INFO: Started server process [898243]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:5000 (Press CTRL+C to quit)
127.0.0.1 - - [10/Jul/2022 23:02:45] "POST /mgmt/shared/authn/login HTTP/1.1" 200 -
127.0.0.1 - - [10/Jul/2022 23:02:45] "POST /mgmt/cm/device/tasks/device-inventory HTTP/1.1" 405 -
- Load the collected tgz file using the
/uploadREST endpoint (note the5001TCP port):
$ curl -i -F 'file=@20220113-0005-bigIQCollect.tgz' http://127.0.0.1:5001/upload
HTTP/1.1 100 Continue
HTTP/1.0 200 OK
Content-Type: application/json
Content-Length: 40
Server: Werkzeug/2.0.1 Python/3.8.10
Date: Sun, 10 Jul 2022 22:56:40 GMT
{"status":"file uploaded successfully"}
- Query Second Sight from another terminal session to get the target JSON file
Uncompressed output:
$ curl http://127.0.0.1:5000/instances
{"instances":[{"bigip":2,"hwTotals":[{"F5-VE":2}],"swTotals":[{"H-VE-AFM":1,"H-VE-AWF":1,"H-VE-LTM":2,"H-VE-APM":1,"H-VE-DNS":1}]}],"details":[{"inventoryTimestamp":1641986071513,"inventoryStatus":"full"[...]
Compressed output:
$ curl -s -H "Accept-Encoding: gzip" http://127.0.0.1:5000/instances --output output-json.gz