Skip to content

Commit

Permalink
Merge pull request #178 from sparkfun/develop
Browse files Browse the repository at this point in the history 
cleaned up documentation a bit - prep for release
  • Loading branch information
@gigapod
gigapod authored Nov 14, 2024
2 parents 424cfab + 355b23f commit 77de1ad
Show file tree
Hide file tree
Showing 2 changed files with 134 additions and 92 deletions.
31 changes: 22 additions & 9 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,20 +1,33 @@

# SparkFun flux-sdk

SDK and Framework to simplify the development of embedded applications
The SparkFun flux-sdk is an embedded framework focused on the rapid development of fully featured embedded applications. Build around the concept of adding high-level "capabilities" to an application with minimal development effort, the sdk provides a wide-variety of high-level objects that automatically fit together to form a cohesive and rapidly developed application.

Key functionality provided by the framework:

* Automatic I2C (qwiic) device discovery, initialization and use.
* Automatic serial driven menu configuration system based on the application definition
* Automatic system persistance - local and SD card file
* Multi-format Data logging support - serial and SD card files
* Support for a wide range of IoT device services (AWS IoT, Azure IoT, Arduino IoT, Thingspeak, MQTT,..)
* Extensive I2C (qwiic) device support
* Easy to extend and customize development

For further details, a detailed documentation for flux is provided [here](https://docs.sparkfun.com/flux-sdk/)

## Status

## Using Flux SDK
### November 2024

The flux-sdk is a development framework/SDK intended for use as a component within other Firmware applications. As such, it's incorporated into other projects as an Arduino library.
The flux-sdk was developed as an internal tool initially, but is now being made public. The current functionality is a key component of several SparkFun products and relatively stable, but the sdk supporting functionality (documentation and examples) are in an **alpha** form at best.

To build firmware using Flux, see [Building with Flux](docs/build_with_flux.md)
Over the next several quarters improvements to the sdk include:

## Framework Development
* Improved documentation on using the sdk, as well as adding functionality to it (device drivers for example)
* Remove any out of date remnants from the original version of the sdk - "spark" (which was started in 2020).
* Add support for additional platforms. Currently the focus is ESP32, but a Raspberry Pi RP2350 solution is being developed
* Add additional devices as needed.

Adding new capabilities to the Flux Framework
## Documentation

* Implementing a [Device Class](docs/device_writing.md)
* Implementing [Properties](docs/properties.md)
* Implementing [Parameters](docs/parameters.md)
As noted above, the current documentation for the flux-sdk is located [here](https://docs.sparkfun.com/flux-sdk/)
195 changes: 112 additions & 83 deletions docs/build_with_flux.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,145 +3,174 @@

This section outline the steps needed to support Flux in an Arduino Build Environment. Both support for Arduino IDE development, as well as automated builds that use the Arduino CLI (GitHub actions).

Since Flux is private, only available to SparkFun employees and approved partners, the build integration is different from standard Open Source projects.
Unlike standard Arduino Libraries, the flux-sdk uses the `cmake` build system to create a custom Arduino Library for the target application. This allows the specification of a target platform ias well as the specific features and functionality used in the target application.

## Using Flux within the Arduino IDE
The cmake build system creates the target Arduino Library, the library is either located in builds systems Arduino library directory (normally in the users Documentation/Arduino/libraries folder) or specified using the ```--library``` option to the ```arduino-cli``` tool.

While Flux is structured as an Arduino library, being private, it's not available in the Arduino Library Manager. To use Flux within the Arduino IDE, it is manually installed in the Arduino libraries directory.
## Building an Application Specific flux-sdk Library

Flux is installed by cloning the SparkFun_Flux GitHub repository. To do this, open a terminal on the desired development machine, and change the current directory to your Arduino Libraries install directory. This is normally ```HOME_DIRECTORY/Documents/Arduino/libraries```
A custom library is configured and built using the standard build tool `cmake`

From this location, use the following git command to download and install the ***SparkFun_Flux*** library.
### Install the *flux sdk*

The flux sdk is used to create a custom Arduino Library called ```SparkFun_Flux``` that is made available for use by the Arduino build environment. This allows tailoring which components are needed for the specific application - in this case the datalogger.

First steps is to download the flux-sdk on your machine - which is cloned from github.

```sh
git clone git@github.com:sparkfun/SparkFun_Flux.git
git clone git@github.com:sparkfun/flux-sdk.git
```

Take note, security keys/access to the Flux Private library is required for the clone command to complete successfully.
The normal location is to set install the flux-sdk in the same directory as the ```sfe-dataloger``` (this) repository was installed. If you install it some other location, set the flux-sdk path using the FLUX_SDK_PATH environment variable. This is easily done by cd'ing into the flux-sdk root directory and executing the following command:

Once installed, the SparkFun_Flux library is available to other applications/sketches within the Arduino Environment and used like any other Arduino library.
```sh
export FLUX_SDK_PATH=`pwd`
```

Once you installed Flux locally, you can rapidly install the Arduino library dependencies for Flux using the ```arduino-cli``` as highlighted [later on this page](#install-flux-dependencies).
### Configure cmake

## Using Flux within a Server/Batch Build Process
To specify what components of flux are used by a project, a ```CMakeLists.txt``` file is used. This file is placed in the root of your project folder and outlines what modules of the flux-sdk to use make available.

To use Flux in another project that is being build using build automation (GitHub Actions), Flux should be added as a git submodule to the project, normally in the root directory of the projects repository. The following command is used:
The following is an example of a cmake file from the SparkFun DataLogger IoT application:

```cmake
cmake_minimum_required(VERSION 3.13)
# set project name - setting language to NONE disables the default compiler checks
project(DataLoggerIoT NONE)
# Import the flux-dk cmake system
include(flux_sdk_import.cmake)
# Where is directory that the flux stuff will be added to? This is the relative path from this file
# to the arduino sketch directory this is also used as the name of the cmake project
flux_sdk_set_project_directory(.)
# datalogger is esp32 based
flux_sdk_set_platform(platform_esp32)
# Currently we are using all modules of the SDK
flux_sdk_add_module(flux_all_modules)
# now call the init function/macro - this will build the Arduino Library SparkFun_Flux under this
# main directory
flux_sdk_init()
```sh
git submodule add git@github.com:sparkfun/SparkFun_Flux.git
```

With this structure in place, access to the Flux within a GitHub action is accomplished by using ssh keys.
The key components of this file are the following include the standard cmake version check, and then setting the project name using this line:

The first step is to generate a new key locally - in a shell
```cmake
# set project name - setting language to NONE disables the default compiler checks
project(DataLoggerIoT NONE)
```

```sh
ssh-keygen -t rsa -b 4096 -C "Access to flux"
After the project is defined, the cmake flux-sdk functions and configuration routines are included as follows:

```cmake
# Import the flux-dk cmake system
include(flux_sdk_import.cmake)
```

For this command, don't add a password and you can save the key to a temporary place.
The file `flux_sdk_import.cmake` is located in `external` folder of the flux-sdk ($FLUX_SDK/external) and should be copied to the same folder that contains your projects CMakeLists.txt file.

Next add the public part of the key to the Flux repo as a deploy key. In the Flux github repository, open the page: ```Settings > Deploy Keys``` and click the ***Add deploy key*** button on the page.
The next line sets the location of the project. In this example, it's just the current working directory.

![Add Deploy Key](images/github_deploy_keys.png)
```cmake
# Where is directory that the flux stuff will be added to? This is the relative path from this file
# to the arduino sketch directory this is also used as the name of the cmake project
flux_sdk_set_project_directory(.)
```

Give the key a descriptive name (***my project access key***) and paste the public part of the key into the dialog. Keep the key read-only.
The platform used for the build is set using the following line (note, currently the flux-sdk is only supported on ESP32):

The next step is to add the private part of the key as a ***secret*** in main project (the project using flux) github repository. This is done in ```Settings > Secrets and variables > Actions``` page. On this page, on the **Secrets** tab, select the ***New repository secret*** button.
```cmake
# datalogger is esp32 based
flux_sdk_set_platform(platform_esp32)
```

![Action Secret](images/github_action_secret.png)
The flux-sdk components or `modules` that are included in the application are specified using the `flux_sdk_add_module()` call. The modules correspond to folder names within the `src` folder of the sdk - for example to add the BME280 sensor device, the module name is `device_bme280`.

In the provided dialog, enter a name for the secret (follow variable naming methodology) and set the value to the private portion of the generated key.
To add everything, the name `flux_all_modules` is used, and shown in the example above.

## Download the Flux Submodule
```cmake
# Currently we are using all modules of the SDK
flux_sdk_add_module(flux_all_modules)
```

Within a github action, the key is used to download the Flux submodule. Once the main repository is checked out, the Flux submodule is checked out with the following action code:
The last step in the cmake file is calling the sdk init function `flux_sdk_init()`.

```yaml
# setup the ssh key used to pull in the Flux submodule source. This was the
# only way found to make this work when using private models (ssh private key here, public on Flux deploy keys
- name: Download Flux Submodule
uses: webfactory/ssh-agent@master
with:
ssh-private-key: |
${{ secrets.FLUX_PULL_KEY_2 }}
# checkout flux
- name: Checkout Flux submodule
run: git submodule update --init --recursive
```cmake
# now call the init function/macro - this will build the Arduino Library SparkFun_Flux under this
# main directory
flux_sdk_init()
```

NOTE: In this example, ```FLUX_PULL_KEY_2``` is the name of the Flux key secret within this repository.
### Building the custom Flux Arduino Library

Once the Flux submodule is checked out, the application is setup and built using the Arduino Command Line (```arduino-cli```)
configure the library used during the Arduino build process, the ```cmake``` system is used. The following steps outline how the custom library is built.

## Using Arduino CLI
Set the current directory the root of your project. Then create a directory called build and cd into it.

```sh
mkdir build
cd build
```

### Installation
Now run CMake with the following command:

The Arduino CLI installation is outlined on this [page](https://arduino.github.io/arduino-cli/0.20/installation/).
```sh
cmake ..
```

If working within a github action, the following code will install and setup the arduino-cli:
This will create an Arduino library called ```SparkFun_Flux``` in the root directory of the project. Once completed, you can delete the build directory, and build the projects firmware.

```yaml
# Setup Arduino command line - install esp32 and all the libs flux needs
- name: Arduino - Install and setup the Arduino CLI
uses: arduino/setup-arduino-cli@v1
This custom library can be copied to the Arduino libraries install folder and used like a standard Arduino library, or referenced using the ```---library``` flag of the arduion-cli command

- name: Arduino - Start config file
run: arduino-cli config init --additional-urls "https://raw.githubusercontent.com/espressif/arduino-esp32/gh-pages/package_esp32_index.json"
## Building using the Arduino CLI

- name: Arduino - Update index
run: arduino-cli core update-index
### Install Arduino CLI

# Install ESP32
- name: Arduino - Install ESP32 platform
run: arduino-cli core install esp32:esp32
```
First, install the ```arduino-cli``` on your system. Details on how to do this are located [here](https://arduino.github.io/arduino-cli/0.20/installation/). While the arduino-cli isn't required to configure or build a flux build Arduino library, it helps with configuration.

Note: The above example also installed the ESP32 platform
#### Install the Target Arduino Core

### Install Flux Dependencies
Once the CLI is installed, the target core for the application can be installed using the arduino-cli. Note: this can also be performed within the Arduino application itself.

The Flux repository includes a script that installs all library dependencies using the ```arduino-cli```. This command, ```install-libs.sh``` is located within the root directory of the Flux repository.
For example,the following commands complete the installation and install the ESP32 Arduino platform

To run this command within a github Action, the following code is used:
```sh
arduino-cli config init --additional-urls "https://raw.githubusercontent.com/espressif/arduino-esp32/gh-pages/package_esp32_index.json"

arduino-cli core update-index

```yaml
# install the libraries Flux uses
- name: Install Flux dependant libraries.
run: ./SparkFun_Flux/install-libs.sh
arduino-cli core install esp32:esp32
```

Note: The above command assumes Flux is installed in the root directory of the repository. Adjust the command path to the structure of your repository if needed.
### Install dependant Arduino libraries

Once the flux-sdk is installed, you can install all libraries the framework depends on by issuing the following command (note this command uses the ```arduino-cli```):

```sh
$FLUX_SDK_PATH/install-libs.sh
```

### Compile and Build

Once all the dependencies are installed, the ```arduino-cli compile``` option is called to build the desired application. To use Flux as a library, the ```--library``` switch is used with the compile call.

The following is an example of building an ESP32 based sketch, which uses the Flux library.
The following is an example of building an ESP32 based sketch (the SparkFun DataLogger Application), which uses the custom Flux library.

Note that the location of the Flux library is passed in using the ```--library'`` switch, and that the ***full*** path to the Flux directory is provided. Using a relative path to the Flux library directory causes this command to fail

```yaml
- name: Compile DataLogger firmware binary
run: arduino-cli compile --fqbn esp32:esp32:esp32
./sfeDataLoggerIoT/sfeDataLoggerIoT.ino
```sh
arduino-cli compile --fqbn esp32:esp32:esp32 ./sfeDataLoggerIoT/sfeDataLoggerIoT.ino \
--export-binaries --library `pwd`/SparkFun_Flux
```

### After the Build
Once the build is complete, the resultant binary files are located in: `sfeDataLoggerIoT/build/esp32.esp32.esp32/`

For reference, once the above compile command is completed, the resultant Firmware binary is renamed and uploaded as an Action artifact using the following code:
## Further Details

```yaml
- name: Rename Library
run: |
cd sfeDataLoggerIoT/build/esp32.esp32.esp32/
mv sfeDataLoggerIoT.ino.bin SparkFun_DataLoggerIoT.bin
- uses: actions/upload-artifact@v3
with:
name: SparkFun_DataLoggerIoT.bin
path: sfeDataLoggerIoT/build/esp32.esp32.esp32/SparkFun_DataLoggerIoT.bin
```
A great example of how to build a flux-sdk application at the command line, as well as part of a github action is the SparkFun DataLogger IoT application, located [here](https://github.com/sparkfun/sfe-datalogger)

0 comments on commit 77de1ad

Please sign in to comment.