The widely popular ESP32 microcontroller board is a great choice for any hobbyist to build smart electronics and home automation sensors.
The board’s powerful but low-cost design inspired millions of DIY enthusiasts and laid the ground for many great products, such as the Dynatrace DevOps UFO.
Firmware software for ESP32 boards is mostly implemented by either using the Arduino stack or the native Expressif IDF tools that come along with the ESP32 microcontroller boards.
While hobbyists prefer the more user-friendly Arduino stack to implement their firmware, professionals prefer to use the Expressif IDF tools to build the firmware in native C++ as only the native tools unlock the full potential of this System-on-a-chip (SoC) platform.
One example for the preference of the native IDF tools is the possibility to use both CPU cores for real parallel processing of instructions.
The image below shows the typical IDF tool process of building and uploading firmware to an ESP32 microcontroller board.
What’s the value of a fully automated firmware build process?
The automation of build and integration (CI/CD) pipelines for any kind of software is state-of-the-art nowadays.
With an ESP32 board involved it gets a bit more tricky as the cross-compiling IDF toolchain comes with several dependencies and the resulting binaries can’t be executed on your build machine, unlike with traditional software binaries.
The real value, nevertheless comes with having a clean build environment at hand that allows you to build your firmware on a clean machine.
A fully automated IDF tools build pipeline relieves you from maintaining the IDF build environment on your local machine and it also allows you to identify regressions early as each commit leads to a clean result.
On local machines you often run into the so called ‘runs on my local machine’ scenario, where previously built libraries are still remaining in your build environment and you don’t realize that the source would not build nor run in another clean build environment.
A fully automated firmware build environment also helps you to keep things reproducible and in a consistent state, while in your local build machine you could run into the version hell of maintaining all the right versions of all the necessary dependencies.
Run the IDF tools within a Docker container
Before we start to configure our IDF tools build pipeline, let’s see how Docker can make our lives easier when dealing with ESP32 builds.
Expressif also offers their toolchain as a Docker container that already comes with all necessary dependencies and libraries installed.
You can simply download and use this Docker container to build your own ESP32 project without the need to keep a local ESP32 toolchain installed.
As the Docker container is offered for all major IDF versions, it also helps you to still build your project by using a predefined IDF version. Keeping different IDF versions and their dependencies on a local machine typically is a nearly impossible task.See below the IDF docker container and how to download it from Dockerhub.
Simply pull the latest container version and run the idf build command through the local container along with mapping your local ESP32 project directory, as it is shown below:
docker pull espressif/idf docker run --rm -v $PWD:/project -w /project espressif/idf idf.py build
This IDF docker container is also used as the basis for the GitHub IDF Action that we will now use to configure a fully automatic firmware build pipeline.
Run the IDF tools within a GitHub Action CI/CD Pipeline
GitHub Actions are a great way of automating tasks and building a workflow CI/CD pipeline that is directly stored within your GitHub project repository.
Using a simple YAML workflow configuration file that is stored within your local project repository folder named ‘.github/workflows’ you can implement any kind of automation workflows that GitHub will then execute upon subscribed event triggers.
Best approach is to start with an empty GitHub Action workflow and then add automation steps one-by-one according to your detailed needs.
As a first step we do register for GitHub events that trigger our workflow, namely to react on push events and upon manual dispatches, as it is shown below:
name: Dynatrace UFO CI
on:
push:
branches: [ main ]
# Allows you to run this workflow manually from the Actions tab
workflow_dispatch:
jobs:
build:
runs-on: ubuntu-latest
The workflow_dispatch configuration allows users to not only run the workflow on code pushes but also to manually trigger the workflow, as it is shown below:
The jobs definition specifies the workflows and their steps along with the machine type the workflow will be run.
Lets now see the first step that checks out the repositories source code onto the local machine as a preparation for building the firmware:
- name: Checkout repo
uses: actions/checkout@v2
with:
submodules: 'recursive'
After the checkout is finished, we have to convert some of our resource files to ESP32 usable header files:
- name: Data to header file conversion
run: |
python data2h.py data/index.html main/indexhtml.h
python data2h.py data/material-design-icons.ttf main/fontttf.h
python data2h.py data/material-design-icons.woff main/fontwoff.h
python data2h.py data/material-design-icons.svg main/fontsvg.h
python data2h.py data/material-design-icons.eot main/fonteot.h
python data2h.py data/cert.pem main/certpem.h
python data2h.py data/key.pem main/keypem.h
After the conversion to header files is finished, we will trigger the IDF build by adding the IDF GitHub action as it is shown below:
- name: esp-idf build
uses: espressif/esp-idf-ci-action@main
with:
esp_idf_version: v3.3.1
path: '.'
A successful build will generate a local firmware binary that can then be offered as a download directly as a GitHub Action workflow result.
- name: Store Artifacts
uses: actions/upload-artifact@v2
with:
name: ufo-esp32.bin
path: |
build/ufo-esp32.bin
See below the GitHub IDF workflow running each individual build step along with logging the steps debug messages.
After a successful GitHub Action run, the last Store Artifacts step does persist the firmware binary and offers a direct download, as it is shown below:
The Dynatrace UFO Example
The above workflow was taken from an existing open source hardware product that was developed and published by Dynatrace. The so-called ‘Dynatrace UFO’ helps DevOps teams to visualize the status of their build pipeline as well as to monitor the health state of their Dynatrace monitored environment.
The Dynatrace UFO open source firmware is available on GitHub and its latest firmware binary is also available in a subfolder of the GitHub repository.
The Dynatrace UFO is built on the basis of an ESP32 board and its firmware can be updated over the air by loading the latest firmware version directly from GitHub.
The UFO checks the firmware version by using a version.json file that resides within the firmware binary folder.
As the build process of the Dynatrace UFO was cumbersome and involved the setup and maintenance of an IDF toolchain, I implemented a fully automated firmware build pipeline within my forked UFO repository.
The UFO build pipeline not only builds the firmware binary but it also modifies the version.json file and automatically releases a new nightly build firmware version.
The following pipeline step does modify the local version json file and pushes the new firmware binary as well as the updated version file back into the project repository:
- name: Get current date
id: date
run: echo "::set-output name=date::$(date +'%h %d %Y - %H:%M:%S')"
- name: Update version.json file
uses: jossef/action-set-json-field@v1
with:
file: nightly/version.json
field: version
value: ${{ steps.date.outputs.date }}
- name: Copy firmware to nightly folder
run: |
cp build/ufo-esp32.bin nightly/ufo-esp32.bin
git config --global user.email "user@email.com"
git config --global user.name "username"
git add . -A
git commit -m "nightly build"
git push
As a result it is possible to easily build and update a Dynatrace UFO with the latest nightly firmware versions.
You can find the full Dynatrace UFO CI/CD pipeline available on GitHub.
Summary
The use of GitHub Actions to automate the build and release of Expressif IDF based firmware versions for your ESP32 board saves a lot of time and effort and avoids the hassle of managing a local build machine.
The use of a fully automated CI/CD pipeline should not only be considered for traditional server software but also dramatically simplifies your development process for projects where ESP32 hardware is involved.
If you liked this blog post and you want to know more about what you can do with ESP32 boards, head over to Amazon and read my book about home automation with ESP32 and Home Assistant:
