tflite-micro TensorFlow Lite Micro

TensorFlow Lite for Microcontrollers (TFLM) is a C++ interpreter for running .tflite models on memory-constrained targets. NuttX integrates it from apps/mlearning/tflite-micro.

The build downloads a pinned TFLM snapshot (cfa4c91d1b36c37c7c104b9c664615e59f1abfe3, 24 February 2024) from https://github.com/tensorflow/tflite-micro and applies NuttX patches that add INT8-only operator registrations and an extern "C" entry point for the hello-world example.

Dependencies

CONFIG_TFLITEMICRO depends on all of:

TFLM is C++, so the configuration also needs C++ support (CONFIG_HAVE_CXX and a C++ standard library such as CONFIG_LIBCXX).

Optional acceleration:

  • CONFIG_MLEARNING_CMSIS_NN — replace reference kernels with CMSIS-NN implementations. See cmsis-nn CMSIS-NN Library.

  • CONFIG_ARM_NEON — when CMSIS-NN is enabled, a few kernels are taken from apps/mlearning/tflite-micro/operators/neon instead of CMSIS-NN.

Configuration

From Application Configuration ‣ Machine Learning Support:

CONFIG_TFLITEMICRO

Enable the TFLM library.

CONFIG_TFLITEMICRO_DEBUG

Keep TFLM error strings and print memory-use information (TF_LITE_SHOW_MEMORY_USE).

CONFIG_TFLITEMICRO_SYSLOG

Route TFLM logging through NuttX syslog via tflm_syslog.cc. CONFIG_TFLITEMICRO_SYSLOG_LEVEL selects the syslog priority (default 6, LOG_INFO). See include/syslog.h. When neither debug nor syslog is enabled, TF_LITE_STRIP_ERROR_STRINGS is set to reduce code size.

CONFIG_TFLITEMICRO_TOOL

Build the tflm NSH command from tflm_tool.cc. CONFIG_TFLITEMICRO_TOOL_PRIORITY and CONFIG_TFLITEMICRO_TOOL_STACKSIZE set the task attributes (defaults: priority 100, stack 4096).

CONFIG_TFLITEMICRO_HELLOWORLD

Build the tflm_hello example. The example runs the upstream TFLM hello-world float and INT8 sine models. CONFIG_TFLITEMICRO_HELLOWORLD_PRIORITY and CONFIG_TFLITEMICRO_HELLOWORLD_STACKSIZE set the task attributes (defaults: priority 100, stack 4096).

Building

The sim:tflm defconfig enables TFLM, the tflm tool, the hello-world example, libc++, FlatBuffers, gemmlowp, KissFFT, and Ruy.

Makefile build (produces tflm and tflm_hello):

$ cd nuttx
$ make distclean
$ ./tools/configure.sh sim:tflm
$ make -j$(nproc)
$ ./nuttx

CMake build (produces both tflm and tflm_hello):

$ cd nuttx
$ cmake -B build -DBOARD_CONFIG=sim:tflm -GNinja
$ cmake --build build
$ ./build/nuttx

The first build downloads the TFLM sources. Subsequent builds reuse the unpacked tree under apps/mlearning/tflite-micro/tflite-micro. make distclean in the apps tree removes that snapshot.

Using the tflm tool

tflm is a host-oriented NSH helper for sim:tflm. It loads a .tflite file from the filesystem with ifstream, allocates the tensor arena with new, constructs a tflite::MicroInterpreter, and calls AllocateTensors(). That heap-and-filesystem path is intentional on the simulator. On-target applications should embed the model as a C array; see Embedding a model in an application below.

nsh> tflm -h

Utility to use tflite micro on nuttx.
[ -C       ] Compile tflite model into c++ codes.
[ -E       ] Run inference and print the outputs.
[ -I       ] Print model information and arena usage.
[ -i <str> ] Readable model file path.
[ -o <str> ] Writable c++ file path (required with -C).
[ -p <str> ] Prefix of compiled code.
[ -a <int> ] Arena size (mempool).
[ -d <str> ] Comma separated input values, e.g. 0.5,1,2 (implies
             -E). One value fills the whole tensor. Quantized
             inputs are quantized automatically.
[ -x <str> ] Raw input tensor file (implies -E).
             -d/-x may be repeated for models with several inputs.
[ -n <int> ] Number of inferences to run for timing (implies -E).
[ -h       ] Print this message.

-i is required. -o is required only with -C. Defaults are prefix NXAI and arena size 8192 bytes.

The tool checks the file with the FlatBuffers verifier before using it. It then registers only the operators that the model uses. About 90 built-in operators are supported, including convolution, pooling, fully connected, LSTM, activation, element-wise, and reshaping operators. An unsupported builtin or custom operator is reported by name, for example Unsupported custom operator: SignalWindow.

-I prints the operators, the name, type, shape, and quantization parameters of every input and output tensor, and the number of arena bytes used. Use that number to choose -a.

-E runs inference and prints every output tensor. Quantized outputs are dequantized with the tensor scale and zero point. Tensors with more than one element also print the index of the largest value (argmax). Inputs come from -d or -x. The first -d or -x feeds input 0, the next one feeds input 1, and so on. Inputs without data are set to zero. -x expects a file whose size equals the tensor size in bytes. -n repeats the inference and prints the total and average time spent in Invoke(). The profiler CSV is printed for the last run.

-C appears in the help text but is not functional in NuttX builds. Model compilation requires TFLITE_MODEL_COMPILER, which neither the Makefile nor the CMake integration defines. tflm -C prints Not supported compiling.

Hello-world example

With a sim:tflm image:

nsh> tflm_hello

This runs the upstream hello-world test: it profiles memory and latency, then loads the float and INT8 sine models that are converted to C arrays at build time with xxd. Success ends with:

~~~ALL TESTS PASSED~~~

Testing

The sim:tflm configuration is the supported way to test TFLM on the host. It enables CONFIG_TFLITEMICRO, CONFIG_TFLITEMICRO_DEBUG, CONFIG_TFLITEMICRO_TOOL, and CONFIG_TFLITEMICRO_HELLOWORLD.

The host needs a C++ toolchain, curl, unzip, patch, xxd, and the NuttX apps tree next to nuttx (../apps or CONFIG_APPS_DIR).

  1. Configure and build:

    $ cd nuttx
$ make distclean
$ ./tools/configure.sh sim:tflm
$ make -j$(nproc)

    The first build downloads TFLM and its math/FlatBuffers dependencies. A successful link prints LD:  nuttx. The apps registry must list both tflm and tflm_hello.

  2. Run the simulator and the hello-world test:

    $ ./nuttx
nsh> tflm -h
nsh> tflm_hello

    tflm -h prints the usage text above. tflm_hello prints allocator and profiler information, then ~~~ALL TESTS PASSED~~~.

  3. Optional: run the hello-world sine models from the host filesystem (the sim configuration includes hostfs). NSH limits the command line length, so mount a short host path, for example a symlink /tmp/m to apps/mlearning/tflite-micro/tflite-micro/tensorflow/lite/micro/examples/hello_world/models:

    nsh> mount -t hostfs -o fs=/tmp/m /models
nsh> tflm -I -i /models/hello_world_int8.tflite
nsh> tflm -i /models/hello_world_int8.tflite -d 1.5708

    -I lists three FULLY_CONNECTED operators and one INT8 input and output with their quantization parameters. Running with -d 1.5708 (pi/2) prints an output close to 1.0, the sine of the input.

    If the arena is too small the tool prints AllocateTensors failed and suggests a larger -a.

CMake is equivalent: cmake -B build -DBOARD_CONFIG=sim:tflm -GNinja then cmake --build build and ./build/nuttx.

Embedding a model in an application

The TFLM library is designed for targets without a filesystem and without dynamic allocation for the model itself. The tflm NSH tool is an exception used on sim:tflm. Typical on-target applications compile the .tflite file into a C array and pass it to tflite::GetModel().

The CMake helper tflite_generate_data() in apps/mlearning/tflite-micro/CMakeLists.txt wraps xxd -i for that purpose. The same conversion can be done manually:

xxd -i model.tflite model_data.h

Provide a tensor arena (the -a size in tflm, or a static buffer in application code) large enough for the model’s scratch tensors. Insufficient arena size makes MicroInterpreter::AllocateTensors() fail.

Patches applied by NuttX

  • tflite-micro.patch — guard the ASSERT macro in the TFLM signal circular buffer.

  • 0001-dequantize-int8.patch — Register_DEQUANTIZE_INT8().

  • 0002-quantize-int8.patch — Register_QUANTIZE_FLOAT32_INT8().

  • 0003-mean-int8.patch — Register_MEAN_INT8().

  • 0004-tflite-add-extern-C-to-main-function-to-avoid-c-mang.patch — extern "C" on the hello-world main so NuttX can call it.

See also