Gesture detector

This guide demonstrates using Consentium's TinyML library with an MPU6050 sensor for gesture recognition on an ESP32 or compatible Edge board. The example collects accelerometer and gyroscope data to recognize gestures using a TensorFlow Lite model.

Overview

The example uses an MPU6050 sensor to collect accelerometer and gyroscope data. It detects a gesture based on movement exceeding a threshold, then collects and processes data to recognize gestures using a TensorFlow Lite model. The recognized gestures are printed to the Serial Monitor.

Prerequisites

• Hardware: ESP32 or compatible Edge board, MPU6050 sensor.

• Software: Consentium's TinyML library, EdgeNeuron library, TensorFlow Lite Micro, MPU6050 library.

Code Explanation

Includes and Constants

#include <MPU6050_light.h>  // MPU6050 library
#include <Wire.h>            // I2C library for ESP32
#include <EdgeNeuron.h>      // TensorFlow Lite wrapper for Arduino
#include "model.h"           // Trained model

MPU6050 mpu(Wire);           // Create MPU6050 instance

const float accelerationThreshold = 2.5;  // Threshold (in G values) to detect a "gesture" start
const int numSamples = 119;                // Number of samples for a single gesture
int samplesRead;                           // Sample counter
const int inputLength = 714;               // Input tensor size (6 values * 119 samples)

// Tensor Arena memory area for TensorFlow Lite to store tensors
constexpr int tensorArenaSize = 8 * 1024;
alignas(16) byte tensorArena[tensorArenaSize];

// Gesture labels table
const char* GESTURES[] = {
  "punch",
  "flex"
};
#define NUM_GESTURES (sizeof(GESTURES) / sizeof(GESTURES[0]))

• MPU6050_light.h: Include the MPU6050 library.

• Wire.h: Include the I2C communication library.

• EdgeNeuron.h: Include the TensorFlow Lite wrapper for Arduino.

• model.h: Include the header file for the TensorFlow Lite model.

• mpu: Create an instance of the MPU6050 sensor.

• accelerationThreshold: Threshold for detecting significant movement.

• numSamples: Number of samples required for a gesture.

• inputLength: Total input tensor size.

• tensorArenaSize: Size of memory allocated for TensorFlow Lite tensors.

• GESTURES: Array of gesture labels.

• NUM_GESTURES: Number of gestures.

Setup Function

void setup() {
  Serial.begin(9600);
  Wire.begin();  // Start I2C communication

  // Initialize MPU6050 sensor
  byte status = mpu.begin();
  if (status != 0) {
    Serial.println("MPU6050 initialization failed!");
    while (true);  // Stop execution on failure
  }

  Serial.println("MPU6050 initialized.");
  
  // Set accelerometer and gyroscope sampling rates
  mpu.calcOffsets();  // Calibrate MPU6050
  
  Serial.println();
  Serial.println("Initializing TensorFlow Lite model...");
  if (!initializeModel(model, tensorArena, tensorArenaSize)) {
    Serial.println("Model initialization failed!");
    while (true);  // Stop execution on failure
  }
  Serial.println("Model initialization done.");
}

• Initializes serial communication.

• Starts I2C communication for the MPU6050 sensor.

• Initializes and calibrates the MPU6050 sensor.

• Initializes the TensorFlow Lite model and sets up the tensor arena.

Loop Function

void loop() {
  float aX, aY, aZ, gX, gY, gZ;

  // Wait for significant movement (exceeding the threshold)
  while (true) {
    mpu.update();  // Update sensor readings
    
    // Read accelerometer values
    aX = mpu.getAccX();
    aY = mpu.getAccY();
    aZ = mpu.getAccZ();

    // Compute the total acceleration magnitude
    float aSum = fabs(aX) + fabs(aY) + fabs(aZ);

    // Check if the acceleration exceeds the threshold (gesture detected)
    if (aSum >= accelerationThreshold) {
      samplesRead = 0;  // Reset sample counter
      break;            // Exit waiting loop
    }
  }

  // Collect data for gesture
  while (samplesRead < numSamples) {
    mpu.update();  // Update sensor readings

    // Read accelerometer and gyroscope values
    aX = mpu.getAccX();
    aY = mpu.getAccY();
    aZ = mpu.getAccZ();
    gX = mpu.getGyroX();
    gY = mpu.getGyroY();
    gZ = mpu.getGyroZ();

    // Normalize sensor data (since the model was trained on normalized values)
    aX = (aX + 4.0) / 8.0;
    aY = (aY + 4.0) / 8.0;
    aZ = (aZ + 4.0) / 8.0;
    gX = (gX + 2000.0) / 4000.0;
    gY = (gY + 2000.0) / 4000.0;
    gZ = (gZ + 2000.0) / 4000.0;

    // Place the 6 values (acceleration and gyroscope) into the model's input tensor
    setModelInput(aX, samplesRead * 6 + 0);
    setModelInput(aY, samplesRead * 6 + 1);
    setModelInput(aZ, samplesRead * 6 + 2);
    setModelInput(gX, samplesRead * 6 + 3);
    setModelInput(gY, samplesRead * 6 + 4);
    setModelInput(gZ, samplesRead * 6 + 5);

    samplesRead++;

    // Once all samples are collected, run the inference
    if (samplesRead == numSamples) {
      if (!runModelInference()) {
        Serial.println("Inference failed!");
        return;
      }

      // Retrieve output values and print them
      for (int i = 0; i < NUM_GESTURES; i++) {
        Serial.print(GESTURES[i]);
        Serial.print(": ");
        Serial.print(getModelOutput(i) * 100, 2);
        Serial.println("%");
      }
      Serial.println();
    }
  }
}

• Waits for movement that exceeds the threshold to start gesture detection.

• Collects and normalizes accelerometer and gyroscope data for a defined number of samples.

• Fills the input tensor with normalized data.

• Runs inference once all samples are collected.

• Prints the probability of each gesture recognized by the model.

License

This code is licensed under the MIT license. All text in the license header must be included in any redistribution.