KMKyrylo Matiushenko

Embedded vision evolution

One branch, two generations.

This branch shows how the original single-board ESP8266 experiment evolved into a distributed edge-vision system. Switch between both generations to compare their architecture, processing limits, and responsibilities.

The STM32 generation is shown first because it is the current engineering baseline; the ESP8266 version remains available as the prototype that established the capture, motion-detection, and tracking concepts.

Project 01 · Current generation

STM32–ESP32–NanoPi Vision System

A distributed edge-vision system that turns an OV2640 camera feed into an operator-ready MJPEG dashboard with motion analysis, neural inference, and interactive object tracking. The design separates deterministic camera capture, wireless transport, and NPU perception so each platform handles the work that matches its resources.

STM32H503 controls ArduCAM and transports frames over SPI. ESP32 receives and buffers those frames, performs lightweight Algorithm-mode motion detection, and forwards the stream through Wi-Fi/TCP. NanoPi R5S then runs motion segmentation and YOLOv5s INT8 inference on the RK3568 NPU, maintains masks and track IDs, and serves the only web interface in the system.

60 FPS · 0.8 TOPS

The neural-network inference path can process up to 60 frames per second at 0.8 TOPS. This figure describes model-processing capacity; end-to-end camera FPS still depends on capture resolution, JPEG size, SPI, Wi-Fi, and the selected operating mode.

Open GitHub

Data path

Processing is split across the hardware that fits it best.

01 OV2640 / ArduCAM Frame capture
02 STM32H503 ThreadX camera transport
03 ESP32-WROOM-32U SPI ingest · motion analysis
04 NanoPi R5S RKNN inference · tracking · web UI

What it proves

The project develops the first camera prototype into a complete multi-board product architecture. Real-time work is divided between microcontroller firmware, a Wi-Fi transport bridge, and an embedded Linux NPU service instead of forcing every task onto one constrained device.

STM32 captures and transports OV2640 frames, ESP32 handles SPI reception and Algorithm-mode motion detection, while NanoPi is the only web server and runs neural motion segmentation, YOLOv5s INT8 RKNN inference, masks, and persistent track IDs.

System layers

Deterministic capture

STM32H503 firmware uses STM32 HAL and ThreadX to control ArduCAM and move frames over SPI.

Transport and motion

ESP32 stores incoming frames, performs lightweight motion analysis, and forwards the stream over Wi-Fi/TCP.

NPU perception

NanoPi R5S serves the dashboard and performs RKNN inference, motion segmentation, masks, and interactive tracking.

STM32 as an expandable hardware hub

STM32H503 is not limited to forwarding camera data. It is the deterministic hardware hub of the system: it owns sensor initialization, real-time capture, peripheral timing, diagnostics, and the gateway toward ESP32. The current baseline connects OV2640 through ArduCAM, but the same architecture can be extended with additional sensors and control hardware without moving time-critical I/O into the networking layer.

Peripheral expansion

Additional IMUs, environmental sensors, encoders, distance sensors, relays, or actuators can be integrated through available SPI, I²C, UART, GPIO, ADC, and PWM peripherals.

Real-time isolation

ThreadX tasks keep capture and hardware timing independent from Wi-Fi quality, Linux load, and neural-network latency.

One upstream interface

New devices can be normalized into the STM32 application layer and sent upstream without requiring every sensor to implement networking.

How motion detection works on ESP32

The detector runs as a dedicated FreeRTOS task on core 1 and is active only in Algorithm mode. Processing is rate-limited so JPEG decoding and image analysis never starve SPI reception or the Wi-Fi transport task.

  1. 01The newest JPEG is decoded and reduced to an 80×60 grayscale plane.
  2. 02Gaussian smoothing suppresses JPEG noise; average brightness delta compensates for global exposure changes.
  3. 03The current and previous frames are compared with an absolute threshold of 18 to build a binary motion mask.
  4. 04Isolated pixels are removed and one-pixel gaps are bridged without allocating another full-size mask.
  5. 05An 8-connected flood fill extracts components of at least eight pixels and rejects near-full-frame shake or exposure transitions.
  6. 06Nearby regions are merged, sorted by changed-pixel count, and converted back into source-image coordinates; up to four boxes are retained.
  7. 07Tracking scores candidates by predicted center, luminance, area, and aspect ratio; velocity prediction keeps a target alive through short detection gaps.

Data-transfer protocols

Camera → STM32

I²C1 configures OV2640 registers; SPI1 reads the JPEG data from the ArduCAM FIFO.

STM32 → ESP32

SPI2 mode 0 plus a READY GPIO uses the binary CAM1 protocol: a 20-byte header, START/DATA/END packets, chunks up to 1536 bytes, offsets, frame IDs, and control query/reply messages.

ESP32 → NanoPi

A persistent Wi-Fi/TCP connection to port 5000 carries NPI1 frames: a 20-byte binary header, JPEG payload, optional UTF-8 JSON metadata, and a compact control response for mode, resolution, and tracking.

NanoPi → Browser

HTTP on port 8080 provides the dashboard, REST endpoints, status JSON, snapshots, and multipart MJPEG streaming in source, 800, 640, or 320 pixel quality profiles.

Software features and reliability decisions

Fixed-memory MCU path

STM32 circulates two fixed chunk objects and never stores a full JPEG. ESP32 uses three guarded 32 KiB frame slots, preventing a published frame from being overwritten while TCP or the detector reads it.

Newest-frame scheduling

NanoPi publishes every valid frame immediately, while the NPU worker replaces stale pending work instead of building a queue. The result prioritizes low latency over processing every frame.

Three processing modes

Raw mode streams video, Algorithm mode adds ESP32 motion boxes, and Neural mode adds NanoPi segmentation, YOLO detections, masks, classes, confidence values, and track IDs.

Validation and recovery

Magic values, lengths, offsets, bounds, and JPEG markers are validated. Oversized frames trigger a lower-resolution fallback; Wi-Fi and TCP reconnect automatically, and systemd restarts the NanoPi service.

Core responsibilities

  • Built STM32Cube/ThreadX firmware for camera initialization, capture, diagnostics, and SPI transport.
  • Implemented production PlatformIO firmware for ESP32 frame reception, state management, motion detection, and NanoPi transport.
  • Developed the NanoPi Linux service with an HTTP/MJPEG dashboard, detector abstraction, motion segmentation, and object tracking.
  • Prepared the YOLOv5s ReLU INT8 RKNN pipeline for the RK3568 NPU at 320×320 input resolution.
  • Documented architecture, interfaces, deployment, operations, verification, security, and compliance in English and Ukrainian.