Developer Vyacheslav Serbov, who performs under the pseudonym slvDev, demonstrated completely local text generation on the ESP32-S3 microcontroller. The language model he created contains 28.9 million parameters and produces about 9.9 tokens per second without relying on a server or other external computational resources. The generated short stories are displayed on a connected OLED screen.
Project esp32-ai tested on the ESP32-S3 N16R8 module, which has 512 KB of built-in SRAM, 8 MB of PSRAM, and 16 MB of flash memory. The quantized four-bit model occupies 14.9 MB. The repository contains the C runtime source code, training and model quantization tools, firmware, tests, and scripts for loading it onto the board.
The stated 28.9 million parameters should not be directly compared to the number of parameters of standard desktop LLMs. According to the detailed report of the developer, the model consists of a dense computational core with approximately 559 thousand parameters, an output layer with 3.1 million parameters, and a layer-wise vector representation table of approximately 25 million parameters. It is the latter that provides the bulk of the formal model size but is not processed in its entirety when generating each token.
The model employs three levels of memory. The most frequently used data reside in the internal SRAM, while the output layer, KV cache, and temporary buffers are in PSRAM, and the 25-million table remains in flash memory and is accessible through memory mapping. For each token, only about six rows are read from it, totaling approximately 450 bytes.
Such a device is based on the Per-Layer Embeddings technology used by Google in Gemma 3n. According to Google's documentation, PLE parameters can be stored outside the model's working memory and added to the output process as individual layers are processed. In esp32-ai, this approach has been adapted to the memory hierarchy of the microcontroller, where fast SRAM is particularly limited.
The first correct version of the C execution environment delivered only 0.57 tokens per second. After placing the output layer in PSRAM, switching to eight-bit activations, distributing computations between two Xtensa LX7 cores, and other optimizations, the latency was reduced to 94.9 ms per model step. The final measured speed reached 9.88 tokens per second considering the entire generation process. The main limitation now is the PSRAM bandwidth when reading the output layer.
The model is trained on a synthetic dataset TinyStories, consisting of simple English stories with a limited vocabulary. Therefore, it is able to continue phrases and compose small coherent stories, but is not intended for answering questions, following instructions, programming, or reproducing factual knowledge. The developer primarily views the work as a demonstration of effective model deployment in a microcontroller's memory, rather than as a ready standalone chatbot.
In the current documentation, the author emphasizes that esp32-ai is an independent development. The projects llama2.c by Andrei Karpathy and the earlier launch of a 260-thousand parameter model on the ESP32-S3 are cited only as benchmarks and examples of prior work: the code, checkpoints, and methodology are not directly borrowed from them. The source code of esp32-ai is distributed under the MIT license.
Source: linux.org.ru
