path: root/pres.md

---
title: Ourosboros Flash Reader
author: Patrick Schönberger
date: 16.07.2025
transition: none
revealjs-url: https://cdn.jsdelivr.net/npm/reveal.js@5.2.1
slideNumber: true
---

## Structure
<!-- structure: start with a problem and reproduce the work leading to the working solution -->


- Start with the problem
- Work our way to the solution step by step



<!-- access smart home hw on crime scenes -->
<!-- many shelly devices are based on esp32/esp8266 -->
<!-- they dont contain any usable data but contain identifiable user data -->
<!-- this can be used to inquire about the user account -->
<!-- so we need to extract the content of the esp's flash memory -->
<!-- we then also need to extract filesystems from the memory -->
<!-- do it read-only, verifiably -->
<!-- existing solutions (esptool, mos) can also write and erase memory -->
<!-- they are also complex, making understanding and changing the code time consuming -->
<!-- what about writing a custom extraction tool? -->

## The Problem

- How to access data on Shelly devices?
- Most Shelly devices are based on ESPs
- They don't contain usable data like logs
- They do contain user-identifiable data

## 

- Extract the flash memory and read the data
- Do it read-only and document the process
- Existing solutions (esptool, mos) can modify memory
- They are very complex, and thus hard to modify
- What about a custom tool?



<!-- what is the bare minimum needed to talk to the esp? -->
<!-- - two modes: bootmode and runmode -->
<!-- - decided by GPIO0 at start -->
<!-- - the esp as well as the shelly devices expose uart pins (tx/rx) -->
<!-- - in runmode they output logging information -->
<!-- - in bootmode they listen to a custom serial protocol -->
<!-- so we need a serial connection and the ability to enter boot mode! -->
<!-- what can the serial protocol do? -->
<!-- - sync -->
<!-- - write ram/flash/registers -->
<!-- - configuration etc. -->
<!-- - on the esp32 it can read flash, but not on esp8266, esp32c3, esp32c6 -->

## Talking to the ESP

- What is the bare minimum needed to talk to an ESP?
- Two modes: boot and run
- Serial pins are exposed on some Shelly devices and accessible on others
- In boot mode they can be used to communicate using a custom protocol
<!-- * So we need a serial connection and the ability to enter boot mode -->

## 

- What can we do with the protocol?
  - Synchronize
  - Write RAM, flash and registers
  - Configure memory etc.
  - On the ESP32 it can read flash, but not on the ESP8266, ESP32C3 or ESP32C6
- How do other tools read flash?



<!-- how do other tools read flash? -->
<!-- - we cant directly read flash, but we can write ram -->
<!-- - write a program, load it into ram, run it and then talk to it instead -->
<!-- - flash loader/stub -->
<!-- - esptool uses two different variants, c based and rust based -->
<!-- - the c based one is older and getting replaced, but it is also dramatically simpler and also supports the esp8266 -->
<!-- - so we use the c based one and customize it (remove write and erase flash commands) -->
<!-- technically this means we do have write access until the flash loader is activated -->
<!-- the extraction tool is also small and runs a fixed number of commands -->
<!-- -> as sure as we can be -->

## Reading Flash Memory

- We cannot directly read flash, but we can write RAM
- Write a program for the ESP, load it into RAM and run
- This is called the flash loader or stub loader
- Esptool uses two versions:
  - Legacy C version
  - New Rust version

## 

- The C version is getting replaced but it is much simpler and it supports ESP8266
- We customize this version by removing write and erase commands
- This way we technically have write access until the flash loader is activated
- The extraction tool is small and contains no write or erase commands



<!-- how does the serial protocol work? -->
<!-- - data is encoded using SLIP frames -->
<!-- - the host sends a request and the target (esp) sends a response -->
<!-- - steps to read flash: -->
<!--   - sync -->
<!--   - identify chip -->
<!--   - read mac -->
<!--   - (change baud) -->
<!--   - upload stub -->
<!--   - read flash -->

## Serial Protocol

- SLIP frames...



<!-- modifying the flash loader -->
<!-- - stub consists of 6 .c files: -->
<!--   - miniz.c         // compression -->
<!--   - slip.c          // slip -->
<!--   - stub_commands.c // handle commands -->
<!--   - stub_flasher.c  // main program -->
<!--   - stub_io.c       // serial communication -->
<!--   - stub_write_flash.c // write flash -->
<!-- - so we remove stub_write_flash.c and modify stub_commands.c -->
<!-- - additionally simplify the makefile -->
<!-- compiling and uploading the flash loader -->
<!-- - download toolchains -->
<!-- - compile the stub using specific toolchains -->
<!-- - this gives us an elf file -->
<!-- - use a python script to extract the .text and .data sections from the elf -->
<!-- - generate a header file and write the raw bytes to a `const unsigned char[]` -->
<!-- - this header gets compiled with the extraction tool (host) -->
<!-- - at runtime, after the chip is identified, upload .text and .data using MEM_ ram commands -->
<!-- - addresses for the sections and for the entry point are in elf file and get written to header alongside the elf sections -->
<!-- and how do we make the esp enter bootmode? -->
<!-- - wire two gpio pins to RST and GPIO0 -->
<!-- - pull both low -->
<!--   - RST low turns the esp off -->
<!--   - GPIO0 has to be low when it is turned back on -->
<!-- - pull RST high to turn it on -->
<!-- - pull GPIO0 high after the esp has started -->
<!-- overview: -->
<!-- - bootmode/serial -->
<!-- - serial protocol -->
<!-- - flash loader -->
<!-- differences between esp versions -->
<!-- - identification: -->
<!--   - ESP32-C3 and later use GET_SECURITY_INFO which contains a chip_id -->
<!--   - previous models have a register with a magic value identifying the chip -->
<!-- - mac address: -->
<!--   - different registers -->
<!--   - esp8266 mac has to be calculated -->
<!-- - different flash loader versions -->
<!--   - esp8266 has no data section -->
<!-- different hosts: -->
<!-- - linux (usb) -->
<!-- - rpi (gpio) -->
<!-- - esp (gpio) -->
<!-- extracting the file system -->
<!-- - esp8266 -->
<!-- - esp32 -->
<!-- interesting files -->
<!-- - wifi credentials -->
<!-- - certificates -->
<!-- - jwt token -->


## Demo


## cloc

## 

```sh
$ cloc esp-flasher-stub/
-----------------------------------------------------------
Language          files       blank     comment        code
-----------------------------------------------------------
Rust                 12         327          78        1863
Logos                14          32           0         249
YAML                  3          34          12         214
Markdown              1          34           0          89
TOML                  4           8           2          61
-----------------------------------------------------------
SUM:                 34         435          92        2476
-----------------------------------------------------------
```

## 

```sh
$ cloc esp-hal
-----------------------------------------------------------
Language          files       blank     comment        code
-----------------------------------------------------------
Rust                492       18739       26120      115809
Linker Script        51         499        1404       11315
Markdown             66        2014          12        5262
TOML                 43         505         412        4481
Logos                44         229          15        1105
YAML                 13         151          67        1054
Jinja Template        3          52           0         255
JSON                  2           0           0          48
CSV                   3           0           0          21
SVG                   1           0           0           4
-----------------------------------------------------------
SUM:                718       22189       28030      139354
-----------------------------------------------------------
```

## 

```sh
$ cloc esptool-legacy-flasher-stub/
-----------------------------------------------------------
Language          files       blank     comment        code
-----------------------------------------------------------
Linker Script        32         787        1188       18751
C                     6        1284         636        7689
C/C++ Header          8         463         557        1687
make                  1          36          33         130
YAML                  4          19           0         114
Python                2          25          21          80
Markdown              3          58           0          76
Bourne Shell          2           9           8          23
TOML                  1           1           0          20
Jinja Template        1           4           2          14
-----------------------------------------------------------
SUM:                 60        2686        2445       28584
-----------------------------------------------------------
```