path: root/pres.md

---
title: Ouroboros Flash Reader
author: Patrick Schönberger
date: 16.07.2025
transition: none
revealjs-url: https://cdn.jsdelivr.net/npm/reveal.js@5.2.1
slideNumber: true
navigationMode: linear
---

## Structure
<!-- structure: start with a problem and reproduce the work leading to the working solution -->


- Start with the problem
- Work our way to the solution step by step

::: notes

This is my note.

- It can contain Markdown
- like this listt

:::



<!-- access smart home hw on crime scenes -->
<!-- many shelly devices are based on esp32/esp8266 -->
<!-- they dont contain any usable data but contain identifiable user data -->
<!-- this can be used to inquire about the user account -->
<!-- so we need to extract the content of the esp's flash memory -->
<!-- we then also need to extract filesystems from the memory -->
<!-- do it read-only, verifiably -->
<!-- existing solutions (esptool, mos) can also write and erase memory -->
<!-- they are also complex, making understanding and changing the code time consuming -->
<!-- what about writing a custom extraction tool? -->

## The Problem

How to access data on Shelly devices?

## 

- Extract the flash memory and read the data
- Do it read-only and document the process

## 

- Most Shelly devices are based on ESP micro controllers
- Existing solutions (esptool, mos) can modify memory
- They are very complex, and thus hard to modify
- What about a custom tool?



<!-- what is the bare minimum needed to talk to the esp? -->
<!-- - two modes: bootmode and runmode -->
<!-- - decided by GPIO0 at start -->
<!-- - the esp as well as the shelly devices expose uart pins (tx/rx) -->
<!-- - in runmode they output logging information -->
<!-- - in bootmode they listen to a custom serial protocol -->
<!-- so we need a serial connection and the ability to enter boot mode! -->
<!-- what can the serial protocol do? -->
<!-- - sync -->
<!-- - write ram/flash/registers -->
<!-- - configuration etc. -->
<!-- - on the esp32 it can read flash, but not on esp8266, esp32c3, esp32c6 -->

## Talking to the ESP

- What is the bare minimum needed to talk to an ESP?
- Two modes: boot and run
- Serial pins are exposed on some Shelly devices and accessible on others
- In boot mode they can be used to communicate using a custom protocol
<!-- * So we need a serial connection and the ability to enter boot mode -->

## 

- What can we do with the protocol?
  - Synchronize
  - Write RAM, flash and registers
  - Configure memory etc.
  - On the ESP32 it can read flash, but not on the ESP8266, ESP32C3 or ESP32C6
- How do other tools read flash?



<!-- how do other tools read flash? -->
<!-- - we cant directly read flash, but we can write ram -->
<!-- - write a program, load it into ram, run it and then talk to it instead -->
<!-- - flash loader/stub -->
<!-- - esptool uses two different variants, c based and rust based -->
<!-- - the c based one is older and getting replaced, but it is also dramatically simpler and also supports the esp8266 -->
<!-- - so we use the c based one and customize it (remove write and erase flash commands) -->
<!-- technically this means we do have write access until the flash loader is activated -->
<!-- the extraction tool is also small and runs a fixed number of commands -->
<!-- -> as sure as we can be -->

## Reading Flash Memory

- We cannot directly read flash, but we can write RAM
- Write a program for the ESP, load it into RAM and run
- This is called flash loader or flasher stub
- Esptool uses two versions:
  - Legacy C version
  - New Rust version

## 

- C version is deprecated but it is much simpler and supports ESP8266
- Customize this version by removing write and erase commands
- Technically there is write access until the flash loader is activated
- The extraction tool is small and contains no write or erase commands

## 

```txt
$ cloc esp-flasher-stub/
-----------------------------------------------------------
Language          files       blank     comment        code
-----------------------------------------------------------
Rust                 12         327          78        1863
Logos                14          32           0         249
YAML                  3          34          12         214
Markdown              1          34           0          89
TOML                  4           8           2          61
-----------------------------------------------------------
SUM:                 34         435          92        2476
-----------------------------------------------------------
```

## 

```txt
$ cloc esp-hal
-----------------------------------------------------------
Language          files       blank     comment        code
-----------------------------------------------------------
Rust                492       18739       26120      115809
Linker Script        51         499        1404       11315
Markdown             66        2014          12        5262
TOML                 43         505         412        4481
Logos                44         229          15        1105
YAML                 13         151          67        1054
Jinja Template        3          52           0         255
JSON                  2           0           0          48
CSV                   3           0           0          21
SVG                   1           0           0           4
-----------------------------------------------------------
SUM:                718       22189       28030      139354
-----------------------------------------------------------
```

## 

```txt
$ cloc esptool-legacy-flasher-stub/
-----------------------------------------------------------
Language          files       blank     comment        code
-----------------------------------------------------------
Linker Script        32         787        1188       18751
C                     6        1284         636        7689
C/C++ Header          8         463         557        1687
make                  1          36          33         130
YAML                  4          19           0         114
Python                2          25          21          80
Markdown              3          58           0          76
Bourne Shell          2           9           8          23
TOML                  1           1           0          20
Jinja Template        1           4           2          14
-----------------------------------------------------------
SUM:                 60        2686        2445       28584
-----------------------------------------------------------
```



<!-- how does the serial protocol work? -->
<!-- - data is encoded using SLIP frames -->
<!-- - the host sends a request and the target (esp) sends a response -->
<!-- - steps to read flash: -->
<!--   - sync -->
<!--   - identify chip -->
<!--   - read mac -->
<!--   - (change baud) -->
<!--   - upload stub -->
<!--   - read flash -->

## Serial Protocol

- Bytes are grouped using SLIP frames
- Host sends requests containing commands
- Target sends a matching reply

::: notes
Exceptions:

- after loading flash loader it sends a slip frame on its own
- after sync request 8 replies are sent
- when reading flash using 0xd2 replies are not slip/request frames
:::

## Steps to Read Flash

- Sync
- Identify Chip
- (Read MAC)
- (Change baud rate)
- Upload Stub
- Read Flash

::: notes
Starting from boot mode
sync always at 115200 but esp detects other baud rates from known payload
:::



<!-- modifying the flash loader -->
<!-- - stub consists of 6 .c files: -->
<!--   - miniz.c         // compression -->
<!--   - slip.c          // slip -->
<!--   - stub_commands.c // handle commands -->
<!--   - stub_flasher.c  // main program -->
<!--   - stub_io.c       // serial communication -->
<!--   - stub_write_flash.c // write flash -->
<!-- - so we remove stub_write_flash.c and modify stub_commands.c -->
<!-- - additionally simplify the makefile -->

## Modifying the Flash Loader

- Consists mostly of six C files:
  ```txt
  miniz.c             // compression
  slip.c              // slip
  stub_commands.c     // handle commands (modify)
  stub_flasher.c      // main program
  stub_io.c           // serial communication
  stub_write_flash.c  // write flash (remove)
  ```



<!-- compiling and uploading the flash loader -->
<!-- - download toolchains -->
<!-- - compile the stub using specific toolchains -->
<!-- - this gives us an elf file -->
<!-- - use a python script to extract the .text and .data sections from the elf -->
<!-- - generate a header file and write the raw bytes to a `const unsigned char[]` -->
<!-- - this header gets compiled with the extraction tool (host) -->
<!-- - at runtime, after the chip is identified, upload .text and .data using MEM_ ram commands -->
<!-- - addresses for the sections and for the entry point are in elf file and get written to header alongside the elf sections -->

## Uploading the Flash Loader

- Compile using specific toolchains
- Extract text and data sections from resulting ELF file
- Generate header file that is compiled with the extraction tool
- This includes section's sizes, addresses and the entry point
- Upload the corresponding loader after identifying the chip



<!-- and how do we make the esp enter bootmode? -->
<!-- - wire two gpio pins to RST and GPIO0 -->
<!-- - pull both low -->
<!--   - RST low turns the esp off -->
<!--   - GPIO0 has to be low when it is turned back on -->
<!-- - pull RST high to turn it on -->
<!-- - pull GPIO0 high after the esp has started -->

## Boot Mode

- GPIO0 low at start
- Wire pins to RST and GPIO0
- Pull both low
- Then pull RST high first and GPIO0 after
- Supported when using Raspberry Pi or ESP as host

::: notes
For development also possible via USB on linux hosts
:::



<!-- overview: -->
<!-- - bootmode/serial -->
<!-- - serial protocol -->
<!-- - flash loader -->

## Summary

- Connect to serial and set to boot mode
- Use serial protocol to identify the chip
- Upload modified flash loader
- Extract

::: notes
what is left:

- differences between esp versions
- how to extract files
- what is on the files?
:::



<!-- differences between esp versions -->
<!-- - identification: -->
<!--   - ESP32-C3 and later use GET_SECURITY_INFO which contains a chip_id -->
<!--   - previous models have a register with a magic value identifying the chip -->
<!-- - mac address: -->
<!--   - different registers -->
<!--   - esp8266 mac has to be calculated -->
<!-- - different flash loader versions -->
<!--   - esp8266 has no data section -->

## Different ESP Versions

- ESP32-C3 and later are identified using `GET_SECURITY_INFO` command
- Earlier versions have a register with a magic value
- Try security info first and reset on failure
- Different registers containing MAC address

::: notes
- sec info first bc it fails with unknown command
- read register is a known command
- reset -> bootmode -> sync again
- on ESP8266 mac has to be calculated
- esp8266 has no data section when compiling flash loader
:::



<!-- extracting the file system -->
<!-- - esp8266 -->
<!-- - esp32 -->

## Extracting File Systems

- SPIFFS/Littlefs
- Gen 1 devices have known layout
- On newer devices file system is found using metadata
- Files can be extracted using specialized tools



<!-- interesting files -->
<!-- - wifi credentials -->
<!-- - certificates -->
<!-- - jwt token -->

#### Shelly Plus 1 File System

```txt
api_math.js
ca.pem                // signing CA
conf0.json            // empty configuration
conf3.json            // initial configuration
conf9.json            // current configuration
index.html.gz         // web frontend
init.js
rpc_acl_auth.json     // list of rpc endpoints
rpc_acl_no_auth.json
shelly_cloud.pem      // Shelly Cloud certificate
shelly_plugin_api.js
storage.json          // user data
tzinfo
updater.dat
```

#### `conf9.json`

```txt
{
  "wifi": {
    "sta": {
      "enable": true,
      "ssid": "SSID",
      "pass": "PASSWORD"
      "last_bssid": "aa:17:5d:15:ae:03",
    },
  },
  "shelly": {
    "cloud": {
      "enable": true,
      "server": "192.168.112.231:6022/jrpc",
      "token": "eyJhbGciOiJIUzI1NiIsInR..."
    },
  },
  ...
}
```

#### `shelly_cloud.pem`

```txt
Subject: O = Allterco
Not Before: Aug  4 12:03:41 2020 GMT
Not After : Aug  2 12:03:41 2030 GMT
-----BEGIN CERTIFICATE-----
MIICrTCCAZUCFCuIEEAQJOFLZuEtr/CWkvxi9YPAMA0GCSqGSIb3DQEBCwUAMBMx
ETAPBgNVBAoMCEFsbHRlcmNvMB4XDTIwMDgwNDEyMDM0MVoXDTMwMDgwMjEyMDM0
MVowEzERMA8GA1UECgwIQWxsdGVyY28wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw
ggEKAoIBAQDHGDBDHpPUbtC9QAAjX3bi487AVY5JgYB2gyp6R9cjdsNGMbYnWdxn
BmsIUKJPg7B5NcQObiMe6djUvwo0c2Xl9L+P9LOskP2WNDdpquX3XJu580hXHHVB
mwOgJ0fi+5U9mOFHhc1gYGLmhO9oqsE80SgpmsPQHloMIqmcaolLzgC9PWGu8nSD
ToJq+dXyNFHzLVyBEugHQpeIR8Fq0do4dtlsfTWvv9U+fpGPegjdkPenSxGrOVwd
syFzNahxQGKmpZE/1fsq5QSh9+ZgwpdDChVNpkj9TBC1ApDTUasNco/6Meb/0Xur
pxpWPNfkIpZ7ebtGHVd/ZkGTPUnL7FXHAgMBAAEwDQYJKoZIhvcNAQELBQADggEB
ADwbvD7Mf7SOinV8JkOue8D/tvp+OiYTYLHYppzCLcBK3D1kQ7aqla2T8ebEFbLh
hpau7MxJcizVWZs4vJvFYxEwBTQldobhmG5lHnoKQcOSxis1kWr5xdKhji/QYo6T
wS/cird9hAcuc+RmLCMdpEDyia/vX+vvvRdyKsmB7A6Vkdu8s2B2jlhQNkitYzvq
UDKogJrWe6fQUTpTThMyGbqhp9cQ64M4DJG1cwSBZ/hiUAMKO/y5WVNWFbXIb/Om
xwkXgof2RXN1AjjqMcBh3GNVK4ZV5XM9WCocZjOCi2yZxaxaRWyuGR7EqAQZ+wdr
P5XrzYspeVl1WtBzwGFssPc=
-----END CERTIFICATE-----
```

## Thank You