diff options
| -rw-r--r-- | Makefile | 7 | ||||
| -rw-r--r-- | pres.adoc | 185 | ||||
| -rw-r--r-- | pres.md | 241 |
3 files changed, 248 insertions, 185 deletions
diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..9157f44 --- /dev/null +++ b/Makefile @@ -0,0 +1,7 @@ +all: + pandoc -t revealjs --standalone pres.md -o pres_md.html + +embed: + pandoc -t revealjs --standalone --embed-resources pres.md -o pres_md.html + +.PHONS: all embed diff --git a/pres.adoc b/pres.adoc deleted file mode 100644 index 14a6a0c..0000000 --- a/pres.adoc +++ /dev/null @@ -1,185 +0,0 @@ -= Ourosboros Flash Reader -:revealjsdir: ./reveal.js-5.2.1/ - -Patrick Schönberger - -16.07.2025 - -== Structure - -- Problem -- Solution -- Implementation - -== Problem - -== Solution - -== Implementation - -structure: start with a problem and reproduce the work leading to the working solution - -access smart home hw on crime scenes -many shelly devices are based on esp32/esp8266 -they dont contain any usable data but contain identifiable user data -this can be used to inquire about the user account -so we need to extract the content of the esp's flash memory -we then also need to extract filesystems from the memory -do it read-only, verifiably -existing solutions (esptool, mos) can also write and erase memory -they are also complex, making understanding and changing the code time consuming -what about writing a custom extraction tool? -what is the bare minimum needed to talk to the esp? -- two modes: bootmode and runmode -- decided by GPIO0 at start -- the esp as well as the shelly devices expose uart pins (tx/rx) -- in runmode they output logging information -- in bootmode they listen to a custom serial protocol -so we need a serial connection and the ability to enter boot mode! -what can the serial protocol do? -- sync -- write ram/flash/registers -- configuration etc. -- on the esp32 it can read flash, but not on esp8266, esp32c3, esp32c6 -how do other tools read flash? -- we cant directly read flash, but we can write ram -- write a program, load it into ram, run it and then talk to it instead -- flash loader/stub -- esptool uses two different variants, c based and rust based -- the c based one is older and getting replaced, but it is also dramatically simpler and also supports the esp8266 -- so we use the c based one and customize it (remove write and erase flash commands) -technically this means we do have write access until the flash loader is activated -the extraction tool is also small and runs a fixed number of commands --> as sure as we can be -how does the serial protocol work? -- data is encoded using SLIP frames -- the host sends a request and the target (esp) sends a response -- steps to read flash: - - sync - - identify chip - - read mac - - (change baud) - - upload stub - - read flash -modifying the flash loader -- stub consists of 6 .c files: - - miniz.c // compression - - slip.c // slip - - stub_commands.c // handle commands - - stub_flasher.c // main program - - stub_io.c // serial communication - - stub_write_flash.c // write flash -- so we remove stub_write_flash.c and modify stub_commands.c -- additionally simplify the makefile -compiling and uploading the flash loader -- download toolchains -- compile the stub using specific toolchains -- this gives us an elf file -- use a python script to extract the .text and .data sections from the elf -- generate a header file and write the raw bytes to a `const unsigned char[]` -- this header gets compiled with the extraction tool (host) -- at runtime, after the chip is identified, upload .text and .data using MEM_ ram commands -- addresses for the sections and for the entry point are in elf file and get written to header alongside the elf sections -and how do we make the esp enter bootmode? -- wire two gpio pins to RST and GPIO0 -- pull both low - - RST low turns the esp off - - GPIO0 has to be low when it is turned back on -- pull RST high to turn it on -- pull GPIO0 high after the esp has started -overview: -- bootmode/serial -- serial protocol -- flash loader -differences between esp versions -- identification: - - ESP32-C3 and later use GET_SECURITY_INFO which contains a chip_id - - previous models have a register with a magic value identifying the chip -- mac address: - - different registers - - esp8266 mac has to be calculated -- different flash loader versions - - esp8266 has no data section -different hosts: -- linux (usb) -- rpi (gpio) -- esp (gpio) -extracting the file system -- esp8266 -- esp32 -interesting files -- wifi credentials -- certificates -- jwt token - - -DEMO! - - -== cloc - -```sh -$ cloc esp-flasher-stub/ - 38 text files. - 34 unique files. - 5 files ignored. - -github.com/AlDanial/cloc v 2.04 T=0.02 s (2259.9 files/s, 199599.0 lines/s) ------------------------------------------------------------ -Language files blank comment code ------------------------------------------------------------ -Rust 12 327 78 1863 -Logos 14 32 0 249 -YAML 3 34 12 214 -Markdown 1 34 0 89 -TOML 4 8 2 61 ------------------------------------------------------------ -SUM: 34 435 92 2476 ------------------------------------------------------------ - -$ cloc esp-hal - 742 text files. - 718 unique files. - 35 files ignored. - -github.com/AlDanial/cloc v 2.04 T=0.36 s (1978.1 files/s, 522278.6 lines/s) ------------------------------------------------------------ -Language files blank comment code ------------------------------------------------------------ -Rust 492 18739 26120 115809 -Linker Script 51 499 1404 11315 -Markdown 66 2014 12 5262 -TOML 43 505 412 4481 -Logos 44 229 15 1105 -YAML 13 151 67 1054 -Jinja Template 3 52 0 255 -JSON 2 0 0 48 -CSV 3 0 0 21 -SVG 1 0 0 4 ------------------------------------------------------------ -SUM: 718 22189 28030 139354 ------------------------------------------------------------ - -$ cloc esptool-legacy-flasher-stub/ - 63 text files. - 60 unique files. - 4 files ignored. - -github.com/AlDanial/cloc v 2.04 T=0.06 s (952.9 files/s, 535446.8 lines/s) ------------------------------------------------------------ -Language files blank comment code ------------------------------------------------------------ -Linker Script 32 787 1188 18751 -C 6 1284 636 7689 -C/C++ Header 8 463 557 1687 -make 1 36 33 130 -YAML 4 19 0 114 -Python 2 25 21 80 -Markdown 3 58 0 76 -Bourne Shell 2 9 8 23 -TOML 1 1 0 20 -Jinja Template 1 4 2 14 ------------------------------------------------------------ -SUM: 60 2686 2445 28584 ------------------------------------------------------------ -``` @@ -0,0 +1,241 @@ +--- +title: Ourosboros Flash Reader +author: Patrick Schönberger +date: 16.07.2025 +transition: none +revealjs-url: https://cdn.jsdelivr.net/npm/reveal.js@5.2.1 +slideNumber: true +--- + +## Structure +<!-- structure: start with a problem and reproduce the work leading to the working solution --> + + +- Start with the problem +- Work our way to the solution step by step + + + +<!-- access smart home hw on crime scenes --> +<!-- many shelly devices are based on esp32/esp8266 --> +<!-- they dont contain any usable data but contain identifiable user data --> +<!-- this can be used to inquire about the user account --> +<!-- so we need to extract the content of the esp's flash memory --> +<!-- we then also need to extract filesystems from the memory --> +<!-- do it read-only, verifiably --> +<!-- existing solutions (esptool, mos) can also write and erase memory --> +<!-- they are also complex, making understanding and changing the code time consuming --> +<!-- what about writing a custom extraction tool? --> + +## The Problem + +- How to access data on Shelly devices? +- Most Shelly devices are based on ESPs +- They don't contain usable data like logs +- They do contain user-identifiable data + +## + +- Extract the flash memory and read the data +- Do it read-only and document the process +- Existing solutions (esptool, mos) can modify memory +- They are very complex, and thus hard to modify +- What about a custom tool? + + + +<!-- what is the bare minimum needed to talk to the esp? --> +<!-- - two modes: bootmode and runmode --> +<!-- - decided by GPIO0 at start --> +<!-- - the esp as well as the shelly devices expose uart pins (tx/rx) --> +<!-- - in runmode they output logging information --> +<!-- - in bootmode they listen to a custom serial protocol --> +<!-- so we need a serial connection and the ability to enter boot mode! --> +<!-- what can the serial protocol do? --> +<!-- - sync --> +<!-- - write ram/flash/registers --> +<!-- - configuration etc. --> +<!-- - on the esp32 it can read flash, but not on esp8266, esp32c3, esp32c6 --> + +## Talking to the ESP + +- What is the bare minimum needed to talk to an ESP? +- Two modes: boot and run +- Serial pins are exposed on some Shelly devices and accessible on others +- In boot mode they can be used to communicate using a custom protocol +<!-- * So we need a serial connection and the ability to enter boot mode --> + +## + +- What can we do with the protocol? + - Synchronize + - Write RAM, flash and registers + - Configure memory etc. + - On the ESP32 it can read flash, but not on the ESP8266, ESP32C3 or ESP32C6 +- How do other tools read flash? + + + +<!-- how do other tools read flash? --> +<!-- - we cant directly read flash, but we can write ram --> +<!-- - write a program, load it into ram, run it and then talk to it instead --> +<!-- - flash loader/stub --> +<!-- - esptool uses two different variants, c based and rust based --> +<!-- - the c based one is older and getting replaced, but it is also dramatically simpler and also supports the esp8266 --> +<!-- - so we use the c based one and customize it (remove write and erase flash commands) --> +<!-- technically this means we do have write access until the flash loader is activated --> +<!-- the extraction tool is also small and runs a fixed number of commands --> +<!-- -> as sure as we can be --> + +## Reading Flash Memory + +- We cannot directly read flash, but we can write RAM +- Write a program for the ESP, load it into RAM and run +- This is called the flash loader or stub loader +- Esptool uses two versions: + - Legacy C version + - New Rust version + +## + +- The C version is getting replaced but it is much simpler and it supports ESP8266 +- We customize this version by removing write and erase commands +- This way we technically have write access until the flash loader is activated +- The extraction tool is small and contains no write or erase commands + + + +<!-- how does the serial protocol work? --> +<!-- - data is encoded using SLIP frames --> +<!-- - the host sends a request and the target (esp) sends a response --> +<!-- - steps to read flash: --> +<!-- - sync --> +<!-- - identify chip --> +<!-- - read mac --> +<!-- - (change baud) --> +<!-- - upload stub --> +<!-- - read flash --> + +## Serial Protocol + +- SLIP frames... + + + +<!-- modifying the flash loader --> +<!-- - stub consists of 6 .c files: --> +<!-- - miniz.c // compression --> +<!-- - slip.c // slip --> +<!-- - stub_commands.c // handle commands --> +<!-- - stub_flasher.c // main program --> +<!-- - stub_io.c // serial communication --> +<!-- - stub_write_flash.c // write flash --> +<!-- - so we remove stub_write_flash.c and modify stub_commands.c --> +<!-- - additionally simplify the makefile --> +<!-- compiling and uploading the flash loader --> +<!-- - download toolchains --> +<!-- - compile the stub using specific toolchains --> +<!-- - this gives us an elf file --> +<!-- - use a python script to extract the .text and .data sections from the elf --> +<!-- - generate a header file and write the raw bytes to a `const unsigned char[]` --> +<!-- - this header gets compiled with the extraction tool (host) --> +<!-- - at runtime, after the chip is identified, upload .text and .data using MEM_ ram commands --> +<!-- - addresses for the sections and for the entry point are in elf file and get written to header alongside the elf sections --> +<!-- and how do we make the esp enter bootmode? --> +<!-- - wire two gpio pins to RST and GPIO0 --> +<!-- - pull both low --> +<!-- - RST low turns the esp off --> +<!-- - GPIO0 has to be low when it is turned back on --> +<!-- - pull RST high to turn it on --> +<!-- - pull GPIO0 high after the esp has started --> +<!-- overview: --> +<!-- - bootmode/serial --> +<!-- - serial protocol --> +<!-- - flash loader --> +<!-- differences between esp versions --> +<!-- - identification: --> +<!-- - ESP32-C3 and later use GET_SECURITY_INFO which contains a chip_id --> +<!-- - previous models have a register with a magic value identifying the chip --> +<!-- - mac address: --> +<!-- - different registers --> +<!-- - esp8266 mac has to be calculated --> +<!-- - different flash loader versions --> +<!-- - esp8266 has no data section --> +<!-- different hosts: --> +<!-- - linux (usb) --> +<!-- - rpi (gpio) --> +<!-- - esp (gpio) --> +<!-- extracting the file system --> +<!-- - esp8266 --> +<!-- - esp32 --> +<!-- interesting files --> +<!-- - wifi credentials --> +<!-- - certificates --> +<!-- - jwt token --> + + +## Demo + + +## cloc + +## + +```sh +$ cloc esp-flasher-stub/ +----------------------------------------------------------- +Language files blank comment code +----------------------------------------------------------- +Rust 12 327 78 1863 +Logos 14 32 0 249 +YAML 3 34 12 214 +Markdown 1 34 0 89 +TOML 4 8 2 61 +----------------------------------------------------------- +SUM: 34 435 92 2476 +----------------------------------------------------------- +``` + +## + +```sh +$ cloc esp-hal +----------------------------------------------------------- +Language files blank comment code +----------------------------------------------------------- +Rust 492 18739 26120 115809 +Linker Script 51 499 1404 11315 +Markdown 66 2014 12 5262 +TOML 43 505 412 4481 +Logos 44 229 15 1105 +YAML 13 151 67 1054 +Jinja Template 3 52 0 255 +JSON 2 0 0 48 +CSV 3 0 0 21 +SVG 1 0 0 4 +----------------------------------------------------------- +SUM: 718 22189 28030 139354 +----------------------------------------------------------- +``` + +## + +```sh +$ cloc esptool-legacy-flasher-stub/ +----------------------------------------------------------- +Language files blank comment code +----------------------------------------------------------- +Linker Script 32 787 1188 18751 +C 6 1284 636 7689 +C/C++ Header 8 463 557 1687 +make 1 36 33 130 +YAML 4 19 0 114 +Python 2 25 21 80 +Markdown 3 58 0 76 +Bourne Shell 2 9 8 23 +TOML 1 1 0 20 +Jinja Template 1 4 2 14 +----------------------------------------------------------- +SUM: 60 2686 2445 28584 +----------------------------------------------------------- +``` |