A Reset Without Witnesses

Notes from bringing up our own OS on a Samsung Galaxy Tab S6 (SM‑T865, Snapdragon 855 / SM8150). A continuation of the article about the "parasite" — the Wi‑Fi firmware that lives on the modem and makes demands of the host. This one is about what happens when a demand is not met, and how that can be seen at all.

0. The hook: the console went quiet

The handshake with the WLAN firmware is going to plan. MSA_INFO went out over QMI, the secure world transferred a memory region into the modem's ownership, CAP returned the firmware version, BDF_DOWNLOAD delivered five segments of board data, CAL_REPORT was accepted with status SUCCESS. All that remains is to wait for FW_READY_IND.

Instead, the console over USB goes quiet. A second later the tablet shows the bootloader logo and starts over. There is nothing in the log: the last line is about CAL_REPORT. The panic handler was not called. The exception vector did not fire. Not a single interrupt arrived. The watchdog core, which is supposed to describe a hang, left no verdict — it died together with the whole chip. The PMIC says exactly what it says after any ordinary reboot.

There were about ten such episodes during the Wi‑Fi bring‑up. The causes turned out to be different — cache maintenance, the heap, a power rail, an unenabled clock, a read four bytes past where it should have stopped. What they had in common was one thing: the SoC answers a host error not with an error but with a reboot, and leaves no evidence. This article is about how to prove a cause under such conditions anyway.

1. Why there are no witnesses

All the usual means of post‑mortem diagnostics rely on one of two things: either the kernel keeps executing at least a few instructions after the error (a panic, an exception handler, the watchdog core's verdict), or RAM survives the reboot (a log ring, ramoops). A reset from the secure world gives neither.

Who resets. Between our kernel (EL1) and memory on this SoC stand two foreign layers: Samsung's hypervisor (EL2, stage‑2 translation) and Qualcomm's TrustZone (EL3, XPU — hardware protection units over memory regions). A violation noticed by either of them — a write into a page TZ marked as its own, or a speculative read into another core's carve‑out — is handled not as an exception for the normal world but as a decision to reboot the whole chip. No trap is delivered to the normal world; at least, we have not seen a single such exception. The exact mechanism inside TZ is not available to us, so from here on "a reset from the secure world" is a description of observed behaviour, not a claim about Qualcomm's code.

DRAM does not survive. This was verified, not assumed: our own NOLOAD page of the image, the last 64 KiB of our RAM, free memory at 2.5 GiB, even Samsung's ramoops region — every boot read itself back in each of these places, and every one came back clean. The S‑Boot bootloader returns DRAM empty after any warm reset. Half a day of empty verdicts went into establishing this.

The PMIC's PON registers are not a witness. The PM8150's power‑on block honestly records why the chip turned on and how it last turned off: PON_REASON1, WARM_RESET_REASON1/2, POFF_REASON1/2, OFF_REASON, FAULT_REASON1/2, S3_RESET_REASON. The problem is that a reset from the secure world leaves in them the same picture as a PSCI SYSTEM_RESET, which we ourselves call routinely: hard reset as the power‑on reason, warm reset, an orderly shutdown sequence (POFF_SEQ). The FAULT_N bits in FAULT_REASON are present, but they are sticky — they hold until the next event of their kind — and belong to earlier episodes. In other words, PON distinguishes a button press from a battery drop and from the SoC pulling PS_HOLD itself, but does not distinguish "we asked for a reboot" from "we were rebooted".

There is no TrustZone log. Qualcomm has a TZ journal in IMEM, and there is a tz-log node in the device tree. On Samsung it is encrypted, and IMEM is wiped on reset. An attempt to read it through a debugging tool produced the eleventh episode instead of evidence — see § 3.

The watchdog catches hangs, not resets. The board has two watchdogs: the hardware APSS one (qcom,wdt), wound to 10 s before the device tree is read, and the software watchdog core — a second processor brought up via PSCI CPU_ON, which looks at the scheduler's gauge four times a second and, after 8 s of the same poll, writes a verdict, draws it on the panel and reboots the machine 30 s later. This works well when the main core is stuck on a bus that does not answer. But a reset is not a hang: there is nothing to observe and nobody to observe it.

Put together it looks like this: at the top — three who can reboot or power down the chip without reporting; in the middle — DRAM, which any of these resets wipes; at the bottom — the only chip the reset does not reach. Solid arrows are the mechanism, dashed ones are "can, if it wants to"; bold outlines are the nodes this article is built around. The full map of the device's tenants, with the same node names, is in the introduction, § 2a.

2. The witness we built

2.1 Where to store it

At the moment of a reset we have no log line, no register, no exception; the only thing that survives the event is what was written before it into memory the reset does not reach. The PMIC runs off the battery and passes through an SoC reset intact. What it holds:

SDAM (shared direct access memory) is a PMIC peripheral block that does nothing except store bytes; Android sees it as nvmem. Writes go over SPMI, 8 bytes per command, through an arbiter channel owned by our execution environment (if the channel is not ours, the word can be read but not written, and we have no witness — this is also checked at start‑up).

2.2 Three layers of words

In the 72 bytes of "last words" live records of two kinds, distinguished by the header magic:

• Death words (PANI): the panic text — file, line, the start of the message — or the watchdog core's verdict (hang: \call:x` polled 8 s at pc 0x…`). Written from the panic handler or from the watchdog core, without the heap (the panic may be the allocator's). The first words of a run stay: the watchdog's verdict after 8 s does not overwrite the panic that stopped the core in the first place.
• Mark (MARK): the name of a step that has started and has not yet finished. This is the witness of a silent reset.

Marks come in two levels, and both live in the scheduler's gauge (sched::gauge), not in the platform:

• gauge::marked("pas auth") — a guard for a short step that may hang the core with interrupts masked: a call into the secure world over smc, the first read of SMEM, a register of a block whose clock may be off. While the guard is alive, the word is its name; on drop the mark is cleared. Marked this way are scm detect, scm pas, pas init, pas place, pas auth, pas shutdown, qmp open, qmp load_state, smem toc, smem header, smem items, smp2p edge, glink attach, glink poll, glink send.
• gauge::Phase::begin("wlfw ind_register") / phase.set("wlfw cap") — the phase of a long operation that stretches across many awaits and scheduler polls: the WLFW handshake, powering on the target. The operation itself sets and advances it; drop clears it. Marks nest on top of a phase: while a guard is alive, the word is the mark; when it is released — the phase again.

Both are a pointer and a length to a &'static str in atomics plus one listener (gauge::on_mark). The board subscribes persist::mark to it: on setting a word — an SPMI write of the header and the name, on clearing — eight zero bytes. One or two SPMI writes per step, +1.5 KiB to the image of the microcontroller board, where Phase is also compiled in. That is the whole price.

2.3 What is read on the next boot

persist::open runs first thing after the SPMI arbiter is up: it reads PON, the crumb, SDAM, immediately clears the words record (so the verdict comes out once) and hands everything to crash::death. That one assembles the verdict from three sources:

• there are death words → the cause is panic or watchdog, the text is as left;
• no words, the crumb is Ready, PON says nothing about buttons or PS_HOLD → reset: "the last run was up when it ended, and left no words";
• and in any case, if a mark is set → the verdict is extended with: "it was in wlfw fw_ready wait — a step marked as one that can stall the core …, and the mark was never cleared".

The answer is visible in sys.boot (last_reset, previous {reason, died_at, step, words}) and in the sys.crashes history. The first command after every reboot in this work is sys.boot.

2.4 What a phase proves and what it does not

It proves one thing: a step with this name started, and its end was not recorded before the reset. The resolution is one step: if the phase stands at wlfw fw_ready wait, that is the step after the last request — which means all our commands to the firmware had already gone out and been accepted, and the reset arrived while we were waiting for its answer. This shifts suspicion from "we sent something wrong" to "the firmware did something at our request and hit an obstacle". In case #8 of the table below, exactly this logic led to the clocks.

It does not prove who reset and why. A mark does not distinguish "TZ rebooted on this step" from "the core hung on this step and 10 s later the hardware watchdog bit" — here PON helps (the PS_HOLD bit in WARM_RESET_REASON1 is a sign that the SoC pulled the line itself), but, as said above, we cannot rely on it for a reset from the secure world. Nor does it speak about delay: reset #2 arrived 6–10 s after the step that caused it, and no phase was set at that moment — only the Ready crumb. The witness says where we were, not what happened.

3. The resets, one by one

The table below is the centre of this article. A caveat: not all of its rows are SoC resets. Five of the ten (#1, 2, 3, 8, 9) are reboots of the whole chip without words; #4 is a panic on our side plus a modem restart; #5, 6 are modem deaths (also without words: before its error service starts, the reason is not published); #7 is the only case where the secure world refused rather than reset; #10 is a risk closed before it manifested. They are gathered together because the method is one.

3.1 Cache and MMU: #1, 4, 10

The common root: the kernel's identity map divides the gigabyte with RAM into 2 MiB blocks with different cacheability. Everything another master reads or writes — the modem via DMA, the secure world, the hypervisor — must lie in a Normal Non‑cacheable or Device block, and the whole block. Reset #1 taught us not to touch the cache over shared memory at all (Linux here does no-map + ioremap_wc — the same thing in other words). Reset #4 showed that "the window is non‑cacheable" means nothing until the window starts on a block boundary: half the buffer ended up in the neighbouring cacheable block, the modem read from DRAM what we were still holding in cache — zeros — and behaved rationally: it restored EFS from the "golden" copy. The symptom was two layers above the cause. #10 is a conclusion from the first two, drawn before a third reset: the tree's no-map is translated to Device, because A76 prefetch into a Normal region is a read, and the XPU does not ask whether it was intentional.

3.2 Memory ownership: #2, 7, 9

Here the adversary is the XPU. There is one rule: everything the secure world reads or guards lies outside the heap and outside windows we reuse. #2 is the hardest of the ten: the reset is separated from its cause by 6–10 seconds and several allocations, and there are no marks on it because all PAS steps formally completed. The reference helped: in Linux the PAS metadata is released after auth_and_reset, and that is no accident — TZ keeps reading it for seconds after returning. #7 is a rare case of an honest refusal: the wrong MSA address was hypervisor memory, and assign_mem returned an error instead of a reset. A general conclusion was drawn from this: region addresses are taken only from the live device tree, because they drift between firmware releases. #9 is a reset caused by a debugging tool: tz-log claims 12 KiB where IMEM has 4, and a read past the end of the block is also a violation. The tool now checks the parent's boundary.

3.3 Power and clocks: #3, 8

These two are about the modem firmware and the SoC holding resources through RPMh and the AOP by voting, with the host being one of the voters. #3: by withdrawing our vote for cx.lvl, we overwrote the vote XBL had left on our behalf; the system lived exactly as long as the modem was busy and voting on its own. The symptom here is different: it is not a reset from TZ but a stall of the interconnect — both cores go silent, the hardware watchdog bites. But from the evidence point of view there is no difference: "no marks, no words, PMIC clean". #8 is the case the phase in SDAM was written for. Without it, the reset after CAL_REPORT would have looked like a reset "somewhere in the handshake"; the word wlfw fw_ready wait said that all our commands had already been accepted. The bisection tools for this spot: wifi.power {on, until: ind_register | cap | bdf} stops the handshake after the named step (cap — without board data and CAL_REPORT); remove elf_loader from the facts — the WLAN PD does not start, and if the reset disappears, the culprit is on the WLFW path. The picture came together like this: after CAL_REPORT the firmware initialises the radio and touches a block whose clock the host did not enable; an access to an unclocked block is a bus error, and the SoC goes into reset — reproducibly, with any MSA size and with the right address. The snoc.c reference names two clocks; after enabling them, FW_READY_IND arrived ~8 s after modem start.

3.4 Answers to the modem: #5, 6

Strictly speaking, these are not SoC resets — the modem dies. But it dies just as silently: the cause of death is published in SMEM by its error service, which at these stages is not yet running, so a fatal without a reason always means "early". The witness here is different — the timeline of the modem's requests to our servers (ipc.hosts): what it asked last and how many milliseconds later it died. 23 ms between the refusal on fsg and fatal is not a coincidence, it is a reaction.

4. Lessons

Every shared window is non‑cacheable and on a 2 MiB boundary. Three of the ten rows are about this. It is checked by const asserts at build time and by debug_assert in the MMU setup; an alignment error can no longer reach the board.

The SoC punishes deviation with a reset, not an error. Of the ten cases, only one (#7) produced an error code. Hence a consequence for the method: hypotheses are tested not by a message that will not come, but by difference — two control builds with one variable removed (#1), correlation with the modem's state (#3), stopping the chain after a step (#8). One build is about two minutes; so a hypothesis is worth testing with the cheapest edit before building the target form.

The reference is not documentation but a protocol. In four cases (#2, 3, 6, 8) the answer was in the Linux sources — not in the comments but in the order of actions: when the metadata is released, which votes are withdrawn on handover, which clocks are enabled before CAL_REPORT. The order there was paid for with the same resets.

A word about the phase is cheaper than a debugger. JTAG is unavailable on a consumer tablet, the TZ log is encrypted, DRAM does not survive. For fifteen hundred bytes of code and one SPMI write per step we got the only source that answers the question "where were we". It does not replace reasoning — but without it, #8 would have had to be bisected blind across a dozen handshake steps, two minutes per build.

An honest list of what cannot be seen. The PMIC LDOs are read only as an observer (the channels belong to the AOP), the TZ log is encrypted, IMEM is wiped on reset, the QDSP6SS registers under PAS are a bus error on read, the SMMU globally belongs to the hypervisor. Diagnostics is designed from this list, not from what one would like to have.

5. The reset as a way of testing the landlord

In the article about the parasite the Wi‑Fi firmware is described as a tenant with a landlord's rights: it demands memory, files, disk, IPC, seven rails and two clocks from the host, and it cannot be refused. The resets in this article are what the checking of those demands looks like. It does not arrive as an error code that can be handled. The host either met the conditions — and then eight seconds later FW_READY_IND arrives — or the chip reboots, and the landlord is invited to guess for himself what he did wrong.

The mark in the PMIC is a way to at least remember on which question we were rebooted. The answers themselves — the memory map, the servers, the rails — are described in the neighbouring articles (The Megabyte That Killed the Modem, 23 Milliseconds to Lie, The Hypervisor Said No); the map of all the device's tenants is in the introduction. What matters here is something else: a modern SoC has a layer that makes decisions about the host and does not report to it, and the only strategy that works with this is to write a word before every step rather than hope to read something after.

Sources. The reset table, memory map and methodology — docs/wifi-sm8150.md (§ 0, § 2, § 7, § 8); the limits of the observable — docs/wifi-threat-model.md § 8; marks and phases — crates/kernel/src/sched.rs (gauge); the listener, the SDAM layout and reading at boot — platforms/sdm855/src/persist.rs; the SDAM and PON blocks — crates/qcom/src/sdam.rs, crates/qcom/src/pon.rs; the two watchdogs — platforms/sdm855/src/watchdog.rs; assembling the verdict — crates/tools/src/crash.rs, crates/tools/src/crash_crumbs.rs; sys.boot — crates/tools/src/sys.rs; WLFW phase names — crates/tools/src/wifi/wlfw.rs; the live run of 18–19.09 — TASKS.md, line 87. The PAS/SMEM/GLINK step marks — platforms/sdm855/src/{remoteproc,ipc}.rs.*

Recommended

VPN Chrome extension: what it protects and what it doesn't

A Chrome VPN extension is a proxy for browser traffic only. What it really protects, where it leaks, and when you need a full VPN app instead.

Turnkey smart homes on our own OS: 0xd-os and 0xd-ui

We build turnkey smart homes on components we designed ourselves. At the core is 0xd-os — one operating system that runs identically on a flagship tablet and on a device with 512 KB of memory.

Why security agents need a sandbox

An agent that can 'just run tools' on a security review is a second attacker — unless every call has a grant. AGI Core isolates tool execution; Forge keeps the skill list in UIDE.