Notes from bringing up our own OS on a Samsung Galaxy Tab S6 (SM‑T865, Snapdragon 855). A continuation of "The Parasite in the Tablet": that one was about who the tenant with the keys to the house is; this one is about a single wall between us, which we put a metre to the left of where it should have been, and what came of it.
0. What this is about
On a SoC with a cellular modem, a secure world and a hypervisor, our OS is not the only owner of RAM. Part of the RAM we are obliged to lend out: to the WLAN firmware — for descriptor rings and DMA buffers; to the modem — for the buffer of its file system; to TrustZone — for the metadata of the signed image it verifies. Part of the RAM does not belong to us at all: the hypervisor, XBL, the AOP, the carve‑outs of other cores. And each of these neighbours has its own idea of what lies at an address — an idea our cache is not part of.
Hence something you have nowhere to get used to on a single‑processor breadboard: cacheability attributes and window alignment in the translation table are not performance settings but conditions of correctness and, on this SoC, of security. We paid for this with several silent resets of the whole chip; one is worth telling in detail, because its symptom was as far from its cause as it ever gets.
1. The symptom: the modem repairs what we honestly gave it
The modem keeps its persistent storage — EFS — in the modemst1, modemst2 and fsg (golden copy) partitions on our disk, and reaches them through the rmtfs server that the host is obliged to run. Our server hands the modem a shadow: on OPEN the partition is read into RAM once, sector transfers (RW_IOVEC) go to the copy, the disk does not change. Transfers pass through a shared buffer — 2 MiB that the modem requests from the host with ALLOC_BUFF and then reads by its own path.
How the modem reads EFS is visible in the server log: one sector at offset 512 from each of fs1, fs2 and fsg — the headers — then a full read of the chosen copy. In one run the picture became this: the modem deemed the fs1 and fs2 headers empty, chose fsg (which does not exist on this tablet and opens as zeros — an erased golden copy), restored EFS from scratch, wrote full images into fs1 and fs2 — three full writes in its lifetime — activated the MCFG configuration and ~20 s later fell over with "Modem Graceful Restart failed". (In the same run the kernel showed panic: mutex locked pre-entrantly; the connection between the panic and the cause below is not worked out in our notes.)
Everything looked like an rmtfs bug: a wrong sector, a wrong offset, a broken shadow. The shadow turned out to be valid, the offsets correct. The modem really did see zeros where we really had written the headers.
2. The cause: a megabyte that stayed cacheable
Our identity map is simple: L1 by the gigabyte, and the gigabyte with the shared windows is split into 2 MiB blocks, each with its own attribute. Windows that someone other than us writes to or reads from are Normal Non‑cacheable: every load and store goes to memory, none of our cache lines lives there, there is nothing to maintain and no need. That is how SMEM is mapped (in Linux — ioremap_wc), and that is how the lent RAM is mapped.
The lent RAM (LENT) at that stage started at 0x8530_0000. That is not a block boundary: 0x8530_0000 is exactly the middle of the block 0x8520_0000..0x8540_0000. The map is built by blocks: for each block, its base is taken and a window that contains it is looked up; the base 0x8520_0000 fell into no window — and the block stayed what everything around it was: Normal write‑back, the cacheable RAM of the heap. The first megabyte of the rmtfs buffer ended up behind the cache, the second half in memory.
From there everything is honest on both sides. The host copied sectors into the buffer with ordinary stores; the buffer was considered non‑cacheable, so the server got the "empty" coherence (Coherent, nothing to clean) — and part of the stores settled in the core's write‑back cache. The modem read memory by its own path (Q6 → XPU, bypassing apps_smmu) and saw what lay in DRAM: zeros. The fs1 and fs2 headers are in the first megabyte. The rest is § 1.
The cure is one line: LENT onto a block boundary. The rmtfs buffer moved to 0x8520_0000, where it has stayed in the current map, and the run with the aligned buffer showed modemst1 r1, modemst2 r2, fsg r1, w0: one header read each, one read of the copy, not a single write.
3. Why exactly 2 MiB
Because there are two such units in our map: the gigabyte and 2 MiB. One L1 table, one L2 table for the gigabyte that is allowed to be shared, no 4 KiB pages. This is deliberate poverty: the map is built before the heap, in the first seconds of boot, and must be understandable as a whole. The price is that a window is mapped exactly only if it starts and ends on a 2 MiB boundary. A window started "in between" is mapped partly as its surroundings, and does so silently: in a debug build enable_identity catches this with a debug_assert, in release it maps as it can.
The same property works in the other direction. The tree's reserves (no-map) start with hyp_mem at 0x8570_0000 — also the middle of a block. We map them Device from 0x8560_0000: a megabyte of real RAM given away, because the alternative — leaving the hypervisor's block Normal — is worse (§ 5.3). "The 2 MiB boundary" is not only "our window is entirely non‑cacheable", but also "nothing of ours lies in the same block as something foreign".
4. The map
Addresses are from soc.rs; the heap ends where the lent RAM begins (HEAP_END = LENT.0).
address window MMU attribute who else touches it
-----------------------------------------------------------------------------------------------
… heap Normal WB nobody, only us
0x84e0_0000 +- LENT, 8 MiB: our RAM lent to others ---------------------------------------------+
| LENT_WLAN 4 MiB Normal Non-cacheable WLAN copy engines, DMA by phys. |
| addresses (stage-1 bypass) |
0x8520_0000 | LENT_RMTFS 2 MiB Normal Non-cacheable modem: rmtfs buffer, hlos+mss_msa |
0x8540_0000 | LENT_PAS 2 MiB Normal Non-cacheable secure world: XPU over metadata |
0x8560_0000 +- RESERVED: the tree's no-map, up to 0xa4c0_0000 ---------------------------------+
| the given-away MiB Device nobody — the price of the boundary|
0x8570_0000 | hyp_mem … Device hypervisor |
0x85e0_0000 | XBL_AOP 2 MiB Uncached (cut-out) XBL, AOP; CMD_DB @ 0x85f2_0000 |
0x8600_0000 | SMEM 2 MiB Uncached (cut-out) modem, secure world, hypervisor |
… | core carve-outs Device their cores, TZ |
0x8d80_0000 | MPSS 160 MiB Device; Uncached while modem's Hexagon, TZ |
| the image is being loaded |
0x9c40_0000 | SPLASH 32 MiB Normal WB (cut-out) display: our frame, clean |
0xa4c0_0000 +----------------------------------------------------------------------------------+The order of windows in the list matters — the first window containing an address wins — so the cut‑outs (SMEM, XBL_AOP, SPLASH) are declared before the RESERVED they are part of. Three attributes — three relationships with a neighbour: Normal WB — "only us here"; Non‑cacheable — "others write here and read from here, a cache between us is unacceptable"; Device — "not ours, the processor does not look in here without a command".
The table answers "where"; "who, and by what path" is answered by the graph below: the same windows, and for each one all the masters that write into it or read from it, and the path by which they do it. A solid arrow is a mechanism; a dashed one is a right a neighbour exercises when it sees fit: reading the metadata after we decided the job was done, a speculative load through a fence, a reset. A bold frame marks this article's windows; the full map of relations of the whole series is in Whose Computer Is This?, § 2a.
5. Three other resets with the same root
The summary table of Wi‑Fi bring‑up resets has ten rows; four of them are about the memory map. The one described above is row 4. The other three.
5.1 Cache maintenance over a shared window (row 1)
The first version mapped SMEM cacheable and, as is proper for DMA memory, cleaned and invalidated the cache by address (dc cvac/dc civac). ~10 s after the first remoteproc.list the SoC reset — from the secure world, without a word, even while idle. Proven by two control builds: without the mutex and without calling the modem — dies; without cache maintenance — lives. The exact mechanism — how dc civac over a region that also belongs to the secure world and the hypervisor causes a reset — we did not establish; what we did establish is that the reference does it differently: SMEM in Linux is no-map plus ioremap_wc, there is no cache there at all. We did the same: SMEM is a Non‑cacheable window, qcom::smem gets Coherent (a coherence that does nothing). The lesson is wider than SMEM: "a cacheable window plus careful maintenance", for memory guarded by someone else, is here not equivalent to a non‑cacheable window.
5.2 PAS metadata on the heap (row 2)
The modem is loaded through PAS: the host gives the secure world the image metadata (init_image), writes the segments into the carve‑out, calls auth_and_reset. The first version kept the metadata in a Vec on the heap — a buffer that is logical to free after a successful auth_and_reset. The reset arrived 6–10 s after modem start, with the application processor idle, without marks.
The cause: TrustZone puts an XPU over the metadata and reads it while authorising the segments and after auth_and_reset returns — seconds after the host considered the job done. The heap reused the freed buffer; a write into it became a write into a protected page — and a reset. In the reference there is a trace of the same: qcom_scm_pas_metadata_release is called after auth (since Linux 5.18), not earlier.
The cure is a window of its own: LENT_PAS, 2 MiB, Non‑cacheable, outside the heap. The metadata is copied there, the secure world gets that address, and until loading completes nobody else writes into that window. Common with § 2: memory that another master reads lies neither where it is reused nor behind our cache.
5.3 Speculation through the fence (row 10)
This row is the only one in the table marked "not observed, closed preventively", and we honestly repeat that. A Normal mapping — even Non‑cacheable — permits the processor to read speculatively: the core may execute a load ahead of the program, at an address the program perhaps will never go to. If that address is behind the secure world's XPU fence or behind the hypervisor's stage‑2, the answer to such a read is a bus error, and on this SoC a bus error from the normal world means a reset at a moment the program does not choose. Hence the rule: everything the tree marks no-map is Device for us — memory the processor does not read ahead into. And hence the megabyte given away before hyp_mem: the hypervisor's block must not be Normal by a single byte.
The rule has a dynamic half. The modem carve‑out (MPSS, 160 MiB) is Device from boot: not ours. While the image is being written it is opened Non‑cacheable (remap on a running MMU, TLB invalidation broadcast), the segments are written straight into memory, and after auth_and_reset the window is closed back to Device. It can be closed only because it was opened Non‑cacheable: the window has no dirty lines that would have to be flushed before changing the attribute. The conditions of remap are the same as for the whole map: 2 MiB boundaries, nothing of ours inside.
6. The invariant and how it is checked at compile time
After four resets the rule formulated itself briefly: nothing of ours shares either a cache line or a 2 MiB block with anything foreign. Expanded:
- Memory that another master (the modem, the WLAN target, the secure world) writes or reads is Normal Non‑cacheable, in its own window, outside the heap.
- Memory that does not belong to us (
no-map) is Device; it is opened Non‑cacheable only while we are writing into it, and closed back. - Every window starts and ends on a 2 MiB boundary.
- The heap ends where the first lent window begins.
Points 3 and 4 are checked before launch, by the compiler. In soc.rs, next to the map constants, stands a block const _: () = { … }: the aligned! macro for every window — SMEM, LENT, LENT_WLAN, LENT_RMTFS, LENT_PAS, RESERVED, XBL_AOP, SPLASH — asserts that the base and the length are multiples of 2 << 20; a separate assert! — that the three LENT windows fill it exactly. Shifting a window by a megabyte, as in § 2, now means failing to build the image, not getting a silent reset twenty seconds after modem start. The heap runs up against LENT.0 also by a constant (HEAP_END), and cannot diverge from the map.
The rest is run‑time checks, because the data does not exist earlier. enable_identity in a debug build checks window alignment once more, independently of soc.rs. Before lending anything, the platform looks at where the bootloader put the device tree: if it landed inside LENT, nothing is lent — not the rmtfs buffer, not the PAS metadata, not the WLAN rings — because the tree is read for the machine's whole life and sharing a window the modem writes to with it is not allowed. remap refuses on an unaligned or foreign window, changing nothing.
Methodologically important: the const asserts appeared as a cure for a specific reset, and were fixed in place as a security requirement — the first item of the cheapest level in the threat model. A requirement, unlike a cure, will not be deleted during a refactor as a superfluous line.
7. What this means for the threat model
In "The Parasite" the boundary with the tenant is described through channels: DMA, rmtfs, tqftp, memshare, SMEM. The memory map is not one more channel but the substrate under all of them: every channel is a window in RAM, and the window's properties — attributes, boundaries, who else writes there — determine what the tenant will and will not see in it.
The case from § 1 is telling: the modem did not attack us. It received from the host data other than what we wrote, interpreted it by its own rules and took an action with irreversible consequences for its own state. Had EFS been on a real disk, as in Linux and Android, three full rewrites of fs1/fs2 with images restored from an empty golden copy would have landed on disk. The shadow saved the disk; but the source was not in rmtfs but in the attribute field of a translation descriptor.
Hence two consequences for the boundary with the parasite:
- The cache is a channel. What we hold in cache and consider written does not exist for the neighbour until it reaches DRAM; what the neighbour wrote into DRAM does not exist for us while we read from cache. A non‑cacheable window is the only way for both sides to see one memory without relying on a maintenance discipline which, here, is moreover punished with a reset (§ 5.1).
- The block is the unit of trust. If the tenant's window and our data ended up in one block, they have one attribute, and one of the two got the wrong one. The coincidence of the map's boundaries with the boundaries of what is lent is part of the definition of "ours" and "foreign", not an implementation detail.
This does not replace the main thing — DMA translation through the SMMU for stream 0x640, which "The Parasite" discussed. But without a correct map, translation protects a window whose contents we ourselves are unable to present correctly.
8. Conclusions
- On a SoC with other masters and a secure world, cacheability and alignment are not performance knobs. A cacheable buffer read by another master is a buffer with two different contents; an unaligned window is a window with two different attributes.
- Three attributes — three relationships with a neighbour: Normal WB "only us", Non‑cacheable "us and them, no cache in between", Device "only them, we do not look". Changing the relationship is changing the attribute, not a maintenance discipline.
- There is one rule: nothing of ours shares a line or a block with anything foreign. It cost four resets out of ten, and one looked like a file server bug.
- Such invariants must be made checkable before launch. The map constants and the
constasserts next to them are this OS's cheapest fence and the only one that cannot silently reboot the tablet. - The memory map is part of the boundary with the tenant from "The Parasite in the Tablet": the subsection "Cache — nothing shared" in its § 7 is not one measure among others but the foundation the rest stand on.
Where the facts come from. The map and the comments on it — platforms/sdm855/src/soc.rs (LENT, LENT_WLAN, LENT_RMTFS, LENT_PAS, RESERVED, XBL_AOP, SMEM, SPLASH, the block const _: () = { … aligned!(…) }); the window order and HEAP_END — platforms/sdm855/src/main.rs, platform.rs. Map construction, attributes, remap, clean_dcache/invalidate_dcache — crates/aarch64/src/mmu.rs. The carve‑out, PAS metadata, lent_is_free — platforms/sdm855/src/remoteproc.rs (Identity). MMU rules, the reset table (rows 1, 2, 4, 10), the EFS read picture — docs/wifi-sm8150.md § 2, § 7, § 5.3. The threat model — docs/wifi-threat-model.md § 3 (the paragraph on the MMU), § 5 level 1 item 1. The live run — TASKS.md, line 87. A caveat: § 2 of docs/wifi-sm8150.md describes an earlier map (LENT from 0x8520_0000, 4 MiB, without LENT_WLAN); the addresses here follow soc.rs, where LENT is 8 MiB from 0x84e0_0000 and the rmtfs buffer has stayed at 0x8520_0000.*
VPN Chrome extension: what it protects and what it doesn'tA Chrome VPN extension is a proxy for browser traffic only. What it really protects, where it leaks, and when you need a full VPN app instead.
Turnkey smart homes on our own OS: 0xd-os and 0xd-uiWe build turnkey smart homes on components we designed ourselves. At the core is 0xd-os — one operating system that runs identically on a flagship tablet and on a device with 512 KB of memory.
Why security agents need a sandboxAn agent that can 'just run tools' on a security review is a second attacker — unless every call has a grant. AGI Core isolates tool execution; Forge keeps the skill list in UIDE.