From the marginalia
The Dig: Five Months of OpenDS
The first commit in the repository is dated May 3, and its message is and so we begin, which is what you write when you do not yet know whether the thing can be done at all. Five months and three hundred and eighty-four commits later, I put the whole project on the table and audited it: every tool, every tag, every document, the complete state of an effort to take apart two of my favourite CRPGs and write down how they work. This post is that audit. The project is OpenDS, a community toolkit for SSI’s Dark Sun: Shattered Lands (1993) and Wake of the Ravager (1994), and the headline reads like this: the data side of both games is now read, written, and mined end to end; two player-facing bugfix patches have shipped through a verified pipeline; and a Godot spike plays the games’ opening loop, built from nothing but the original files. What remains is the hard half, and for the first time the hard half has a map.
I. The Premise
Dark Sun has a graveyard. Dark Sun World ran from 2004 to 2008 and went quiet. A Dark Sun Online emulator of the 2010s was shut down by Wizards of the Coast. paulofthewest’s soloscuro-archive, the most serious attempt, accumulated roughly 567 commits and stalled in 2023. The Beamdog forums hosted a plan to rebuild Shattered Lands inside the Infinity Engine; it went inactive. Half a dozen prototypes live in the dsoageofheroes organisation alone, none playable end to end. Every one of them died on the same rock: the engine’s embedded scripting language, a bytecode VM called GPL, which carries most of the games’ actual logic and had no public spec. You can render the maps and you can play the animations, and then a slave in the pens of Draj wants to talk to you, and the conversation is bytecode nobody can read.
OpenDS’s founding decision was to stop promising the engine. The spec says it outright: a from-scratch reimplementation is the aspiration the name encodes, not a deliverable, and the project does not commit to it. What it commits to instead is an ordering: tools before patches, patches before the engine, and anything that makes the digging easier wins. Ship the artifacts you accumulate on the way to an engine as standalone, useful tools; each one chips at the GPL problem; each one is something the next attempt can pick up instead of reinventing. Whether the engine ever gets built by me is, in the spec’s own words, a thing we get to if we get there. Five months in, that sentence has aged better than any roadmap I have ever written, mostly because the toolkit it describes actually exists now.
❦
II. The Toolkit
Fourteen tools ship today, all independently versioned and MIT-licensed, each with its own README. Six are Rust crates in a workspace whose shared dependency list is seven entries long and gated by the spec; eight are Python tools, stdlib-only by rule. The shape of the stack is a sentence: gff-edit reads and writes the games’ GFF containers, gpl-disasm turns the bytecode into labelled text at 100% corpus alignment, gpl-asm turns it back with a 600-of-600 byte-identical round-trip, and everything else stands on that spine. save-inspect decodes and edits saves; dialog-extract pulls NPC dialog as browsable trees; image-extract and region-render decode sprites and composite whole regions, entities animated and, as of a decode that required reading the colour-cycle routine out of the executable, with the VGA palette cycling exactly as the engine cycles it. repro drives the game under DOSBox with one absolute rule: every run happens over an overlay mount, because DSUN.EXE zeroes its own world-state file the moment it finds itself in a writable directory. The install is sacred. verify-install hashes an install against canonical manifests and can repair it from the GOG installer; ovr-map and exe-patch are the binary side; opends is the front door that dispatches by magic bytes; atlas publishes a whole install as a static site.
The test surface is smaller than a web project’s and harder to fake. One hundred and eighty-five Rust test functions gate the workspace; CI runs the repo’s eight Python selftest entry points on both Python 3.11 and 3.14 with the linter pinned to the exact version CI uses, so a local run cannot silently diverge. The corpus tests walk every shipped file of both games and assert round-trips; without the games installed they skip on the record rather than pass on nothing.
The binary side is where the project stopped being file formats. The executables are Borland overlay binaries, not the DOS/4GW extenders everyone assumes for 1993 DOS, a fact established the hard way and written down. ovr-map turns each ~600 KB blob into 52 and 49 overlay segments, 935 and 854 confirmed function entry points at about 93% coverage, disassembled at correct bases and importable into Ghidra as labelled blocks. Both games’ GPL dispatch tables are resolved, 129 handlers each, which means every opcode has a named handler address in both engines; the fifteen bytes that never had names are proven unimplemented, reserved and unreachable, in both. The symbol catalogues stand at 130 and 132 curated rows, every entry carrying an evidence chain: a dispatch-table offset, a verified string cross-reference, or a relocation-confirmed base.
None of this is pretending to be virgin territory. The manifest in CREDITS.md maps forty-one shipped features to the exact upstream file or function each was ported from, and the heaviest debt is to paulofthewest’s libgff with twenty-eight rows. John Glassmyer’s GFF writer policy and Greg Kennedy’s symbol table are credited where they are used. Over-crediting is the stated policy; the alternative has a poor record in this corner of the hobby.
❦
III. The Patches
The first real fix came from a sweep that counted things nobody had counted. Both games register their interactable objects against handler scripts, and the sweep found registrations pointing at handlers that contain nothing: in Shattered Lands, six dead rows, five look-triggers and one attack-trigger, all aimed at a single orphaned exit gpl byte in the Darkhold endgame. In plain terms: a portcullis guard who cannot fight, chamber creatures with no examine text, a wyvern that will not engage. This is the concrete instance behind the community’s thirty-year report that enemies refuse to engage. The fix repoints the four rows whose correct handler can be proven statically to each object’s own working handler, eleven bytes across three chunks, and deliberately leaves two rows whose object is a runtime variable no static pass can resolve. Everything about that shape is policy: fix what you can prove, document what you cannot, never let enthusiasm write bytes. The patched file re-disassembles with all 250 chunks in alignment. It shipped as darkfix-ds1 v0.1.0 on September 11.
Wake of the Ravager got its first fix the same week: darkfix-ds2 v0.1.0 repoints 27 of 39 dead registrations, five bytes per edit, chunk lengths untouched. Getting to 39 from the sweep’s first answer of 30 is my favourite bug of the year: the census had been keying a mnemonic as one word when the catalogue spells it with a space, so every pickup trigger in both games had gone unswept, and a parameter misread silently skipped every use-with row. Two quiet counting bugs, both found by audit, both now pinned by selftests, and the corrected census changed the shipped fix. The release notes carry both numbers, the wrong one and the right one, because a correction you cannot see is a correction that did not happen.
The player-facing half is one Python file with no dependencies. The applier refuses to run against any build whose hash does not match the manifest; it backs up everything it touches; it writes its journal as pending before the first byte and completes it after the last, so a crash mid-write is recoverable instead of stranding a half-patched game; it refuses to let two fixes edit the same file; and --unapply restores byte-identically. The whole cycle is proven under Wine with Windows Python against scratch copies of the real install. The one box still open is the proof on real Windows, which is a ten-minute job I keep not doing because it requires a Windows machine and a spare hour in the same place.
The packaging story is the audit’s best catch. The project’s first release had shipped with zero assets: the pipeline existed, the tags existed, the release page existed, and the zip the README told players to download did not. build-release.sh closed that, and building the first zip through it caught a genuine ship-blocker the Wine proof had sailed past: the tagged applier resolved its patch root one directory above the flattened zip layout, so the shipped artifact could never have found its own manifest. The release zips are now deterministic, fixed timestamps, byte-identical on rebuild, with their hashes quoted into the release notes.
❦
IV. The Mining
September belonged to the campaigns. Once the tools could read everything, the question became what the everything says, and the answer arrived in waves of read-only analysis agents. The object database of both games is now written out end to end: 290 creature records in Shattered Lands and 352 in Wake of the Ravager, about 750 and 1,230 items, 196 and 320 spells, 595 and 698 containers with their contents, every creature’s inventory, and 13,028 plus 13,559 map placements across all 53 regions. The dialog corpus is 46,053 strings over 531 chunks. These are not transcriptions; they are generated by one script straight from the shipped files, and the generator is gated on anchor checks, a named monster’s stats, a known chest, the elevator’s sprite, and exits nonzero on any drift. The twelve catalogue documents total about 2.1 megabytes of markdown, and the two world dumps are the largest files in the whole repository.
The engine documents went deeper than the data. There is now a complete spec of the GPL VM, read out of both binaries, and the finding that matters most for anyone who ever attempts the engine: the two games ship one engine, identical semantics, down to the instruction-ring mechanics and the expression evaluator’s fifteen operators. The fifteen-year-old blocker is not blocked anymore; the spec exists, here, and it was produced by tools that are themselves shipped artifacts.
The most humbling read was SSI’s own. Diffing the CD 1.0 release against GOG’s 1.10 answers the question how did the publisher fix the games, and the answer is: hardly in the binary at all. All 117,566 differing bytes in the executable decode as recompile rebias, address arithmetic from a shifted link, with zero behavioural changes; the real fixes shipped as data. The redrawn volcano overhead maps are in there, and so is the spell-text table: the 1.0 release shipped twenty spell descriptions as the literal placeholder fooey!, and 1.10 filled them with Power Word Kill, Time Stop, and friends. One entry carries a leaked fragment of SSI’s own build script, copy c:foo.bat ..\RES\text\BIGBYFST.spn, packaged into the shipped game in 1993 and still sitting there in the GOG build. Thirty years of software archaeology in one string literal.
By the third campaign the target had shifted from what is in the files to what would a reimplementation need to know: the AD&D rules tables (the 168-byte block that holds the class HP and THAC0 rates), the combat loop down to pseudocode, spell casting and effect machinery, character generation (best of four rolls of 4d4 with a prime-stat floor of 17; the random number generator is Borland’s rand(), deterministic, and nothing in either binary ever seeds it), exploration and line of sight, the intro cinematic’s frame codec, validated on all 708 frames, audio routing, the window manager. The campaign’s closing verdict is the sentence the whole project has been walking toward: the knowledge side of a port is mined out. What remains is engineering, plus an explicit list of behaviours that only exist at runtime and have to be captured from the live game.
❦
V. The Port Spike
Which is how a Godot project appeared in a repository that is nominally about 1993. The spike’s discipline is total: no new art, no invented UI, every pixel sourced from the original files by nine exporter scripts that consume the toolkit’s own decoders. It boots the way the game boots: the SSI logo, the AD&D card, the title, the full intro as a 715-entry baked timeline running about 125 seconds against the original’s 130. The main menu is the game’s own window resources; character creation rolls stats with the engine’s own rule, best of four, floor 17. Then the Slave Pens of Draj, walkable, with the announcer’s real dialog; the arena; the first fight, run by the engine’s own combat machinery: the d200 tie-break, the three-blows-per-two-rounds alternator, natural 20s and 1s, morale thresholds; then back to the pens. Twenty-nine GDScript files, about 4,600 lines, against 858 generated UI resources and 232 sprites. Saves are written as real GFFI structures.
Accuracy is not asserted, it is diffed. Twenty-five DOSBox captures are committed as ground truth, and every screen gets a side-by-side parity sheet; the latest artifact is a video pair with the original running on the left under recorded input and Godot on the right. Ten surfaces went through that loop in the last pass: item icons, slot glyphs, backdrops, the double-strike bitmap font, the combat HUD. When the interact screen’s controls did not match the assumed layout, the answer came from the disassembler, not from squinting: the buttons are talk, steal, and give.
The spike is gated by a lock I wrote on purpose. The gladiator pits screen is not done until it is 1:1 against the original, inside the pits, before anything past them gets built, and the lock’s own document ends with the only rule that matters: it lifts on my say-so, not when the checklist looks done. Scope discipline in a port is the difference between a playable slice and a ten-year graveyard, and this graveyard is well populated already.
❦
VI. The Method
The honest accounting has to include how the work got done. Most of the digging since early September was executed by agent fleets: ZCode running GLM models, dispatching read-only research agents four at a time in waves, each agent returning evidence with file-and-line citations, each wave integrated, verified, and committed in the main thread before the next one launched. The final audit ran eight lenses plus a prose grader over its own findings; the blitz executed all ten of the audit’s ranked items in one day; the bestiary took four waves of four agents; the port mining took three; a single sweep day ran fourteen. Nobody is going to mistake the last five months for a lone programmer with a decompiler and a dream, and the previous posts on this site already settled the question of whether I would hide that.
The reason it works on this project and would not work on all of them is that reverse engineering is evidence gathering against a ground truth that does not care about your confidence. A decode either re-aligns the whole corpus or it does not, and a patched file either re-disassembles clean or it does not; the same is true of every rendered sprite against its capture. The repo is full of instruments that make wrong claims loud: corpus tests, anchor-gated generators, hash manifests, oracle screenshots. Under those conditions an agent’s capacity for plausible nonsense stops being something you manage by hope and becomes a failure mode with an alarm wired to it. The census bug is the case study in both directions: machines counted wrong for weeks, machines found the counting bug, machines now test for it, and the human decided what the corrected number meant for the shipped fix.
That division is the part I would keep even if the tooling changes again next month. The fleet gathers the evidence and drafts the findings; the decisions stayed with me all September, ten of them in one blitz, each one surfaced as an actual question with options and a recommendation rather than buried in a summary. The agents never cut a release on their own authority, never pushed without a standing grant, and never got to write the files the public sees without those files passing through hands that sign the release. It is the same shape as a good kitchen: prep can be delegated; the pass is not.
❦
VII. What the Audit Found
An audit that only congratulates is a press release, so here is the ledger of things the sweep flagged, all small, none excused. The foundation crate has never been tagged: gff-edit sits at 0.6.0 with a release record but no gff-edit-v* tag, because tagging only went forward-only from September 4, and three of the Python tools are in the same boat; the VERSION files and patchnotes are the record, so nothing is lost, but the asymmetry itches. The selftest idiom has gaps: CI comments call --selftest the repo’s Python test convention, and three tools never grew one, while the opends umbrella CLI carries a single test for a job, dispatching to other tools, that is mostly argument-shape. Documentation drift runs in the unusual direction: two format docs still describe the dispatch loop as unfound, a problem resolved so thoroughly that two other documents exist to report the solution, and the roadmap has several boxes whose work landed but whose checkboxes stayed unticked, the opposite of the usual failure and much easier to fix. The darkfix tag-push release path builds and hashes the zip but runs no tests on the way, an obvious rider for the release workflow. And the DS2 patch’s player-facing install section still says forthcoming and points at the DS1 walkthrough.
The true remainder is more interesting than the debt, because the boundary is clean. Everything left is one of three things. It is calendar-gated on my own play: the save-format chunk mapping and the opcode semantics loop both need played sessions and DOSBox time, which is the one resource no agent wave can manufacture. It is runtime capture: the mines elevator, the headline bug of Wake of the Ravager, the freeze that broke the late game in 1994, has a dossier that reads like it should already be fixed; the transition machine is read, the region ids are named, the elevator’s trigger is traced to an object id whose rendered sprite is, visually confirmed, the elevator shaft itself, and the fix shape is pre-decided and safe under either of the two plausible engine semantics. What stands between the project and that fix is one runtime capture from a played save. Which is to say: me, DOSBox, an evening. Or it is engineering, enumerated and unromantic: a relocation-table overlap check before the first real executable patch, the cinematic container decode that image-extract still misparses, the armour-overlay sprites for the port’s centre figure.
⁂
The audit’s summary line, for the version of me who typed and so we begin in May: the tools are real, the games are open, two fixes shipped, the data is written down, and the engine you would not promise is now a question of evenings instead of mysteries. The name still says OpenDS and the spec still refuses to commit to it, and that remains correct. But spec section 12 set a condition for the engine becoming sensible rather than heroic: a working disassembler, a working assembler, a reader and writer in the language of choice, a region renderer, enough documented opcodes to read the bulk of the game’s scripts. Every one of those exists. The condition is met; the only thing left to decide is whether the thing gets built, and that decision, like every one that mattered this whole project, is not the fleet’s to make.