Ingesting unmodified games

The headline capability: vgai’s editor can load and edit a three.js game it didn’t author, without modifying the game’s code.

The render-accessor trap

IngestGameAdapter (packages/editor/src/authoring/ingest-game-adapter.ts) captures an unmodified game’s live scene + camera via the render-accessor trap — it observes the game’s own renderer.render(scene, camera) call to grab the objects, with zero edits to the game. It returns a MountedGame that drives its own loop.

After capture, the host adopts the game’s canvas into the editor’s game pane. A hybrid DOM+canvas game — a React/R3F app with HTML HUDs, overlay panels, or drei <Html> portals — can declare the element that owns its canvas by setting window.__vgaiGameRoot in its entry module; the host then adopts that root wholesale, so the game’s DOM UI travels with its canvas instead of stranding at page level over the editor chrome. Games that declare nothing keep the bare-canvas behavior. The whole game-side obligation is ordinary embed hygiene: create a container, mount into it, declare it, and keep your own position: fixed UI contained (transform: translateZ(0) on the container).

Editing the live tree (anti-shim)

IngestAuthoringAdapter implements the AuthoringAdapter directly over the live Object3D tree — it does not fabricate a SceneEntity. Nodes get structural-path ids (stable across reloads) so selection/transform/inspector work on the foreign objects as-is.

Overlay persistence

Because you can’t rewrite the game’s source, edits persist as an overlay (ingest-overlay.ts): each edit is keyed by its structural-path id and written to a .vgai/overlays/<game>.json file under the project root — the same .vgai/ namespace the editor’s other per-project state uses. After the game rebuilds its scene on reload, the overlay is re-applied — so your edits survive without touching the original. IngestAuthoringAdapter also implements the persistence reload contract (applyExternal), so if the overlay file changes on disk while the session is live — another tool editing it, or a file-watcher push — it re-applies to the running scene directly, with no remount needed.

The file is a versioned envelope — {formatVersion, authoredAgainst: {gameVersion, upstreamPin}, overrides} — where gameVersion records the game-manifest version the edits were authored against (null if the game declares none; it is never fabricated) and upstreamPin records the vendored game’s UPSTREAM.md commit pin when one is discoverable (null when the editor looked and found none; the key is absent in pre-pin overlays, which load as “legacy, never checked” — never guessed either way). Re-vendoring the game to a new upstream commit surfaces as a loud upstreamPin mismatch in the same report. If a reapply can’t place some edits — say the game updated and a structural path no longer resolves — those entries are orphaned, not dropped: they stay in the file, and the editor raises an orphan report (a dismissible banner in the hierarchy panel plus one structured overlay-report console warning listing the stale ids and any version mismatch). Pre-envelope flat overlay files still load and upgrade on the next save.

The overlay is also the ship path: a game owner deploying their own unmodified build can opt in with one call — applyVgaiOverlay({ three, gameId }) from @engine/adapter/ingest/overlay-applier, made before booting the game. It fetches the deployed .vgai/overlays/<game>.json, applies it at the game’s first rendered frame (the game’s own systems win afterward), and logs the same orphan/version report — zero editor code involved at runtime. A missing overlay file just runs the game unmodified.

iframe tiers

For games that can’t share the editor’s three instance, the iframe ingest path (ingest-iframe-adapter.ts, mountIngestGameEmbed) hosts them in a sandboxed iframe. Two tiers:

embed-only — an opaque bundle is hosted and disposed, but not introspected or paused (a sandboxed opaque iframe can’t be controlled).
reachable-three — the iframe shares the editor’s three, enabling in-realm editing (with documented cross-realm gizmo limitations).