Preserve Codex UI access across compatible backend upgrades - Codex

Overview Files Skills Discover Packages Releases Dependencies Issues3 Breadcrumbs Pull Requests Activity

Preserve Codex UI access across compatible backend upgrades #6

Merged opened by John Lauer 2026-09-10

Merges fix/runtime-upgrade-continuity → main

Fixes the failure path reported in adom/codex #2 without interrupting active work.

A newer extension can reuse an active older backend only when their complete generated app-server schemas, including experimental APIs, match exactly. Version numbers alone never imply compatibility. New backends persist a protocol fingerprint; legacy binaries must be unchanged for attestation. Deferred upgrade status is exposed by --runtime-status. Different/unknown protocols, downgrades and startup-option mismatches remain gated.

Validated with real native CLIs 0.153.0 (extension 26.901.22334) and 0.153.4 (26.903.71938), using an isolated CODEX_HOME and local model fixture: new client connects with old views present and with no views while a turn remains active; PID and turn ID survive three reconnects; one model request completes; idle upgrade reopens saved history. 17 Python tests and four Node entries pass.

Applied runtime.py locally to the installed package. A fresh installed-runtime proxy initialized and listed loaded threads; live backend PID 49708 remained unchanged. Existing titled editor tabs remain. No editor reload, user-turn interruption or arav-rog access. No public package release yet.

No proactive watcher swap or interrupt UI is claimed. Evidence: docs/RUNTIME-UPGRADE-VALIDATION.md. Please review before merging.

Diff Skip to comments

docs/RELOAD-INTEGRATION.md+14−5

@@ -43,11 +43,20 @@ beats the watcher, a subsequent editor reload is needed; the watcher does not force reloads or cancel work.  A changed binary starts only after all old adapter connections have closed and-the backend reports that every loaded thread is idle. Until then, the new-connection explains that the upgrade is deferred and existing work continues.-Retry after work completes and the previous editor connections close. The-adapter does not silently connect a new frontend to the old backend or force-an active turn to stop. Unrecognized upstream renderer code is left intact+the backend reports that every loaded thread is idle. A newer extension can+connect to the older backend while work continues when their complete generated+app-server JSON schemas (including experimental APIs) match exactly. Compatibility+is not inferred from the extension version or CLI major version. Schema hashes+are cached by executable identity, and new backends retain their schema hash so+removing an old VSIX does not invalidate the attestation.++`adom-codex-runtime --runtime-status` exposes a deferred `upgrade` record, including+the requested/running binary and whether connected views or backend work delayed+replacement. The next connection upgrades once all other views have closed and+the backend is authoritatively idle. The watcher does not proactively swap it.+Unknown/different protocols retain the conservative gate; downgrades, startup+option changes and unrecognized socket owners remain refused. No active turn is+interrupted. Unrecognized upstream renderer code is left intact and reported as unsupported in integration status; it needs a package update.  Ordinary panel reloads use the existing backend immediately. This is not

docs/RUNTIME-UPGRADE-VALIDATION.mdadded+70

@@ -0,0 +1,70 @@+# Runtime upgrade continuity: issue 2++Verified 2026-09-10 for [adom/codex #2](https://wiki.adom.inc/adom/codex/issues/2).++## Cause and change++The durable runtime's version gate rejected a newer extension when the older+backend had connected clients or active work. The launcher exited and the extension+could not create a usable conversation editor. The navigator warning was not the+cause, as the issue reporter's correction explains.++The runtime now compares the native app-server's generated JSON schema bundles,+including experimental APIs. Matching schemas allow the newer extension to proxy+to the existing writer. Different/unknown schemas still fail closed. It does not+assume compatibility from a shared major version and never downgrades a backend.+Schema hashes are cached by executable identity (bounded to eight entries). New+backends retain an attestation in server.json; legacy backends require their+original, unchanged binary to remain available for attestation.++A failed exclusive-lock conversion restores the shared client lock before reuse.+Backend-idle probe errors cannot authorize termination. Deferred state appears in+`adom-codex-runtime --runtime-status`. Replacement still requires no other proxy+clients and an authoritatively idle backend, on a subsequent connection. This+change does not implement a proactive watcher swap or an interrupt-and-upgrade UI.++## Real-binary regression test++The installed extension builds were used in an isolated CODEX_HOME:++- openai.chatgpt 26.901.22334: native codex-cli 0.153.0.+- openai.chatgpt 26.903.71938: native codex-cli 0.153.4.+- Their complete generated experimental schema bundles matched byte-for-byte.+- A local HTTP/SSE model fixture held a turn open; no real model call or billing.+- A newer client connected while old clients were attached, read the existing+  active turn, and preserved the old backend PID.+- With all old clients gone but that turn still active, the newer client again+  connected without replacing the backend.+- Three proxy replacements preserved the turn ID and delivered its completion.+- Exactly one fixture model request occurred; reconnect did not restart work.+- Once clients closed and work completed, the newer backend replaced the old one+  and reopened the saved conversation successfully.+- An unattested identity change still refused replacement during active work.++Reproduce with the installed native paths in ADOM_TEST_CODEX_OLD,+ADOM_TEST_CODEX_NEW and ADOM_CODEX_NATIVE (the latter selects the normal CLI):++```sh+python3 -m unittest discover -s tests -v+node --test tests/test_webview.cjs tests/test_dashboard.cjs+```++All 17 Python tests and four Node test entries passed. Unit tests also cover+shared-lock retention, failed idle probes, different/unknown protocols, replaced+legacy binaries, retained attestations after VSIX removal, cache invalidation,+and downgrade refusal.++## Local deployment++Only the installed package's runtime/runtime.py was patched from the reviewed+source. No editor reload, backend restart, settings changes, auth changes, or+arav-rog calls occurred. This is a local runtime patch, not a registry publication.++Before and after: native backend PID 49708, extension 26.903.71938. A fresh proxy+using the installed runtime initialized and listed three loaded threads. The+editor tab API retained the existing titled conversation editors. No new live+user turn was submitted and no UI reload was used to manufacture an upgrade.+The actual version transition was exercised in the isolated regression fixture.++Source and installed runtime SHA256:+`5c3a3cc957df5663bc08ebea0458c7b3e2a2f5afa5c5dcbd579da19c6af9b927`.

runtime/runtime.py+89−4

@@ -12,6 +12,7 @@ import sys import threading import time import signal+import tempfile  sys.path.insert(0, str(Path(__file__).resolve().parent / 'vendor')) @@ -119,6 +120,66 @@ def extension_version(binary):     return tuple(map(int, match[1].split('.'))) if match else None  +def protocol_fingerprint(binary, directory):+    """Cache the complete native schema by executable identity, not VSIX version."""+    before = identity(binary)+    key = hashlib.sha256(json.dumps(before).encode()).hexdigest()+    path = directory / 'protocols.json'+    try:+        cache = json.loads(path.read_text())+        if not isinstance(cache, dict): cache = {}+    except (OSError, ValueError):+        cache = {}+    if key in cache: return cache[key]+    with tempfile.TemporaryDirectory(prefix='codex-protocol-') as temporary:+        subprocess.run([str(binary), 'app-server', 'generate-json-schema',+                        '--experimental', '--out', temporary], check=True,+                       stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL,+                       stderr=subprocess.PIPE, timeout=15)+        files = sorted(Path(temporary).rglob('*.json'))+        if not files: raise RuntimeError('Native app-server schema generation returned no schemas')+        digest = hashlib.sha256()+        for file in files:+            schema = json.loads(file.read_text())+            digest.update(json.dumps([str(file.relative_to(temporary)), schema],+                                     sort_keys=True, separators=(',', ':')).encode())+        if identity(binary) != before:+            raise RuntimeError('Codex binary changed while checking its protocol')+    cache[key] = digest.hexdigest()+    cache = dict(list(cache.items())[-8:])+    temporary = path.with_suffix('.tmp')+    temporary.write_text(json.dumps(cache) + '\n')+    temporary.replace(path)+    return cache[key]+++def compatible_backend(metadata, binary, directory):+    running = Path(metadata.get('binary', ''))+    if not extension_version(binary) or not extension_version(running): return False+    try:+        running_protocol = metadata.get('protocolFingerprint')+        if not running_protocol:+            # Legacy servers have no saved attestation. A replaced/deleted path+            # cannot prove which protocol their running image implements.+            if metadata.get('identity') != identity(running): return False+            running_protocol = protocol_fingerprint(running, directory)+        return running_protocol == protocol_fingerprint(binary, directory)+    except (OSError, ValueError, RuntimeError, subprocess.SubprocessError):+        return False+++def defer_upgrade(directory, metadata, binary, reason):+    record = {'state': 'deferred', 'reason': reason, 'protocolCompatible': True,+              'runningBinary': metadata['binary'], 'requestedBinary': str(binary),+              'pid': metadata['pid'], 'checkedAt': time.time(),+              'message': 'Connected to the existing compatible backend. Active work is preserved; '+                         'the next connection can upgrade after all views close and work is idle.'}+    path = directory / 'upgrade.json'+    temporary = path.with_suffix('.tmp')+    temporary.write_text(json.dumps(record, indent=2) + '\n')+    temporary.replace(path)++ def backend_idle(sock):     """Ask the backend; disk history can lag an active turn."""     with unix_connect(str(sock), compression=None, max_size=None, open_timeout=5,@@ -174,15 +235,30 @@ def ensure_server(binary, args, client_guard=None):             if requested_version and running_version and requested_version < running_version:                 raise RuntimeError('This editor uses an older Codex extension than the running backend. '                                    'Reload the editor to use the updated extension; backend downgrade refused.')-            # An extension upgrade may switch protocols. Drain the old version-            # only with no other proxy clients and an authoritatively idle server.+            # Matching generated schemas permit reconnecting without replacing+            # the writer. Different/unknown protocols retain the conservative gate.+            compatible = compatible_backend(metadata, binary, directory)             if client_guard is None:                 raise RuntimeError('Backend version differs; upgrade requires exclusive client ownership')             try: fcntl.flock(client_guard, fcntl.LOCK_EX | fcntl.LOCK_NB)             except BlockingIOError:+                if compatible:+                    # Failed lock conversion may drop our shared lock on Linux.+                    fcntl.flock(client_guard, fcntl.LOCK_SH)+                    defer_upgrade(directory, metadata, binary, 'connected_views')+                    return sock                 raise RuntimeError('Codex updated; the previous version still has connected views. '                                    'Its work is preserved. Close those views after work finishes, then retry.')-            if not backend_idle(sock):+            try:+                idle = backend_idle(sock)+            except Exception:+                if not compatible: raise+                idle = False+            if not idle:+                if compatible:+                    fcntl.flock(client_guard, fcntl.LOCK_SH)+                    defer_upgrade(directory, metadata, binary, 'backend_busy_or_unverified')+                    return sock                 raise RuntimeError('Codex updated while a turn is running. Its work is preserved; '                                    'retry when it finishes to activate the new version.')             if process_matches(metadata): os.kill(metadata['pid'], signal.SIGTERM)@@ -193,6 +269,10 @@ def ensure_server(binary, args, client_guard=None):         if socket_alive(sock):             raise RuntimeError('An unrecognized process owns the Codex runtime socket')         sock.unlink(missing_ok=True)+        protocol = None+        if extension_version(binary):+            try: protocol = protocol_fingerprint(binary, directory)+            except (OSError, ValueError, RuntimeError, subprocess.SubprocessError): pass         log = directory / 'server.log'         with log.open('ab') as output:             process = subprocess.Popen(@@ -201,6 +281,7 @@ def ensure_server(binary, args, client_guard=None):                 start_new_session=True, close_fds=True,             )         metadata = {'pid': process.pid, 'binary': str(binary), 'identity': identity(binary), 'args': args,+                    'protocolFingerprint': protocol,                     'socket': str(sock), 'started_at': time.time(),                     'start_ticks': Path(f'/proc/{process.pid}/stat').read_text()                         .rsplit(')', 1)[1].split()[19]}@@ -212,6 +293,7 @@ def ensure_server(binary, args, client_guard=None):             if process.poll() is not None:                 raise RuntimeError(f'Codex backend exited ({process.returncode}); inspect {log}')             if socket_alive(sock):+                (directory / 'upgrade.json').unlink(missing_ok=True)                 return sock             time.sleep(0.05)         raise RuntimeError(f'Codex backend is still starting; inspect {log}')@@ -250,7 +332,10 @@ def main():     if argv == ['--runtime-status']:         path = state_dir() / 'server.json'         metadata = json.loads(path.read_text()) if path.exists() else {}-        print(json.dumps({**metadata, 'alive': process_matches(metadata)}, indent=2))+        pending = path.with_name('upgrade.json')+        upgrade = json.loads(pending.read_text()) if pending.exists() else None+        if upgrade and upgrade.get('pid') != metadata.get('pid'): upgrade = None+        print(json.dumps({**metadata, 'alive': process_matches(metadata), 'upgrade': upgrade}, indent=2))         return     binary = native_binary()     args = server_args(argv)

tests/test_runtime.py+82−3

@@ -73,6 +73,69 @@ class RoutingTests(unittest.TestCase):             with patch.dict(os.environ, {'ADOM_CODEX_RUNTIME_STATE': directory, 'CODEX_HOME': '/tmp/b'}):                 self.assertNotEqual(runtime.state_dir(), first) +    def test_compatible_upgrade_preserves_busy_backend_and_connected_views(self):+        for connected, idle in [(False, False), (True, True), (False, RuntimeError('probe failed'))]:+            with self.subTest(connected=connected, idle=idle), tempfile.TemporaryDirectory() as tmp:+                directory = Path(tmp)+                old = directory/'openai.chatgpt-1.0.0-linux-x64/codex'+                new = directory/'openai.chatgpt-1.0.1-linux-x64/codex'+                for binary in [old, new]:+                    binary.parent.mkdir(); binary.write_text('fixture')+                metadata = {'pid': 123, 'binary': str(old), 'identity': runtime.identity(old), 'args': ['app-server']}+                (directory/'server.json').write_text(json.dumps(metadata))+                with (directory/'clients.lock').open('a') as guard, (directory/'clients.lock').open('a') as other:+                    runtime.fcntl.flock(guard, runtime.fcntl.LOCK_SH)+                    if connected: runtime.fcntl.flock(other, runtime.fcntl.LOCK_SH)+                    with patch.object(runtime, 'state_dir', return_value=directory), \+                         patch.object(runtime, 'process_matches', return_value=True), \+                         patch.object(runtime, 'socket_alive', return_value=True), \+                         patch.object(runtime, 'protocol_fingerprint', return_value='same-schema'), \+                         patch.object(runtime, 'backend_idle', **({'side_effect': idle} if isinstance(idle, Exception) else {'return_value': idle})), \+                         patch.object(runtime.os, 'kill') as kill, \+                         patch.object(runtime.subprocess, 'Popen') as spawn:+                        self.assertEqual(runtime.ensure_server(new, ['app-server'], guard), directory/'server.sock')+                        kill.assert_not_called(); spawn.assert_not_called()+                    record = json.loads((directory/'upgrade.json').read_text())+                    self.assertEqual(record['pid'], 123)+                    self.assertTrue(record['protocolCompatible'])+                    runtime.fcntl.flock(other, runtime.fcntl.LOCK_UN)+                    with self.assertRaises(BlockingIOError):+                        runtime.fcntl.flock(other, runtime.fcntl.LOCK_EX | runtime.fcntl.LOCK_NB)++    def test_unknown_changed_or_different_protocol_does_not_attest_compatibility(self):+        with tempfile.TemporaryDirectory() as tmp:+            directory = Path(tmp)+            old = directory/'openai.chatgpt-1.0.0-linux-x64/codex'+            new = directory/'openai.chatgpt-1.0.1-linux-x64/codex'+            for binary in [old, new]:+                binary.parent.mkdir(); binary.write_text('fixture')+            metadata = {'binary': str(old), 'identity': runtime.identity(old)}+            with patch.object(runtime, 'protocol_fingerprint', side_effect=['old-schema', 'new-schema']):+                self.assertFalse(runtime.compatible_backend(metadata, new, directory))+            with patch.object(runtime, 'protocol_fingerprint', side_effect=RuntimeError('unsupported')):+                self.assertFalse(runtime.compatible_backend(metadata, new, directory))+            old.write_text('replacement')+            with patch.object(runtime, 'protocol_fingerprint') as fingerprint:+                self.assertFalse(runtime.compatible_backend(metadata, new, directory))+                fingerprint.assert_not_called()+            metadata['protocolFingerprint'] = 'attested-at-start'+            old.unlink()+            with patch.object(runtime, 'protocol_fingerprint', return_value='attested-at-start'):+                self.assertTrue(runtime.compatible_backend(metadata, new, directory))++    def test_schema_fingerprint_is_cached_by_binary_identity(self):+        with tempfile.TemporaryDirectory() as tmp:+            directory = Path(tmp); binary = directory/'codex'; binary.write_text('fixture')+            def generate(args, **kwargs):+                (Path(args[-1])/'schema.json').write_text('{"type":"object","properties":{}}')+            with patch.object(runtime.subprocess, 'run', side_effect=generate) as run:+                first = runtime.protocol_fingerprint(binary, directory)+                self.assertEqual(runtime.protocol_fingerprint(binary, directory), first)+                self.assertEqual(run.call_count, 1)+                binary.write_text('changed fixture')+                self.assertEqual(runtime.protocol_fingerprint(binary, directory), first)+                self.assertEqual(run.call_count, 2)+  class Client:     def __init__(self, env):@@ -160,6 +223,8 @@ class ReconnectTest(unittest.TestCase):                 'wire_api = "responses"\nrequires_openai_auth = false\n')             env = dict(os.environ, CODEX_HOME=str(home), ADOM_CODEX_RUNTIME_STATE=str(test_root / 'state'))             env.pop('OPENAI_API_KEY', None)+            if os.environ.get('ADOM_TEST_CODEX_OLD'):+                env['ADOM_CODEX_NATIVE'] = os.environ['ADOM_TEST_CODEX_OLD']             try:                 # Simultaneous clients must share one server, even before any                 # conversation exists. IDs are scoped to each connection.@@ -182,6 +247,20 @@ class ReconnectTest(unittest.TestCase):                     'input': [{'type': 'text', 'text': 'Local fixture', 'text_elements': []}]})['turn']                 self.assertTrue(started.wait(10), 'Fixture did not receive the model request')                 self.assertFalse(runtime.backend_idle(Path(json.loads(metadata_path.read_text())['socket'])))+                if os.environ.get('ADOM_TEST_CODEX_NEW'):+                    upgrade_env = dict(env, ADOM_CODEX_NATIVE=os.environ['ADOM_TEST_CODEX_NEW'])+                    newer = Client(upgrade_env)+                    clients.append(newer)+                    observed = newer.rpc(2, 'thread/read', {'threadId': thread_id, 'includeTurns': True})['thread']+                    self.assertEqual(observed['turns'][-1]['id'], turn['id'])+                    self.assertEqual(json.loads(metadata_path.read_text())['pid'], original_pid)+                    self.assertEqual(json.loads(metadata_path.with_name('upgrade.json').read_text())['reason'], 'connected_views')+                    newer.close(); first.close(); second.close()+                    env = upgrade_env+                    first = Client(env)+                    clients.append(first)+                    self.assertEqual(json.loads(metadata_path.read_text())['pid'], original_pid)+                    self.assertEqual(json.loads(metadata_path.with_name('upgrade.json').read_text())['reason'], 'backend_busy_or_unverified')                 second.close()                 for index in range(3):                     first.close(abrupt=index % 2 == 0)@@ -193,11 +272,11 @@ class ReconnectTest(unittest.TestCase):                     self.assertEqual(resumed['turns'][-1]['id'], turn['id'])                     self.assertEqual(resumed['turns'][-1]['status'], 'inProgress')                     self.assertEqual(json.loads(metadata_path.read_text())['pid'], original_pid)-                # A store update must not connect its new frontend to an old-                # backend, nor kill the active turn even with zero clients.+                # An unattested binary change must still fail closed, preserving+                # the active turn even when there are no proxy clients.                 first.close()                 original_metadata = json.loads(metadata_path.read_text())-                changed_metadata = {**original_metadata, 'identity': ['older-extension']}+                changed_metadata = {**original_metadata, 'identity': ['older-extension'], 'protocolFingerprint': None}                 metadata_path.write_text(json.dumps(changed_metadata))                 rejected = subprocess.run([sys.executable, str(ROOT/'runtime/runtime.py'),                     '-c', 'features.code_mode_host=true', 'app-server', '--analytics-default-enabled'],

Comments

No comments yet.