adom-cli carbon containers list \
  | jq -r '.[] | select(.repository.name == "service-kicad") | "\(.slug)  \(.ssh_credentials.command)  \(.port_mappings)"'

gh auth token | ssh [email protected] '
  umask 077; tok=$(cat)
  printf "https://x-access-token:%[email protected]\n" "$tok" > ~/.git-credentials
  git config --global credential.helper store
'

pgrep -x cron >/dev/null || sudo service cron start

PATH=/home/adom/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
*/2 * * * * $HOME/service-kicad/watchdog.sh >> $HOME/.cache/service-kicad-watchdog.log 2>&1

ssh [email protected] "
  sudo DEBIAN_FRONTEND=noninteractive apt-get update -qq
  sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq kicad kicad-libraries
  kicad-cli version
"

adom-cli carbon user repos create --slug service-kicad \
  --description "shared headless KiCad + kicad-packages3d"
REPO_ID=$(adom-cli carbon user repos get service-kicad --format json | jq -r '.id')
adom-cli carbon containers create \
  --image-id 69b43b3e58d13e5ce628cdc5 \
  --name service-kicad \
  --repo-id "$REPO_ID"
# SSH in and run deploy.sh

# Pull binaries from v10 default
scp [email protected]:/usr/local/bin/service-kicad-api /tmp/
scp [email protected]:/usr/local/bin/service-kicad /tmp/
# Push to fallback
ssh [email protected] '
  sudo install -m 0755 /tmp/service-kicad-api /usr/local/bin/
  PID=$(pgrep -f /usr/local/bin/service-kicad-api | head -1)
  [ -n "$PID" ] && kill "$PID"; sleep 2
  setsid -f nohup /usr/local/bin/service-kicad-api >> /var/log/service-kicad.log 2>&1 < /dev/null
'

adom-cli carbon containers port-add xqkh3lhfl43f --port 8780 --prefix kicad-v9
ssh [email protected] '
  sudo service cron start
  [ -f /etc/default/service-kicad ] && { set -a; . /etc/default/service-kicad; set +a; }
  setsid -f nohup /usr/local/bin/service-kicad-api >> /var/log/service-kicad.log 2>&1 < /dev/null
'
# Add to TARGETS in sk-watchdog-of-watchdogs.sh

---
name: service-kicad-admin
description: >
  Adom-internal operations guide for the service-kicad container.
  SSH access, watchdog architecture, deploy procedures, credential
  rotation, container topology, KiCad upgrades, and known traps.
  NOT for external users — they use the public service-kicad skill.
  Trigger words: service-kicad admin, service-kicad container,
  kicad watchdog, service-kicad ssh, service-kicad deploy,
  service-kicad upgrade, kicad container ops.
---

# service-kicad Admin — Container Operations

Internal guide for Adom engineers operating the service-kicad
infrastructure. External users should use the public `service-kicad`
skill (CLI usage only).

**Repo:** `adom-inc/service-kicad` (private)
**Port:** 8780
**API contract:** `docs/service-kicad-api.md` in the repo.

## Container topology

| Label | SSH | Public host | KiCad | Watchdog | Status |
|---|---|---|---|---|---|
| **v10 default** | `[email protected]` | `kicad-rk5ue5pcfemi.adom.cloud` | 10.0.x | source-pull (auto-deploys) | **active** |
| v9 fallback | `[email protected]` | _(no mapping)_ | 9.0.x | health-only | **dormant** |
| v10 secondary | `[email protected]` | _(no mapping)_ | 10.0.x | health-only | **retired** |

Rediscover current containers:
```bash
adom-cli carbon containers list \
  | jq -r '.[] | select(.repository.name == "service-kicad") | "\(.slug)  \(.ssh_credentials.command)  \(.port_mappings)"'
```

## Modifying the API code

1. Clone: `git clone https://github.com/adom-inc/service-kicad.git`
2. Edit `api/src/main.rs` (the Axum HTTP server)
3. Test locally: `cargo build --release --bin service-kicad-api && ./target/release/service-kicad-api`
4. Push to `origin/main`
5. The v10 default container auto-deploys within 2 minutes via `watchdog.sh`

## Watchdog architecture

### Per-container watchdog (`watchdog.sh`, cron */2)

Runs on the production container:
1. `git fetch origin main` (exits on auth failure)
2. If HEAD moved: `git reset --hard origin/main`, `cargo build --release`
3. `sudo install` binaries to `/usr/local/bin/`
4. `kill_api()` — uses `pgrep -f` + `readlink /proc/PID/exe` to find
   and kill the running API (safe against 15-char comm truncation)
5. `launch_api()` — sources `/etc/default/service-kicad`, nohup launch
6. Health-check; relaunch if down

### Watchdog-of-watchdogs (`sk-watchdog-of-watchdogs.sh`, cron */5)

Runs on the gallia dev box. Monitors remote containers because PID 1
is sshd (not systemd) — cron doesn't survive container restarts:
1. External health check via public URL
2. If down: SSH in, check cron + API status
3. Remediate: `sudo service cron start`, relaunch API
4. Notify via `adom-cli notify`

### Credential watchdog (`sk-credential-watchdog.sh`, cron */5)

Runs on the gallia dev box. Checks git fetch health on the production
container and auto-refreshes the GitHub token from local `gh auth`:
1. Health-check the public URL
2. SSH in, `git fetch --dry-run`
3. If fetch fails: grab `gh auth token` locally, push to
   `~/.git-credentials` on the container
4. Report deployed vs latest commit

## Git credential rotation

The production container uses a `gho_` OAuth token in
`~/.git-credentials`. These tokens expire. Manual refresh:

```bash
gh auth token | ssh [email protected] '
  umask 077; tok=$(cat)
  printf "https://x-access-token:%[email protected]\n" "$tok" > ~/.git-credentials
  git config --global credential.helper store
'
```

Automated: `sk-credential-watchdog.sh` (cron */5 on gallia dev box).

## PID-1-is-sshd fragility

PID 1 on every service-kicad container is `/bin/sh -c sudo /usr/sbin/sshd -D`.
Cron never auto-starts on container restart. If the container restarts:
- Watchdog.sh stops → no auto-deploy, no health recovery
- Any API crash is permanent until someone SSHes in

Fix from outside: `ssh ... 'sudo service cron start'`

Hardening (add to `~/.bashrc`):
```bash
pgrep -x cron >/dev/null || sudo service cron start
```

## Crontab setup

The production container's crontab MUST have the PATH line:
```
PATH=/home/adom/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
*/2 * * * * $HOME/service-kicad/watchdog.sh >> $HOME/.cache/service-kicad-watchdog.log 2>&1
```

Without the PATH line, `cargo build` fails with `cargo: command not found`.

## KiCad version upgrade

```bash
ssh [email protected] "
  sudo DEBIAN_FRONTEND=noninteractive apt-get update -qq
  sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq kicad kicad-libraries
  kicad-cli version
"
```

No restart needed — `kicad-cli` is a subprocess, so the running API
picks up the new binary immediately.

## Provisioning a new container

```bash
adom-cli carbon user repos create --slug service-kicad \
  --description "shared headless KiCad + kicad-packages3d"
REPO_ID=$(adom-cli carbon user repos get service-kicad --format json | jq -r '.id')
adom-cli carbon containers create \
  --image-id 69b43b3e58d13e5ce628cdc5 \
  --name service-kicad \
  --repo-id "$REPO_ID"
# SSH in and run deploy.sh
```

## Cross-container deploy (for revived fallbacks)

Only needed if v9 or a new fallback is brought online:
```bash
# Pull binaries from v10 default
scp [email protected]:/usr/local/bin/service-kicad-api /tmp/
scp [email protected]:/usr/local/bin/service-kicad /tmp/
# Push to fallback
ssh [email protected] '
  sudo install -m 0755 /tmp/service-kicad-api /usr/local/bin/
  PID=$(pgrep -f /usr/local/bin/service-kicad-api | head -1)
  [ -n "$PID" ] && kill "$PID"; sleep 2
  setsid -f nohup /usr/local/bin/service-kicad-api >> /var/log/service-kicad.log 2>&1 < /dev/null
'
```

## Reviving the v9 fallback

```bash
adom-cli carbon containers port-add xqkh3lhfl43f --port 8780 --prefix kicad-v9
ssh [email protected] '
  sudo service cron start
  [ -f /etc/default/service-kicad ] && { set -a; . /etc/default/service-kicad; set +a; }
  setsid -f nohup /usr/local/bin/service-kicad-api >> /var/log/service-kicad.log 2>&1 < /dev/null
'
# Add to TARGETS in sk-watchdog-of-watchdogs.sh
```

## Known traps

- **pkill -x never matches** — binary name is 17 chars, comm truncates
  to 15. Watchdog uses `readlink /proc/PID/exe` instead.
- **git reset overwrites watchdog.sh** — local patches survive at most
  2 min. All watchdog changes MUST land upstream.
- **Drill export bug** — `--generate-map false` rejected by kicad-cli;
  the `let _ = run_kicad(...)` swallows the error silently.
- **Log locations** — watchdog: `~/.cache/service-kicad-watchdog.log`;
  service: `/var/log/service-kicad.log`.

## Decommissioning history (2026-05-05)

| Container | Action | Why |
|---|---|---|
| v10 default (ro8mkwui89vi) | **kept active** | All clients hit this |
| v10 secondary (g566fapq3k5v) | **retired** | Accidental duplicate |
| v9 fallback (xqkh3lhfl43f) | **dormant, recoverable** | Legacy board compat |

Steps taken: killed API, stopped cron, wrote marker at
`/etc/service-kicad-decommissioned`, removed port mappings.