Adom Hydrogen

Found it. The container is still running but starved: memory at 100% of its limit, CPU pinned at all 4 cores, and 6,001 processes. It hasn't crashed, so Docker never restarts it, and nothing inside can answer, which is the 504.

Two platform problems made this worse. A large container should get 16 GB, but curium caps every container at 8 GiB, so yours has half what it should. And there's no restart for owners: curium has stop and start, but Carbon doesn't pass them through.

Before we restart it, can you capture the process list? A restart keeps your disk, including /tmp and /home/adom/project, but wipes the process list, which is the best evidence of what piled up. Your SSH user is in ssh_credentials.command from adom-cli carbon containers list-for-repo caleb Test. SSH may take a couple of minutes to connect while the container is starved.

ssh <your-ssh-user>@adom.cloud 'f=/home/adom/project/oom-evidence-$(date -u +%Y%m%dT%H%M%SZ).txt; { date -u; cat /sys/fs/cgroup/memory.events; echo "== by count"; ps -eo comm= | sort | uniq -c | sort -rn | head -25; echo "== by memory"; ps -eo pid,ppid,rss,pcpu,etime,args --sort=-rss | head -40; echo "== tree"; ps -eo pid,ppid,etime,args --forest | head -400; df -h; } > "$f" 2>&1; echo saved "$f"'

Reply here once it's saved, or if SSH won't connect, and we'll restart it.

caleb, what i do in these cases, which has happened to me a lot is just use hydrogen desktop to ssh in for me and fix it and it always figures it out.

meaning have the ai fix it from your local laptop

Thanks, Aditya. I couldn't capture the process list.

I added a key and ran your command at 20:35 UTC with a 10-minute limit. The gateway accepted the key (Server accepts key, russh_0.57.1), but the session never got a shell inside the container and timed out at 10 minutes with no output. So nothing was saved, and there may be a partial oom-evidence-*.txt under /home/adom/project.

Please go ahead and restart it, keeping the disk. Once it's back up, I'll go through it for whatever spawned the processes (6,001, at 100% memory). If you can read the process list from the host side before the restart (docker top, or the cgroup's cgroup.procs / memory.events), that would cover the evidence.

+1 on passing stop/start through Carbon for owners, and on fixing the 8 GiB cap for large.

Update: I tried the one control owners have, a resize, to force a restart. It failed partway, and the container now looks stopped rather than starved.

• 20:59 UTC: adom-cli carbon containers resize <slug> medium returned 500 FAILED_TO_UPDATE_CONTAINER. The class still reads large.
• Right after, HTTPS changed from a 30 s 504 to an instant 502.
• SSH now says Received disconnect ... Could not connect to your container.
• 21:03 retry to medium and 21:04 resize to large returned the same 500 FAILED_TO_UPDATE_CONTAINER, still 502.

So curium seems to have stopped it and then failed to recreate it. Could you start it (class large, same disk)? The in-memory process evidence is gone now, sorry. Once it's up, I'll look through the logs on disk for what spawned the processes.

It might also be worth checking why the resize PATCH fails for this container. It may be the same thing that stops the start.

More detail for whoever picks this up. The container can't be started from the owner side at all now.

The Hydrogen web app's container card has a Restart button that calls curium directly through /api/curium-proxy/containers/<slug>/stop and /start. I called those for this container (21:10 UTC). stats, stop and start all return the same thing:

500 INTERNAL_SERVER_ERROR
container 0f4c0ad307f7… not found in network

The same stats call on a healthy container of mine returns normal Docker stats, so the proxy and auth are fine.

So after the failed resize (FAILED_TO_UPDATE_CONTAINER), curium's record points at a Docker container that is no longer attached to the network. It is stopped, removed, or was replaced by a new container that curium doesn't know about. Curium looks the container up in the network before it does anything, so every call fails, including the start that would fix it.

It needs a host-side fix: start or reattach that container, or repair curium's record. Please check the disk state before recreating anything, since I can't tell from here whether the resize committed it.

Two bugs worth filing from this:

1. A resize on a starved container leaves it stopped and returns 500.
2. start fails on a container that isn't in the network, which is exactly when a start is needed.

It's back up: same container, same disk, started at 21:29 UTC. HTTPS answers with the normal 302 again.

One correction to the timeline: the stop was ours, not your resize. We stopped it through curium at 20:51, and it exited at 20:53 after a forced kill. That's when the 502 started. Your resizes failed at the same curium lookup that broke start, so they never ran. The container is still the original one, and its limit is unchanged.

For the root cause: Docker recorded no out-of-memory kill, and the node's kernel log showed none for this container. The account we used may not be able to read kernel messages, though. If that holds, it never got killed; the processes filled memory and swap and starved it. The files on your disk are the place to look. We'll file the bugs you listed privately on curium, along with the 8 GiB cap on large containers.