fix(provisioning): recover from vLLM readiness timeout and increase poll window

When create timed out during vLLM readiness polling (common for large
models like Qwen3.6-27B-FP8), rerunning create would stop and restart
the already-running container, restarting the whole startup sequence.

Now the vLLM install script checks if the container is already running
and serving the correct model before touching it. If it detects a
healthy container, it skips the stop/pull/start cycle entirely.

Also increases the readiness timeout from 20 min (240x5s) to 30 min
(360x5s) to accommodate cold starts with model download and CUDA graph
capture on large models.

2 files changed, 33 insertions, 11 deletions

diff --git a/AGENTS.md b/AGENTS.md
index f7f3491..5ea8394 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -155,16 +155,24 @@ the WireGuard subnet (`192.168.3.0/24`) only. Always use the WireGuard IP.
 ## vLLM container startup sequence
 
 After the Docker container starts, the model goes through several phases before
-inference is ready. On an A100 with a warm HuggingFace cache:
+inference is ready. On an H100 with a warm HuggingFace cache:
 
 | Phase | Duration | Log signal |
 |-------|----------|------------|
 | Docker pull (first time) | ~2–3 min | Layer progress bars |
 | Model download from HuggingFace (first time) | ~3–5 min | `Downloading...` |
-| Weight loading | ~47 s | `Loading safetensors checkpoint shards: 100%` |
-| torch.compile + CUDA graph capture | ~1–2 min | `torch.compile took X s` |
+| Weight loading | ~4–5 s | `Loading safetensors checkpoint shards: 100%` |
+| torch.compile + CUDA graph capture | ~2–4 min | `torch.compile took X s` |
 | **Ready** | — | `Application startup complete.` |
 
+Qwen3.6-27B-FP8 on an H100 takes ~5 min from container start to ready on a warm
+cache; cold start (first run, no HuggingFace cache) can take 10+ minutes. The
+provisioning script waits up to 30 minutes (360 × 5 s) for the API to respond.
+
+If `create` fails because the readiness poll timed out but the container is
+still running, re-running `create` will detect the running container and skip
+the restart — it goes straight to the readiness check.
+
 **Monitor startup:**
 
 ```bash
@@ -187,8 +195,10 @@ retries internally.
 
 ## Resuming a failed `create`
 
-If `create` exits non-zero partway through (e.g. WireGuard retries exhausted, Docker
-EOF), the VM is still running and the state file tracks it. Simply re-run:
+If `create` exits non-zero partway through (e.g. WireGuard retries exhausted, vLLM
+readiness timeout, Docker EOF), the VM is still running and the state file tracks it.
+Re-running `create` will skip already-completed steps and, if the vLLM container is
+already running and healthy, will skip the restart entirely.
 
 ```bash
 ruby hyperstack.rb --vm 2 create
diff --git a/lib/hyperstack/provisioning.rb b/lib/hyperstack/provisioning.rb
index 948cd2a..2738275 100644
--- a/lib/hyperstack/provisioning.rb
+++ b/lib/hyperstack/provisioning.rb
@@ -212,10 +212,22 @@ module HyperstackVM
 
       script = []
       script << 'set -euo pipefail'
+      # If the container is already running and serving the correct model, skip
+      # the stop/pull/start cycle entirely — just wait for it to become ready.
+      # This recovers gracefully from a previous create that timed out during the
+      # readiness poll but left the container running successfully.
+      script << "if docker inspect --format='{{.State.Status}}' #{Shellwords.escape(container)} 2>/dev/null | grep -q '^running$'; then"
+      script << "  if curl -sf http://localhost:#{Shellwords.escape(port.to_s)}/v1/models 2>/dev/null | grep -q #{Shellwords.escape(model)}; then"
+      script << "    echo 'vLLM container already running with #{model}; skipping restart.'"
+      script << '    echo vllm-install-ok'
+      script << '    exit 0'
+      script << '  fi'
+      script << "  echo 'Container #{container} is running but not serving #{model}; restarting.'"
+      script << "  docker stop #{Shellwords.escape(container)} 2>/dev/null || true"
+      script << "  docker rm #{Shellwords.escape(container)} 2>/dev/null || true"
+      script << 'fi'
       script << "sudo mkdir -p #{Shellwords.escape(cache_dir)} #{Shellwords.escape(compile_cache)}"
       script << "sudo chmod -R 0777 #{Shellwords.escape(cache_dir)} #{Shellwords.escape(compile_cache)}"
-      script << "docker stop #{Shellwords.escape(container)} 2>/dev/null || true"
-      script << "docker rm #{Shellwords.escape(container)} 2>/dev/null || true"
       script << "docker pull #{Shellwords.escape(image)}" if pull_image
       script << docker_run
       # Stage patterns cover the full vLLM startup sequence:
@@ -229,18 +241,18 @@ module HyperstackVM
       script << 'echo "Waiting for vLLM to become ready (live progress from container logs)..."'
       script << "stage_pat='#{stage_pat}'"
       script << "strip_pfx='#{strip_pfx}'"
-      script << 'for i in $(seq 1 240); do'
+      script << 'for i in $(seq 1 360); do'
       script << "  if curl -sf http://localhost:#{port}/v1/models >/dev/null 2>&1; then echo vllm-ready; break; fi"
       script << "  state=$(docker inspect --format='{{.State.Status}}' #{Shellwords.escape(container)} 2>/dev/null || echo unknown)"
       script << "  progress=$(docker logs --tail 100 #{Shellwords.escape(container)} 2>&1 | grep -E \"$stage_pat\" | tail -1 | sed -E \"$strip_pfx\" | cut -c1-100)"
       script << '  if [ -n "$progress" ]; then'
-      script << '    echo "  vLLM ($i/240, $state): $progress"'
+      script << '    echo "  vLLM ($i/360, $state): $progress"'
       script << '  else'
-      script << '    echo "  vLLM not ready yet ($i/240, container=$state)..."'
+      script << '    echo "  vLLM not ready yet ($i/360, container=$state)..."'
       script << '  fi'
       script << '  sleep 5'
       script << 'done'
-      script << "curl -sf http://localhost:#{port}/v1/models >/dev/null || { echo 'FATAL: vLLM did not become ready within 20 minutes'; exit 1; }"
+      script << "curl -sf http://localhost:#{port}/v1/models >/dev/null || { echo 'FATAL: vLLM did not become ready within 30 minutes'; exit 1; }"
       script << 'echo vllm-install-ok'
       script.join("\n")
     end