docs: update all ROLLOUT*.md files with 1-min 33% canary details

3 files changed, 682 insertions, 578 deletions

diff --git a/f3s/ROLLOUTS-CHECKLIST.md b/f3s/ROLLOUTS-CHECKLIST.md
index b32f1ac..b475f2d 100644
--- a/f3s/ROLLOUTS-CHECKLIST.md
+++ b/f3s/ROLLOUTS-CHECKLIST.md
@@ -1,189 +1,222 @@
 # Argo Rollouts Deployment Checklist
 
-## Pre-Deployment Setup
-
-- [ ] Read `ARGO-ROLLOUTS-SUMMARY.md` to understand what was created
-- [ ] Ensure kubectl access to f3s cluster
-- [ ] Ensure ArgoCD is running and accessible
-- [ ] Git repository (conf.git) synced to git-server
+Quick checklist for deploying and testing Argo Rollouts with canary demo.
 
 ## Installation
 
+- [ ] Read `ARGO-ROLLOUTS-SUMMARY.md` - understand what was created
+- [ ] Ensure kubectl access to f3s cluster
+- [ ] Ensure ArgoCD is running
 - [ ] Navigate to `/home/paul/git/conf/f3s/argo-rollouts`
-- [ ] Run `just install` to deploy controller
-- [ ] Verify controller running: `kubectl get pods -n cicd -l app.kubernetes.io/name=argo-rollouts`
-- [ ] Verify CRD installed: `kubectl get crd | grep rollout`
-
-## Optional: Install kubectl Plugin
-
-- [ ] Download kubectl-argo-rollouts:
+- [ ] Run `just install`
+- [ ] Verify controller: `kubectl get pods -n cicd -l app.kubernetes.io/name=argo-rollouts`
+- [ ] Verify CRD: `kubectl get crd | grep rollout`
+- [ ] (Optional) Install plugin: 
   ```bash
   curl -LO https://github.com/argoproj/argo-rollouts/releases/latest/download/kubectl-argo-rollouts-linux-amd64
   chmod +x kubectl-argo-rollouts-linux-amd64
   sudo install -m 755 kubectl-argo-rollouts-linux-amd64 /usr/local/bin/kubectl-argo-rollouts
+  kubectl argo rollouts version
   ```
-- [ ] Verify: `kubectl argo rollouts version`
 
-## ArgoCD Syncing
+## ArgoCD Integration
 
-- [ ] Create/push `argocd-apps/cicd/argo-rollouts.yaml` to git
-- [ ] Create/push `argocd-apps/services/tracing-demo.yaml` updates to git
-- [ ] Force ArgoCD sync (wait 3 min or manual):
+- [ ] Push changes to git-server:
+  ```bash
+  cd /home/paul/git/conf/f3s
+  git add -A && git commit -m "feat: add Argo Rollouts"
+  git push r0 master
+  ```
+- [ ] Verify ArgoCD app:
   ```bash
-  argocd app sync argo-rollouts
-  argocd app sync tracing-demo
+  kubectl get application argo-rollouts -n cicd
+  argocd app get argo-rollouts
+  ```
+- [ ] Verify tracing-demo app:
+  ```bash
+  kubectl get application tracing-demo -n cicd
+  argocd app get tracing-demo
   ```
-- [ ] Verify tracing-demo application status: `argocd app get tracing-demo`
 
 ## Rollout Verification
 
-- [ ] Check frontend rollout deployed: `kubectl get rollout tracing-demo-frontend -n services`
+- [ ] Check rollout exists: `kubectl get rollout tracing-demo-frontend -n services`
 - [ ] Verify status: `kubectl describe rollout tracing-demo-frontend -n services`
-- [ ] Expected: `Status: Healthy` with `2/2 replicas` in stable state
-- [ ] Check pods running: `kubectl get pods -n services -l app=tracing-demo-frontend`
+- [ ] Expected: `Status: Healthy` with `3/3 replicas` in stable state
+- [ ] Check pods: `kubectl get pods -n services -l app=tracing-demo-frontend`
+- [ ] All 3 pods should be `Running`
+
+## Demo: Basic Canary Rollout
 
-## Basic Demo (First Time)
+**Expected: 0-15s: canary starting, 15-60s: observing, 60-90s: promoting**
 
 ### Terminal 1: Watch Rollout
 ```bash
 cd /home/paul/git/conf/f3s/tracing-demo
 just rollout-watch
 ```
-- [ ] Command running and connected
+- [ ] Command runs and connects to cluster
+- [ ] Waiting for rollout to start
 
-### Terminal 2: Generate Load (Optional)
+### Terminal 2: Trigger Rollout
 ```bash
-cd /home/paul/git/conf/f3s/tracing-demo
-just load-test &
+kubectl patch rollout tracing-demo-frontend -n services \
+  --type='json' \
+  -p='[{"op":"add","path":"/spec/template/spec/containers/0/env/-","value":{"name":"ROLLOUT_V","value":"'$(date +%s)'"}}]'
 ```
-- [ ] Requests being sent to frontend
+- [ ] Patch command successful
+- [ ] Terminal 1 shows change immediately
 
-### Terminal 3: Trigger Rollout
-Choose one method:
+### Terminal 1: Observe Progress
+- [ ] See `Step: 0/3, SetWeight: 33`
+- [ ] 1 canary pod becoming ready
+- [ ] 3 stable pods still running
+- [ ] After ~15 sec: canary pod ready
+- [ ] After ~60 sec: auto-promotion starts
+- [ ] After ~90 sec: all 3 pods running new version
+- [ ] Status shows `Healthy`
+
+## Demo: Abort/Rollback
+
+**Expected: Stop rollout and keep old version running**
 
-**Method A: Kubectl Patch (Fastest)**
+### Terminal 1: Watch Rollout
+```bash
+just rollout-watch
+```
+
+### Terminal 2: Trigger Rollout
 ```bash
 kubectl patch rollout tracing-demo-frontend -n services \
   --type='json' \
-  -p='[{"op":"replace","path":"/spec/template/spec/containers/0/image","value":"registry.lan.buetow.org:30001/tracing-demo-frontend:latest"}]'
+  -p='[{"op":"add","path":"/spec/template/spec/containers/0/env/-","value":{"name":"ROLLOUT_V2","value":"'$(date +%s)'"}}]'
+```
+
+### Terminal 3: Abort at Canary Step (after 20 seconds)
+```bash
+cd /home/paul/git/conf/f3s/tracing-demo
+just rollout-abort
 ```
-- [ ] Executed successfully
+- [ ] Abort command accepted
+- [ ] Terminal 1 shows `Status: Aborted`
+- [ ] Canary pods terminate
+- [ ] Old 3 pods continue running
+- [ ] Verify with: `just rollout-status`
 
-**Method B: Git + ArgoCD (Most GitOps)**
+## Demo: Load Testing
+
+**Expected: Generate traffic while rollout happens**
+
+### Terminal 1: Watch Rollout
+```bash
+just rollout-watch
+```
+
+### Terminal 2: Start Load Test
 ```bash
-cd /home/paul/git/conf/f3s
-# Edit tracing-demo/helm-chart/templates/frontend-rollout.yaml (change image tag)
-git add -A
-git commit -m "chore: update frontend image for demo"
-git remote add r0 ssh://git@r0:30022/repos/conf.git 2>/dev/null || true
-git push r0 master
-kubectl annotate application tracing-demo -n cicd argocd.argoproj.io/refresh=normal --overwrite
+just load-test &
+```
+- [ ] Requests being sent
+
+### Terminal 3: Trigger Rollout
+```bash
+kubectl patch rollout tracing-demo-frontend -n services \
+  --type='json' \
+  -p='[{"op":"add","path":"/spec/template/spec/containers/0/env/-","value":{"name":"ROLLOUT_V3","value":"'$(date +%s)'"}}]'
 ```
-- [ ] Git push successful
-- [ ] ArgoCD syncing (check web UI or CLI)
+- [ ] Rollout progresses with active traffic
+- [ ] Both old and new pods serve requests during canary phase
 
-## Demo Observation
+## Monitoring
 
-- [ ] Terminal 1 shows: "Progressing" → "canary step 1/3"
-- [ ] After ~30 sec: New canary pod appears
-- [ ] After ~2 min: "canary step 2/3" (pause)
-- [ ] After ~4 min: "canary step 3/3" (100% traffic)
-- [ ] After ~4:20 min: Status shows "Healthy"
-- [ ] Old pods terminated, 2 new pods in stable state
+- [ ] Check status: `kubectl argo rollouts status tracing-demo-frontend -n services`
+- [ ] Detailed info: `kubectl argo rollouts describe rollout tracing-demo-frontend -n services`
+- [ ] Pod details: `kubectl get pods -n services -l app=tracing-demo-frontend -o wide`
+- [ ] View logs: `just logs-frontend`
+- [ ] View history: `just rollout-history`
 
-## Monitoring (Optional)
+## Grafana (Optional)
 
-- [ ] Check logs: `just logs-frontend`
-- [ ] Check Grafana Tempo for traces: https://grafana.f3s.buetow.org
-  - [ ] Navigate to Explore → Tempo
-  - [ ] Query: `{ resource.service.name = "frontend" }`
-  - [ ] See traces from old and new versions
-- [ ] Check Prometheus metrics: Port-forward and query
+- [ ] Open Grafana: https://grafana.f3s.buetow.org
+- [ ] Navigate to Explore → Tempo datasource
+- [ ] Query: `{ resource.service.name = "frontend" }`
+- [ ] See traces from old and new versions during rollout
 
-## Advanced Scenarios
+## Integration with Git (GitOps)
 
-### Scenario 1: Manual Promotion
-- [ ] Trigger rollout (step above)
-- [ ] After step 1 (30 sec), run:
+- [ ] Edit rollout config:
   ```bash
-  just rollout-promote
+  nano /home/paul/git/conf/f3s/tracing-demo/helm-chart/templates/frontend-rollout.yaml
   ```
-- [ ] Watch rollout skip step 2, immediately promote to 100%
-- [ ] Verify: `just rollout-status` shows "Healthy"
-
-### Scenario 2: Abort/Rollback
-- [ ] Trigger rollout
-- [ ] While progressing, run:
+- [ ] Change any settings (e.g., duration, setWeight)
+- [ ] Commit and push:
   ```bash
-  just rollout-abort
+  git add -A && git commit -m "chore: adjust canary settings"
+  git push r0 master
   ```
-- [ ] Watch canary pods terminate
-- [ ] Old version continues running
-- [ ] Verify: `just rollout-status` shows "Aborted"
-
-### Scenario 3: Check History
-- [ ] After any rollout:
+- [ ] ArgoCD auto-syncs within 3 minutes (or force):
   ```bash
-  just rollout-history
+  kubectl annotate application tracing-demo -n cicd argocd.argoproj.io/refresh=normal --overwrite
   ```
-- [ ] See previous revisions and their status
-
-## Integration with CI/CD
+- [ ] New settings take effect on next rollout trigger
 
-- [ ] Image builds automatically on git push (or configured pipeline)
-- [ ] New image pushed to registry: `registry.lan.buetow.org:30001/tracing-demo-frontend:NEWTAG`
-- [ ] Git updated with new image tag
-- [ ] ArgoCD detects change
-- [ ] Rollout automatically triggered
-- [ ] Canary strategy executes
+## Post-Demo
 
-## Post-Deployment
+- [ ] Abort any stuck rollouts: `just rollout-abort`
+- [ ] Verify stable state: `just rollout-status` shows `Healthy`
+- [ ] Review documentation:
+  - [ ] `ARGO-ROLLOUTS-SUMMARY.md` - architecture
+  - [ ] `ROLLOUTS-SETUP.md` - detailed scenarios
+  - [ ] `README-ROLLOUTS.md` - quick reference
+  - [ ] `tracing-demo/ROLLOUTS-DEMO.md` - technical details
 
-- [ ] Share documentation:
-  - [ ] `ROLLOUTS-SETUP.md` - Complete setup guide
-  - [ ] `tracing-demo/ROLLOUTS-DEMO.md` - Detailed walkthrough
-  - [ ] `ARGO-ROLLOUTS-SUMMARY.md` - Architecture overview
-- [ ] Add team to `kubectl argo rollouts` usage
-- [ ] Consider next steps:
-  - [ ] Deploy Istio for advanced traffic management
-  - [ ] Add Flagger for automated analysis
-  - [ ] Extend to other services (middleware, backend)
-  - [ ] Create monitoring dashboards
-
-## Troubleshooting Checklist
+## Troubleshooting
 
 ### Controller not running
-- [ ] Check pod: `kubectl get pods -n cicd -l app.kubernetes.io/name=argo-rollouts`
-- [ ] Check logs: `kubectl logs -n cicd -l app.kubernetes.io/name=argo-rollouts`
-- [ ] Check CRD: `kubectl get crd | grep rollout`
+```bash
+kubectl get pods -n cicd -l app.kubernetes.io/name=argo-rollouts
+kubectl logs -n cicd -l app.kubernetes.io/name=argo-rollouts
+```
+- [ ] Pod running and ready
 
-### Rollout not deploying
-- [ ] Check ArgoCD sync: `argocd app get tracing-demo`
-- [ ] Check git changes pushed: `git log --oneline | head -5`
-- [ ] Force sync: `argocd app sync tracing-demo --prune`
+### Rollout not deployed
+```bash
+kubectl get rollout tracing-demo-frontend -n services
+kubectl describe rollout tracing-demo-frontend -n services
+```
+- [ ] Check events section for errors
 
-### Canary pods not starting
-- [ ] Check pod status: `kubectl describe pod -n services <pod-name>`
-- [ ] Check logs: `kubectl logs -n services <pod-name>`
-- [ ] Check resource limits: `kubectl top pods -n services`
-- [ ] Check image: `kubectl get pods -n services -o jsonpath='{.items[*].spec.containers[0].image}'`
+### Canary pods in ImagePullBackoff
+- [ ] Use env var patch instead (don't change image tag):
+  ```bash
+  kubectl patch rollout tracing-demo-frontend -n services \
+    --type='json' \
+    -p='[{"op":"add","path":"/spec/template/spec/containers/0/env/-","value":{"name":"ROLLOUT_V","value":"'$(date +%s)'"}}]'
+  ```
 
 ### Rollout stuck in Progressing
-- [ ] Check health probes: `kubectl get rollout tracing-demo-frontend -n services -o yaml | grep -A 10 health`
-- [ ] Check replica status: `kubectl get rs -n services -l app=tracing-demo-frontend -o wide`
-- [ ] Check controller logs: `kubectl logs -n cicd -l app.kubernetes.io/name=argo-rollouts --tail=50`
+```bash
+kubectl describe rollout tracing-demo-frontend -n services
+kubectl get pods -n services -l app=tracing-demo-frontend
+```
+- [ ] Check pod readiness probes
+- [ ] Check pod resource requests/limits
+- [ ] Check controller logs
 
-## Cleanup (If Needed)
+## Next Steps
 
-- [ ] Stop rollout: `kubectl argo rollouts abort tracing-demo-frontend -n services`
-- [ ] Rollback to previous: `kubectl rollout undo deployment/tracing-demo-frontend -n services` (if needed)
-- [ ] Uninstall Argo Rollouts: `cd argo-rollouts && just uninstall`
+- [ ] Run through all demo scenarios multiple times
+- [ ] Modify rollout settings and observe behavior
+- [ ] Monitor with Prometheus/Grafana
+- [ ] Extend to other services (middleware, backend)
+- [ ] Optional: Install Istio for advanced traffic routing
+- [ ] Optional: Deploy Flagger for automated analysis
 
 ---
 
-**Setup complete when:**
-- ✅ Argo Rollouts controller running in `cicd` namespace
-- ✅ Frontend rollout deployed in `services` namespace
-- ✅ ArgoCD recognizes rollout resource
-- ✅ One demo run successful (git trigger or kubectl patch)
-- ✅ Team can watch and manage rollouts
+**Setup Complete When:**
+- ✅ Controller running in `cicd` namespace
+- ✅ Rollout deployed in `services` namespace
+- ✅ One full demo executed (0-90 seconds)
+- ✅ Can abort and retry
+- ✅ Team trained on canary deployments
diff --git a/f3s/ROLLOUTS-SETUP.md b/f3s/ROLLOUTS-SETUP.md
index b7ebb55..0ea965c 100644
--- a/f3s/ROLLOUTS-SETUP.md
+++ b/f3s/ROLLOUTS-SETUP.md
@@ -1,6 +1,6 @@
 # Argo Rollouts Setup and Demo Guide
 
-This guide covers the complete setup and demonstration of Argo Rollouts with the tracing-demo application.
+Complete setup and demonstration of Argo Rollouts with the tracing-demo application. Canary strategy: 33% traffic (1 pod) for 1 minute, then auto-promote to 100%.
 
 ## Quick Setup
 
@@ -32,19 +32,11 @@ kubectl argo rollouts version
 
 ### 3. Sync ArgoCD with New Applications
 
-The following ArgoCD Applications will be auto-synced:
-
-- **argo-rollouts.yaml** - Installs Argo Rollouts controller
-- **tracing-demo.yaml** - Now uses Rollout (frontend) + Deployments (middleware, backend)
-
-Force ArgoCD to sync:
 ```bash
 argocd app sync argo-rollouts
 argocd app sync tracing-demo
 ```
 
-Or wait for auto-sync (default: 3 minutes).
-
 ### 4. Verify Rollout is Deployed
 
 ```bash
@@ -52,378 +44,330 @@ kubectl get rollout tracing-demo-frontend -n services
 kubectl describe rollout tracing-demo-frontend -n services
 ```
 
-Expected status: `Stable` with `2/2 replicas`.
-
-## Demo Scenarios
-
-### Scenario 1: Basic Canary Rollout (Guided)
+Expected status: `Healthy` with `3/3 replicas` in stable state.
 
-**Duration**: ~5-10 minutes
+## Quick Demo (90 seconds)
 
-**Objective**: Observe frontend rollout from 50% → 100% traffic with auto-promotion.
+### Terminal 1 - Watch Progress
 
-#### Step 1: Prepare Terminals
-
-Terminal 1 - Watch rollout progress:
 ```bash
 cd /home/paul/git/conf/f3s/tracing-demo
 just rollout-watch
 ```
 
-Terminal 2 - Generate load:
-```bash
-cd /home/paul/git/conf/f3s/tracing-demo
-just load-test &
-```
-
-Terminal 3 - Trigger rollout:
+Or use the kubectl command directly:
 ```bash
-# Will use this in next step
+kubectl argo rollouts get rollout tracing-demo-frontend -n services --watch
 ```
 
-#### Step 2: Trigger Rollout (Terminal 3)
+### Terminal 2 - Trigger Rollout
 
-Simulate updating the frontend image:
+Wait 10 seconds for Terminal 1 to start watching, then trigger:
 
 ```bash
 kubectl patch rollout tracing-demo-frontend -n services \
   --type='json' \
-  -p='[{"op":"replace","path":"/spec/template/spec/containers/0/image","value":"registry.lan.buetow.org:30001/tracing-demo-frontend:latest"}]'
+  -p='[{"op":"add","path":"/spec/template/spec/containers/0/env/-","value":{"name":"ROLLOUT_V","value":"'$(date +%s)'"}}]'
 ```
 
-Or via git (more GitOps-like):
+### Watch the Timeline
 
-```bash
-cd /home/paul/git/conf/f3s
-# Edit tracing-demo/helm-chart/templates/frontend-rollout.yaml (change image tag)
-git add -A
-git commit -m "chore: update frontend image for demo"
-git remote add r0 ssh://git@r0:30022/repos/conf.git 2>/dev/null || true
-git push r0 master
+**Terminal 1 will show:**
 
-# Trigger ArgoCD sync
-kubectl annotate application tracing-demo -n cicd argocd.argoproj.io/refresh=normal --overwrite
 ```
-
-#### Step 3: Observe Rollout (Terminal 1)
-
-Watch the output:
-
+Step: 0/3
+SetWeight: 33
+Canary: 1 pod (new version) - starting
+Stable: 3 pods (old version) - handling requests
 ```
-NAME                           KIND        STATUS     AGE    INFO
-tracing-demo-frontend          Rollout     Progressing  0s    canary step 1/3
-tracing-demo-frontend-abc123   ReplicaSet  ✓ canary    5s    1/1 replicas
-tracing-demo-frontend-xyz789   ReplicaSet  ✓ stable    5m    2/2 replicas
-
-NAME                           KIND        STATUS     AGE    INFO
-tracing-demo-frontend          Rollout     Progressing  2m5s  canary step 2/3
-tracing-demo-frontend-abc123   ReplicaSet  ✓ canary    2m    1/1 replicas (ready)
-tracing-demo-frontend-xyz789   ReplicaSet  ✓ stable    5m    2/2 replicas
 
-NAME                           KIND        STATUS     AGE    INFO
-tracing-demo-frontend          Rollout     Progressing  4m10s canary step 3/3
-tracing-demo-frontend-abc123   ReplicaSet  ✓ canary    4m    2/2 replicas (ready, updated)
-tracing-demo-frontend-xyz789   ReplicaSet  ✓ stable    5m    0/2 replicas (pending termination)
+→ After 15 seconds, canary pod becomes ready:
 
-NAME                           KIND        STATUS  AGE    INFO
-tracing-demo-frontend          Rollout     ✓ Healthy  4m20s
-tracing-demo-frontend-abc123   ReplicaSet  ✓ stable  4m    2/2 replicas
+```
+Step: 1/3
+SetWeight: 33
+Canary: 1 pod (new version) - ready, receiving 33% traffic
+Stable: 3 pods (old version) - receiving 67% traffic
 ```
 
-**Timeline:**
-- **0-2 min**: Step 1 (setWeight: 50) - 1 canary pod, 2 stable pods, 50/50 traffic
-- **2-4 min**: Step 2 (pause: 2m) - Waiting for user or auto-promotion
-- **4+ min**: Step 3 (setWeight: 100) - All 2 canary pods promoted, old pods terminated
-- **4:20 min**: Complete - New version fully deployed
-
-#### Step 4: Observe Behavior (Optional)
+→ After ~60 seconds, auto-promotion begins:
 
-Check request latency/errors during rollout:
+```
+Step: 2/3
+SetWeight: 100
+Canary scaling → Stable
+```
 
-```bash
-# View logs from both old and new pods
-kubectl logs -n services -l app=tracing-demo-frontend --timestamps=true | tail -20
+→ After ~90 seconds, complete:
 
-# Check if any requests failed during transition
-grep -i "error\|exception" <(kubectl logs -n services -l app=tracing-demo-frontend)
+```
+Status: Healthy
+Replicas: 3/3 all running new version
 ```
 
-View traces in Grafana:
-1. Navigate to https://grafana.f3s.buetow.org
-2. Explore → Tempo
-3. Query: `{ resource.service.name = "frontend" }`
-4. See traces from both old and new versions
+## Demo Scenarios
 
-### Scenario 2: Manual Promotion (Skip Waiting)
+### Scenario 1: Observe the Full Rollout
 
-**Duration**: ~2 minutes
+Just follow the "Quick Demo" above. Watch all three steps progress automatically over 90 seconds.
 
-**Objective**: Demonstrate manual control - don't wait for auto-promotion.
+### Scenario 2: Abort Rollout (Simulate Failure)
 
-#### Setup
+**Terminal 1**: Watch the rollout
+```bash
+just rollout-watch
+```
 
-Trigger rollout (same as Scenario 1):
+**Terminal 2**: Trigger rollout
 ```bash
 kubectl patch rollout tracing-demo-frontend -n services \
   --type='json' \
-  -p='[{"op":"replace","path":"/spec/template/spec/containers/0/image","value":"registry.lan.buetow.org:30001/tracing-demo-frontend:latest"}]'
+  -p='[{"op":"add","path":"/spec/template/spec/containers/0/env/-","value":{"name":"ROLLOUT_V","value":"'$(date +%s)'"}}]'
 ```
 
-Watch:
+**Terminal 3 (while at step 1)**: Abort the rollout
 ```bash
-just rollout-watch
+cd /home/paul/git/conf/f3s/tracing-demo
+just rollout-abort
 ```
 
-#### Promote Early
-
-After canary looks healthy (step 1 complete, ~30 seconds):
+Result:
+- Canary pods terminate
+- Old 3 pods continue running
+- Status shows "Aborted"
 
+Verify:
 ```bash
-cd /home/paul/git/conf/f3s/tracing-demo
-just rollout-promote
+just rollout-status
 ```
 
-This skips the 2-minute pause and immediately promotes to 100%.
-
-### Scenario 3: Abort/Rollback
-
-**Duration**: ~3 minutes
-
-**Objective**: Demonstrate rollback if canary fails.
-
-#### Setup & Trigger
-
-Same as Scenario 1.
-
-#### Simulate Failure
-
-While at canary step 1 (50% traffic), introduce a failure:
+### Scenario 3: Load Testing During Rollout
 
+**Terminal 1**: Watch rollout
 ```bash
-# Get one of the new canary pods
-CANARY_POD=$(kubectl get pods -n services -l app=tracing-demo-frontend -o name | tail -1)
-
-# Kill it to simulate crash
-kubectl delete $CANARY_POD -n services
+just rollout-watch
 ```
 
-Watch in Terminal 1 - the rollout may stall or fail health checks.
-
-#### Abort
-
+**Terminal 2**: Start load test
 ```bash
-cd /home/paul/git/conf/f3s/tracing-demo
-just rollout-abort
+just load-test &
 ```
 
-This:
-- Stops the rollout
-- Terminates canary replicas
-- Restores stable version with 2 pods
-- Allows investigation
-
-Verify:
+**Terminal 3**: Trigger rollout
 ```bash
-just rollout-status
+kubectl patch rollout tracing-demo-frontend -n services \
+  --type='json' \
+  -p='[{"op":"add","path":"/spec/template/spec/containers/0/env/-","value":{"name":"ROLLOUT_V","value":"'$(date +%s)'"}}]'
 ```
 
-Expected: `Rollout has been aborted. Stable ReplicaSet: 2/2 replicas`
+Load test will hit both old and new pods during the 1-minute canary window.
 
-### Scenario 4: Observability - Prometheus Metrics
-
-**Duration**: ~5 minutes (during any rollout)
-
-**Objective**: Monitor rollout via Prometheus metrics.
-
-During a running rollout:
+### Scenario 4: Check Logs During Rollout
 
+**Terminal 1**: Watch rollout
 ```bash
-# Port-forward Prometheus
-kubectl port-forward -n monitoring svc/prometheus 9090:9090 &
+just rollout-watch
+```
 
-# Open browser: http://localhost:9090
+**Terminal 2**: Trigger rollout
+```bash
+kubectl patch rollout tracing-demo-frontend -n services \
+  --type='json' \
+  -p='[{"op":"add","path":"/spec/template/spec/containers/0/env/-","value":{"name":"ROLLOUT_V","value":"'$(date +%s)'"}}]'
 ```
 
-Query useful metrics:
+**Terminal 3**: Watch logs
+```bash
+kubectl logs -n services -l app=tracing-demo-frontend -f --tail=20
+```
 
-```promql
-# Rollout replica counts
-kube_statefulset_replicas{statefulset=~".*frontend.*"}
-kube_replicaset_created{replicaset=~".*frontend.*"}
+See logs from both old and new pods.
 
-# Pod status during rollout
-kube_pod_status_phase{namespace="services", pod=~".*frontend.*"}
+### Scenario 5: Monitor via Grafana Tempo (Distributed Tracing)
 
-# Request latency (if your app exports metrics)
-rate(http_requests_total{job="frontend"}[5m])
+**Terminal 1**: Watch rollout
+```bash
+just rollout-watch
+```
 
-# Error rate
-rate(http_requests_total{job="frontend", status=~"5.."}[5m])
+**Terminal 2**: Trigger rollout
+```bash
+kubectl patch rollout tracing-demo-frontend -n services \
+  --type='json' \
+  -p='[{"op":"add","path":"/spec/template/spec/containers/0/env/-","value":{"name":"ROLLOUT_V","value":"'$(date +%s)'"}}]'
 ```
 
-### Scenario 5: GitOps Flow (Realistic)
+**Terminal 3**: Open Grafana
+1. Navigate to https://grafana.f3s.buetow.org
+2. Go to Explore → Select "Tempo" datasource
+3. Query: `{ resource.service.name = "frontend" }`
+4. See traces from both old and new versions during canary phase
+
+## Timeline Breakdown
 
-**Duration**: ~10 minutes
+| Time | Event | Status |
+|------|-------|--------|
+| 0s | Trigger rollout | Rollout starts |
+| 0-5s | Canary pod created | `Step 0/3: SetWeight 33` |
+| 5-15s | Canary pod becoming ready | Still not ready |
+| 15s | Canary pod ready | `Step 1/3: SetWeight 33, canary ready` |
+| 15-60s | Observing canary | Requests split 67/33 (old/new) |
+| 60s | Auto-promotion triggered | `Step 2/3: SetWeight 100` |
+| 60-70s | Scaling new pods | Canary → Stable |
+| 70-80s | Terminating old pods | Old pods scaling down |
+| ~90s | Complete | `Status: Healthy, 3/3 replicas` |
 
-**Objective**: Demonstrate GitOps workflow - git commit triggers rollout via ArgoCD.
+## Monitoring During Rollout
 
-#### Step 1: Modify Frontend Code
+### kubectl Commands
 
+Real-time status:
 ```bash
-cd /home/paul/git/conf/f3s/tracing-demo/docker/frontend
-# Edit app.py (e.g., change response message)
-# Commit and push
-git add -A
-git commit -m "feat: update frontend message"
-git push origin master
+kubectl argo rollouts get rollout tracing-demo-frontend -n services --watch
 ```
 
-#### Step 2: Rebuild and Push Image
-
+Check specific details:
 ```bash
-cd /home/paul/git/conf/f3s/tracing-demo
-just build-push
+kubectl argo rollouts describe rollout tracing-demo-frontend -n services
+kubectl argo rollouts history tracing-demo-frontend -n services
 ```
 
-This creates new Docker image tagged with latest commit hash or timestamp.
-
-#### Step 3: Update Helm Chart
-
+Pod status:
 ```bash
-# Edit frontend-rollout.yaml with new image tag
-nano /home/paul/git/conf/f3s/tracing-demo/helm-chart/templates/frontend-rollout.yaml
-# Change image: registry.lan.buetow.org:30001/tracing-demo-frontend:NEWTAG
-
-git add -A
-git commit -m "chore: update frontend rollout image to latest"
-git remote add r0 ssh://git@r0:30022/repos/conf.git 2>/dev/null || true
-git push r0 master
+kubectl get pods -n services -l app=tracing-demo-frontend -o wide
 ```
 
-#### Step 4: ArgoCD Syncs Automatically
+### Prometheus Metrics
 
-Wait 3 minutes or force sync:
 ```bash
-argocd app sync tracing-demo --prune
+# Port-forward Prometheus
+kubectl port-forward -n monitoring svc/prometheus 9090:9090
 ```
 
-ArgoCD detects the new image in git and updates the rollout.
+Then query:
+```promql
+# Pod counts during rollout
+kube_replicaset_replicas{replicaset=~"tracing-demo-frontend.*"}
 
-#### Step 5: Watch Rollout Progress
+# Pod status
+kube_pod_status_phase{namespace="services", pod=~"tracing-demo-frontend.*"}
 
-```bash
-just rollout-watch
+# Pod age (shows which are old vs new)
+time() - kube_pod_created{namespace="services", pod=~"tracing-demo-frontend.*"}
 ```
 
-The canary strategy executes: 50% → wait 2min → 100%.
+### Grafana Dashboards
 
-## Monitoring Dashboard
+1. Open Grafana: https://grafana.f3s.buetow.org
+2. Explore → Tempo datasource
+3. Query: `{ resource.service.name = "frontend" }`
+4. See traces from old and new versions
+5. Notice latency/error differences during rollout
 
-Create a Grafana dashboard to visualize rollout progress:
+## Rollout Configuration
 
-1. Open Grafana: https://grafana.f3s.buetow.org
-2. Dashboards → New → Create
-3. Add panels:
+Located in: `/home/paul/git/conf/f3s/tracing-demo/helm-chart/templates/frontend-rollout.yaml`
 
-**Panel 1: Rollout Status**
-```promql
-kube_rollout_status_current_step{rollout="tracing-demo-frontend"}
+Key settings:
+```yaml
+replicas: 3  # 3 pods total
+strategy:
+  canary:
+    steps:
+    - setWeight: 33       # Send 1 pod (33%) to canary
+    - pause:
+        duration: 1m      # Wait 1 minute, then auto-promote
+    - setWeight: 100      # Promote all to new version
 ```
 
-**Panel 2: Replica Counts**
-```promql
-topk(2, kube_replicaset_replicas{replicaset=~"tracing-demo-frontend.*"})
-```
+To modify pause duration:
+```bash
+# Edit the file
+nano /home/paul/git/conf/f3s/tracing-demo/helm-chart/templates/frontend-rollout.yaml
 
-**Panel 3: Pod Age**
-```promql
-time() - kube_pod_created{namespace="services", pod=~"tracing-demo-frontend.*"}
+# Change duration: 1m to duration: 5m (for example)
+# Then commit and push
+git add -A && git commit -m "chore: extend canary pause to 5 minutes"
+git push r0 master
 ```
 
-**Panel 4: Request Rate**
-```promql
-rate(http_requests_total{job="tracing-demo-frontend"}[1m])
-```
+ArgoCD will auto-sync the new rollout configuration.
 
-## Advanced: Custom Analysis
+## Troubleshooting
 
-To add automated health checks during canary (e.g., error rate thresholds), integrate with **Flagger**:
+### Rollout shows "ErrImagePull" on canary pod
 
-```yaml
-apiVersion: flagger.app/v1beta1
-kind: Canary
-metadata:
-  name: tracing-demo-frontend
-spec:
-  targetRef:
-    apiVersion: argoproj.io/v1alpha1
-    kind: Rollout
-    name: tracing-demo-frontend
-  progressDeadlineSeconds: 300
-  service:
-    port: 5000
-  analysis:
-    interval: 1m
-    threshold: 2
-    maxWeight: 50
-    stepWeight: 10
-    metrics:
-    - name: error_rate
-      thresholdRange:
-        max: 1  # Max 1% error rate
-```
-
-This requires installing **Flagger** and requires a service mesh (Istio/Linkerd).
+This happens if using an image tag that doesn't exist. The env var patch approach forces a rollout without changing the image, so use:
 
-## Troubleshooting
+```bash
+kubectl patch rollout tracing-demo-frontend -n services \
+  --type='json' \
+  -p='[{"op":"add","path":"/spec/template/spec/containers/0/env/-","value":{"name":"ROLLOUT_V","value":"'$(date +%s)'"}}]'
+```
 
-### Rollout Stuck in Progressing
+### Rollout stuck in "Progressing"
 
+Check pod status:
 ```bash
 kubectl describe rollout tracing-demo-frontend -n services
+kubectl get pods -n services -l app=tracing-demo-frontend
 ```
 
-Check for:
-- Pod failures (CrashLoopBackOff)
-- Image pull errors
-- Resource exhaustion
-- Health probe failures
+Check controller logs:
+```bash
+kubectl logs -n cicd -l app.kubernetes.io/name=argo-rollouts --tail=50
+```
 
-### Canary Pods Not Becoming Ready
+### Controller not running
 
 ```bash
-kubectl get pods -n services -l app=tracing-demo-frontend -o wide
-kubectl logs -n services -l app=tracing-demo-frontend --tail=50
+kubectl get pods -n cicd -l app.kubernetes.io/name=argo-rollouts
+kubectl logs -n cicd -l app.kubernetes.io/name=argo-rollouts
 ```
 
-### ArgoCD Not Syncing Rollout Changes
+### Auto-promotion not happening
 
+Verify pause duration is set:
 ```bash
-kubectl get application tracing-demo -n cicd -o jsonpath='{.status.sync.status}'
-argocd app sync tracing-demo
+kubectl get rollout tracing-demo-frontend -n services -o yaml | grep -A 5 "pause:"
 ```
 
-### kubectl argo rollouts Plugin Issues
+## Advanced: Modify Canary Parameters
+
+### Increase observation time to 5 minutes
 
 ```bash
-kubectl argo rollouts version
+# Edit rollout YAML
+nano /home/paul/git/conf/f3s/tracing-demo/helm-chart/templates/frontend-rollout.yaml
 
-# If not installed or outdated:
-curl -LO https://github.com/argoproj/argo-rollouts/releases/latest/download/kubectl-argo-rollouts-linux-amd64
-sudo install -m 755 kubectl-argo-rollouts-linux-amd64 /usr/local/bin/kubectl-argo-rollouts
+# Change:
+#   - pause:
+#       duration: 1m
+# To:
+#   - pause:
+#       duration: 5m
+
+git add -A && git commit -m "chore: extend canary pause to 5 minutes"
+git push r0 master
+```
+
+### Reduce traffic weight to canary (more conservative)
+
+```yaml
+steps:
+- setWeight: 10       # Only 10% traffic (0.3 pods worth)
+- pause:
+    duration: 2m      # Observe longer
+- setWeight: 100
 ```
 
-## Next Steps
+### Add health check analysis (requires Flagger or ArgoCD Analysis)
 
-1. **Try all scenarios** to understand rollout behavior
-2. **Deploy Istio** for advanced traffic management (weighted routing, header-based)
-3. **Add Prometheus queries** to monitor rollout metrics
-4. **Implement Flagger** for automated analysis and rollback
-5. **Migrate other services** to Rollout (start with low-risk apps)
+For automated rollback based on error rate thresholds, see `/home/paul/git/conf/f3s/ROLLOUTS-SETUP.md` → "Advanced: Custom Analysis" section.
 
 ## References
 
 - [Argo Rollouts Canary Strategy](https://argoproj.github.io/argo-rollouts/features/canary/)
-- [Argo Rollouts Blue-Green Strategy](https://argoproj.github.io/argo-rollouts/features/bluegreen/)
-- [Flagger Documentation](https://flagger.app/)
-- [Istio VirtualService](https://istio.io/latest/docs/reference/config/networking/virtual-service/)
+- [Argo Rollouts Best Practices](https://argoproj.github.io/argo-rollouts/best-practices/)
+- [kubectl-argo-rollouts Plugin](https://argoproj.github.io/argo-rollouts/getting-started/#using-kubectl-with-argo-rollouts)
+- [Flagger for Automated Analysis](https://flagger.app/)
diff --git a/f3s/tracing-demo/ROLLOUTS-DEMO.md b/f3s/tracing-demo/ROLLOUTS-DEMO.md
index 53a43b3..775a581 100644
--- a/f3s/tracing-demo/ROLLOUTS-DEMO.md
+++ b/f3s/tracing-demo/ROLLOUTS-DEMO.md
@@ -1,317 +1,444 @@
-# Argo Rollouts Demo Guide for Tracing-Demo
+# Argo Rollouts Demo - Technical Details
 
-This guide demonstrates progressive delivery using Argo Rollouts with the tracing-demo frontend service.
+Detailed technical walkthrough of Argo Rollouts canary strategy for tracing-demo frontend.
 
-## Prerequisites
+## Quick Demo (90 seconds)
 
-- Argo Rollouts installed in `cicd` namespace
-- ArgoCD synced with the latest conf.git
-- tracing-demo-frontend rollout deployed
-- kubectl argo rollouts plugin installed
-
-### Install kubectl argo rollouts plugin
+### Setup
 
 ```bash
-curl -LO https://github.com/argoproj/argo-rollouts/releases/latest/download/kubectl-argo-rollouts-linux-amd64
-chmod +x kubectl-argo-rollouts-linux-amd64
-sudo mv kubectl-argo-rollouts-linux-amd64 /usr/local/bin/kubectl-argo-rollouts
-```
+# Terminal 1: Watch the rollout
+cd /home/paul/git/conf/f3s/tracing-demo
+just rollout-watch
 
-## Demo Workflow
+# Terminal 2: Trigger the rollout (after Terminal 1 is watching)
+kubectl patch rollout tracing-demo-frontend -n services \
+  --type='json' \
+  -p='[{"op":"add","path":"/spec/template/spec/containers/0/env/-","value":{"name":"ROLLOUT_V","value":"'$(date +%s)'"}}]'
+```
 
-### 1. Verify Current State
+### Execution Timeline
 
+**t=0-15s: Canary Launch**
 ```bash
-# Check frontend rollout
-kubectl get rollout tracing-demo-frontend -n services
-
-# Get detailed status
-kubectl argo rollouts status tracing-demo-frontend -n services
-kubectl argo rollouts get rollout tracing-demo-frontend -n services
+# Terminal 1 shows:
+Name:            tracing-demo-frontend
+Status:          ◌ Progressing
+Strategy:        Canary
+  Step:          0/3
+  SetWeight:     33
+  ActualWeight:  0
+Images:          (new) tracing-demo-frontend (canary)
+                 (old) tracing-demo-frontend (stable)
+Replicas:
+  Desired:       3
+  Current:       4  # 3 stable + 1 canary being created
+  Updated:       1
+  Ready:         3  # 3 stable pods ready, canary still starting
 ```
 
-Expected output shows 2 stable replicas, 0 canary.
-
-### 2. Generate Load (Optional but Recommended)
+**t=15-60s: Canary Observation**
+```bash
+# After canary pod becomes ready (~15 seconds)
+Status:          ◌ Progressing
+  Step:          1/3    # Now in pause step
+  SetWeight:     33
+  ActualWeight:  33     # Actual weight achieved
+Replicas:
+  Desired:       3
+  Current:       4
+  Updated:       1
+  Ready:         4      # All 4 pods (3 stable + 1 canary) ready
+  Available:     4
+```
 
-In a separate terminal, generate traffic to the frontend:
+Service routes traffic:
+- **Old version**: 3 pods → ~67% traffic
+- **New version**: 1 pod → ~33% traffic
 
+**t=60s: Auto-Promotion**
 ```bash
-# Port-forward frontend
-kubectl port-forward -n services svc/frontend-service 50