shuriken: separate rsync sync CronJob (publish to web servers)

Per request, decouple publishing from generation: add a second
shuriken-sync CronJob that rsyncs /data/shuriken.sh/<site>/dist to
admin@fishfinger.buetow.org and admin@blowfish.buetow.org (with
--delete) every 4h -- far more often than the daily generate. rsync
is incremental, so post-generation publishes are cheap and a failed
publish retries on the next tick without re-running generation.

Reuses the shuriken image (now with openssh-client). Requires a
shuriken-rsync-ssh-key Secret holding the admin key authorized on
fishfinger/blowfish; until it exists the publish pods fail to mount
the key and publish nothing (safe by design -- no live publish
without the key). README documents the one-time Secret creation and
the image rebuild/push for openssh.

2 files changed, 118 insertions, 0 deletions

diff --git a/f3s/shuriken/README.md b/f3s/shuriken/README.md
index 2025023..62e4e3f 100644
--- a/f3s/shuriken/README.md
+++ b/f3s/shuriken/README.md
@@ -71,6 +71,36 @@ just sync            # refresh the ArgoCD app
 just argocd-status   # argocd CLI view
 ```
 
+## Publishing (separate rsync CronJob)
+
+`shuriken-sync` is a second CronJob that rsyncs the generated
+`/data/shuriken.sh/<site>/dist` trees to the public web servers
+(`admin@fishfinger.buetow.org` and `admin@blowfish.buetow.org`, with
+`--delete`) every 4h -- decoupled from the daily generation so a publish can
+be retried far more often than a (re)generate. It reuses the shuriken image
+(rsync + openssh-client) but overrides the command, so it never runs
+generation.
+
+It needs the admin SSH key authorized on fishfinger/blowfish. Until that key
+is provisioned the publish pods fail to mount the key and publish nothing (safe
+by design -- no live publish without the key). Create the Secret once:
+
+```bash
+kubectl create secret generic shuriken-rsync-ssh-key -n services \
+    --from-file=id_ed25519=/path/to/admin_ed25519
+```
+
+The generation CronJob deliberately has no `SYNC_*` settings -- it only writes
+to NFS; all publishing goes through `shuriken-sync`.
+
+The image includes `openssh-client` (alongside rsync) so the same image serves
+both jobs. If the registry still holds an older `shuriken:0.13.2` without it,
+rebuild and push before activating sync (the generation job is unaffected):
+
+```bash
+cd /home/paul/git/conf/f3s/shuriken && just build-push
+```
+
 ## ArgoCD
 
 `argocd-apps/services/shuriken.yaml` points ArgoCD at
diff --git a/f3s/shuriken/helm-chart/templates/sync-cronjob.yaml b/f3s/shuriken/helm-chart/templates/sync-cronjob.yaml
new file mode 100644
index 0000000..f3155c6
--- /dev/null
+++ b/f3s/shuriken/helm-chart/templates/sync-cronjob.yaml
@@ -0,0 +1,88 @@
+# shuriken-sync: a SEPARATE CronJob that publishes the already-generated
+# /data/shuriken.sh/<site>/dist trees to the public web servers (fishfinger +
+# blowfish) over rsync/SSH. Decoupled from the generation CronJob so it can run
+# more frequently than the daily generate -- rsync only transfers the diff, so
+# an incremental publish after generation is cheap, and a failed publish is
+# retried on the next tick without re-running the (expensive) generation.
+#
+# Requires a Secret `shuriken-rsync-ssh-key` holding the admin private key
+# authorized on fishfinger/blowfish (the same key the local `shuriken --sync`
+# uses). Until that Secret is provisioned this CronJob's pods fail to mount the
+# key and publish nothing -- safe by design (no live publish without the key).
+# Create it once:
+#   kubectl create secret generic shuriken-rsync-ssh-key -n services \
+#       --from-file=id_ed25519=/path/to/admin_ed25519
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: shuriken-sync
+  namespace: services
+spec:
+  # Every 4h -- 6x/day, far more often than the daily 04:00 generation. rsync
+  # is incremental, so after the first publish each tick only ships the diff.
+  schedule: "0 */4 * * *"
+  timeZone: Europe/Sofia
+  concurrencyPolicy: Forbid
+  startingDeadlineSeconds: 300
+  successfulJobsHistoryLimit: 3
+  failedJobsHistoryLimit: 3
+  jobTemplate:
+    spec:
+      # Transient SSH/rsync failures (gateway blip, NFS hiccup) are retried by
+      # the next tick; don't spam retries within one run.
+      backoffLimit: 0
+      activeDeadlineSeconds: 3600
+      template:
+        spec:
+          restartPolicy: Never
+          containers:
+            - name: shuriken-sync
+              image: registry.lan.buetow.org:30001/shuriken:0.13.2
+              imagePullPolicy: Always
+              command: ["/bin/sh", "-c"]
+              args:
+                - |
+                  set -euo pipefail
+                  export HOME=/tmp
+                  mkdir -p "$HOME/.ssh"
+                  SSH="ssh -i /ssh-keys/id_ed25519 -o StrictHostKeyChecking=accept-new -o BatchMode=yes"
+                  for site in irregular.ninja alt.irregular.ninja; do
+                    for srv in fishfinger.buetow.org blowfish.buetow.org; do
+                      echo "shuriken-sync: publishing $site -> $srv"
+                      rsync -a --delete --info=stats1 -e "$SSH" \
+                          "/data/shuriken.sh/$site/dist/" \
+                          "admin@$srv:/var/www/htdocs/$site/"
+                    done
+                  done
+                  echo "shuriken-sync: published both sites to both servers"
+              resources:
+                requests:
+                  cpu: 50m
+                  memory: 64Mi
+                limits:
+                  cpu: "1"
+                  memory: 512Mi
+              securityContext:
+                allowPrivilegeEscalation: false
+                runAsUser: 0
+                runAsGroup: 0
+              volumeMounts:
+                - name: data
+                  mountPath: /data
+                  readOnly: true
+                - name: ssh-keys
+                  mountPath: /ssh-keys
+                  readOnly: true
+                - name: tmp
+                  mountPath: /tmp
+          volumes:
+            - name: data
+              persistentVolumeClaim:
+                claimName: shuriken-data-pvc
+            - name: ssh-keys
+              secret:
+                secretName: shuriken-rsync-ssh-key
+                defaultMode: 0400
+            - name: tmp
+              emptyDir:
+                sizeLimit: 256Mi
\ No newline at end of file