| Age | Commit message (Collapse) | Author |
|---|
|
output_dir\n\nThe comfyui_install_script previously ran\n\n chmod -R 0777 File.dirname(models_dir)\n\nwhich chmods the *parent* directory (e.g. /ephemeral). If models_dir\nis configured directly under /ephemeral that gives world-write access to\nall sibling directories (vLLM hug cache, Ollama models, etc.).\n\nNow chmod only the two directories that actually need it: models_dir\nand output_dir.
|
|
In prune_host_line, body.split(/\s+/) on a line with leading whitespace
produced tokens starting with an empty string, which was then shifted
into as ''. This caused the rewritten /etc/hosts entry to lose
its IP silently.
Fix by stripping the body before splitting: body.strip.split(/\s+/).
Refs: hc
|
|
In run_create_both, VM1's thread rescue unconditionally set
vm1_wg_state[:error], even when the WireGuard step had already
signaled success (vm1_wg_state[:done] = true). If VM2 was
waiting on the condition variable at that moment, it would raise
'VM1 WireGuard setup failed' and abort needlessly.
Now the rescue only sets :error when :done is still false, so a
downstream VM1 failure (e.g. vLLM install) no longer leaks to VM2.
Resolves agent task ic.
|
|
Replace Timeout.timeout(15) around Open3.capture3 with SSH-level
keepalive options (ServerAliveInterval=5, ServerAliveCountMax=3).
Ruby's Timeout raises in a background thread but leaves the ssh
process running; SSH's own timeouts self-terminate cleanly.
|
|
|
|
|
|
|
|
|
|
When all public IP probes fail (network down, DNS broken), detect_public_operator_cidr
raises HyperstackVM::Error. The old code did not cache this failure, so every
call to resolved_allowed_cidrs re-ran all probes, compounding slowness.
Add a rescue block in detected_operator_cidr that stores the exception in
@detected_operator_cidr_error and re-raises it. On subsequent calls the cached
error is re-raised immediately, preventing redundant probe retries.
|
|
|
|
Ensure Manager#delete does not wipe the state file on generic/transient API failures. The rescue now checks whether the error message indicates the VM is already gone (404, not_found, does not exist) before removing state. This prevents orphaned billable VMs after exhausted retries or transient network errors.
|
|
enhance_one
Wrap enhance_one body in begin/ensure to unconditionally delete upload_path
(and tmp_png) on every exit path, not just success. Prevents 50+ MB RAW->TIFF
leaks when upload_image/submit_prompt/wait_for_output/save_with_corrections
raises or when ComfyUI connection errors are caught.
|
|
calls
|
|
Escape regex metacharacters in WG_HOSTNAME before embedding into the sed
delete pattern so '.' (always present in hostnames like hyperstack1.wg1)
is treated as a literal dot rather than a wildcard.
Replace the literal-space anchor with [[:space:]] so tab-indented lines
in /etc/hosts are also matched and removed correctly.
|
|
|
|
|
|
|
|
DO_CLI_REF and resolveDoExecutable use ~/go/bin/ask; matchDoInvocation
still accepts legacy do prefixes. Update README and Nemotron hints.
Made-with: Cursor
|
|
|
|
Use DO_CLI_REF and resolveDoExecutable in agent-plan-mode; accept both
do and ~/go/bin/do in bash guards. Ask-mode shares matchDoInvocation.
Nemotron/Qwen tool discipline points to ~/go/bin/do done.
Made-with: Cursor
|
|
Move the former hyperstack-vm1-coder.toml to hyperstack-vm1.toml as the
standard VM1 profile (Qwen3-Coder-Next on single GPU). Preserve the
dual-H100 Nemotron-3-Super stack as hyperstack-vm1-nemotron.toml. Point
create-both at hyperstack-vm1.toml and refresh README for current defaults.
Made-with: Cursor
|
|
Rename task-wrapper invocations and prompts from ask to do in
agent-plan-mode (exec, bash guards, workflow strings), plan-mode README,
ask-mode readonly-command detection, and nemotron-tool-repair discipline
text. Internal helpers renamed for consistency (runDo, isSafeDoCommand).
Made-with: Cursor
|
|
Adds docker_image and pre_start_cmd config fields to config.rb and
provisioning.rb so the Gemma 4 31B workarounds are baked in:
- docker_image = "vllm/vllm-openai:nightly" (stable lacks Gemma 4 support)
- pre_start_cmd = "pip install -q transformers==5.5.0" (stable pins <5)
- extra_docker_env = ["CUDA_VISIBLE_DEVICES=0"] (required with --entrypoint bash)
When pre_start_cmd is set, the provisioner switches to --entrypoint bash and
chains the patch command before launching vLLM, so create-both works end-to-end
without manual container replacement.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
VM1 (hyperstack-vm1-coder.toml, renamed from hyperstack-vm1-gptoss.toml):
- Default model switched from gpt-oss-120b to qwen3-coder-next
- Config file renamed to reflect actual default model
VM2 (hyperstack-vm2.toml):
- Default model switched from qwen3-coder-next to Gemma 4 31B AWQ
- Uses vLLM nightly image + transformers==5.5.0 workaround: Gemma 4
architecture is registered in transformers 5.x but vLLM stable pins <5
- max_model_len=131072 (128K context); KV cache fills ~95% of H100-80GB VRAM
- Added gemma4-31b preset
watcher.rb:
- Add loading_status field to VmSnapshot to show live model-load progress
(last relevant log line during startup instead of generic "loading" message)
- fetch_vm_stats now captures both Engine 0 stats and loading-phase log lines
in a single SSH call using a shell variable to avoid two docker log invocations
- clean_log_line() strips vLLM PID/timestamp prefix for readable display
cli.rb: update all hardcoded hyperstack-vm1-gptoss.toml references to
hyperstack-vm1-coder.toml
hypr.fish: replace pi-hyperstack-nemotron with pi-hyperstack-coder (VM1),
add pi-hyperstack-gemma4 (VM2)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
|
|
|
|
Was incorrectly pointing at openai/gpt-oss-120b instead of the actual
Nemotron model ID.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
Qwen Coder models are trained on agent frameworks that include a "submit"
tool as a task-completion signal, causing "Tool submit not found" errors.
Adds QWEN_TOOL_DISCIPLINE injected unconditionally into the system prompt
for all Qwen Coder models, redirecting them to respond directly or use
`ask done uuid:<uuid>` for tracked tasks. Also adds the same submit hint
to NEMOTRON_TOOL_DISCIPLINE for consistency.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
Model weights occupy ~73.6 GiB leaving ~5.6 GiB for KV cache. Reduce
max_model_len to 32768 and raise gpu_memory_utilization to 0.98 to fit.
Add --enforce-eager to disable CUDA graph capture, which profiling-phase
requires ~2 GiB headroom that simply isn't available on a single A100.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
- /loop pause: suspends all active loops by clearing their timers and
marking each job paused; jobs are kept so they can be resumed
- /loop cont: resumes all paused loops, resetting nextRunAt to a fresh
interval from now and rescheduling timers
- New jobs created while paused inherit the paused state so they don't
fire immediately on creation
- drainPendingJobs and handleJobDue guard against firing while paused
- UI: status bar shows "loop:N ⏸" when paused; widget header shows
"(paused)"; paused jobs show ⏸, pending jobs ⏳, active jobs ⟳
- Autocomplete: pause and cont added to top-level fixed completions
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
Add watch_config_loaders that filters status_config_loaders results with
a TCP probe on each VM's WireGuard inference port. VMs with stale state
files (deleted from the console without `hyperstack.rb delete`) are
excluded from the watch loop. Falls back to all tracked loaders when
none are reachable so the watcher can still render error output when
WireGuard is down.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
end-to-end test on create
- cli: introduce REPO_ROOT constant so create-both/delete-both/watch
find TOML configs at the repo root instead of lib/hyperstack/
- manager: with_polling prints a heartbeat every 30s so silent waits
(SSH, VM ready, etc.) are visibly alive
- provisioning: bootstrap_guest streams SSH output in real time so
apt-lock waits and setup steps are visible as they happen
- provisioning: vLLM wait loop reads docker logs to show the current
startup stage (shard loading %, torch.compile, CUDA graphs, API up)
instead of a plain "not ready yet" counter
- manager: create automatically runs the end-to-end inference test
after provisioning completes, removing the manual 'test' step
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
command
Root cause: pi's autocomplete.js returns null from getSuggestions() when
getArgumentCompletions() returns null or []. The TUI's outer fallback then
shows filesystem completions. Every branch in getArgumentCompletions now
always returns at least one item:
- cancel branch: falls back to "cancel all" when no jobs exist
- preset branch: falls back to "edit" hint when no presets are loaded
- top-level: falls back to full list instead of null on no match
Also removes /loop-preset (hyphen) command — /loop preset <name> is the
single intended interface.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
~/.pi symlinks to pi/ not pi/agent/, so the path was one level off and
loadPresets() always returned []. Fix the path to include the agent/ segment.
Also add /loop-preset <name> as a dedicated command — its getArgumentCompletions
receives just the preset name as the first argument, which is reliable, avoiding
the multi-word prefix matching issues with /loop's third-level autocomplete.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
Pi bundles extensions so import.meta.url resolves to the wrong location,
causing loadPresets() to always return [] and autocomplete to fall back
to the filesystem. Switch to homedir()/.pi/extensions/loop-scheduler/
which reliably resolves to the correct path.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
Previously typing "/loop preset" returned an exact-match stub which caused
the UI to fall back to filesystem completions. Now cancel/preset expand
directly to their full third-level items ("cancel all"/"cancel <id>",
"preset <name>") as soon as the prefix matches the verb.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
- /loop cancel <partial> now suggests "all" and active job IDs
- /loop preset <name> new explicit subcommand with third-level name autocomplete
- Second-level completions now also include "preset" as a discoverable subcommand
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
autocomplete
- New loop-presets.md file (alongside index.ts) with format: * name: INTERVAL prompt
- /loop edit opens the presets file in $VISUAL/$EDITOR, using TUI stop/restart pattern
- /loop presets lists all loaded presets with intervals and prompt previews
- /loop <name> activates a named preset (single-word args only, no collision with interval syntax)
- getArgumentCompletions provides tab-completion for subcommands and preset names
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
Documents operational gotchas encountered during VM lifecycle:
- A100 first / H100 fallback procedure (manual for now)
- Docker image pull EOF failures and idempotent resume via create
- WireGuard 'already exists' conflict between setup script and systemd
- WireGuard public key mismatch after VM recreate (live fix commands)
- vLLM startup phase timing and log signals
- State file inspection and Hyperstack firewall IP drift
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
ImageMagick's LibRaw delegate can decode all major RAW formats. ComfyUI's
LoadImage node cannot read RAW directly, so auto_orient_tempfile now decodes
RAW → 16-bit TIFF before upload (magick already does this in one pass with
-auto-orient).
RAW inputs always produce _e.jpg output — there is no enhanced RAW format.
EXIF copy (exiftool -TagsFromFile) reads metadata directly from the original
RAW file, so capture time, camera/lens info, and GPS are preserved.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
Superseded by the photo-enhance.rb pipeline with ComfyUI.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
- .gitignore: exclude VM state files (ephemeral, contain keys/IDs)
and untrack the three existing state files already in history
- photo-enhance-review.md: findings review that drove the pipeline redesign
- pi/plans/: gt language planning documents
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
Second round of user-feedback contrast reduction (still a bit high after -0.10):
landscape/golden_hour/overcast/street 1.10→1.05
indoor/beach/default 1.05→1.00 (portrait/night already at 1.00)
Retrospective fix applied to 26 existing _e.jpg photos via:
magick -function polynomial 0.9545,0.02275
(equivalent to undoing 1.10 and re-applying 1.05 around the 0.5 midpoint)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
User feedback: contrast was too high on enhanced photos (~1.20 felt heavy).
Reduced all PROFILES contrast values by 0.10 (floor 1.00):
portrait 1.10→1.00, landscape 1.20→1.10, night 1.05→1.00,
indoor 1.15→1.05, golden_hour 1.20→1.10, overcast 1.20→1.10,
beach 1.15→1.05, street 1.20→1.10, default 1.15→1.05
Retrospective fix applied to 25 existing _e.jpg photos via:
magick -function polynomial 0.9167,0.04167
(equivalent to undoing 1.20 and re-applying 1.10 around the 0.5 midpoint)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
auto-orient bakes rotation into pixels before uploading to ComfyUI, but
copy_exif was then restoring the original Orientation tag (e.g. 8=rotate90)
causing viewers to rotate an already-rotated image.
Fix: exclude --Orientation from the exiftool TagsFromFile copy, then
explicitly set Orientation=1 so all viewers agree no further rotation needed.
Also fixed the 20 already-enhanced photos with: exiftool -Orientation=1 -n
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
When ComfyUI crashes (OOM on large ESRGAN tensors), systemd restarts it in ~30s.
Previously the script immediately failed all remaining photos in the batch.
Now: if upload/submit fails with 'Cannot reach ComfyUI', wait_for_recovery polls
every 10s up to 5 minutes for the service to come back, then retries the photo
once. Each photo gets at most one recovery-wait per run, so a permanent outage
still terminates rather than looping forever.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
Extracts all classes from hyperstack.rb into focused library files:
- lib/hyperstack/config.rb — ConfigLoader + Config (TOML loading, validation)
- lib/hyperstack/state.rb — StateStore + PrefixedOutput (JSON state, threaded output)
- lib/hyperstack/client.rb — HyperstackClient (REST API + retry logic)
- lib/hyperstack/wireguard.rb — LocalWireGuard (wg1.conf peer management, /etc/hosts)
- lib/hyperstack/provisioning.rb — ProvisioningScripts + RemoteProvisioner (SSH bootstrap)
- lib/hyperstack/manager.rb — Manager (VM lifecycle orchestration)
- lib/hyperstack/watcher.rb — VllmWatcher (Prometheus + GPU dashboard)
- lib/hyperstack/cli.rb — CLI (OptionParser command dispatch)
hyperstack.rb becomes a 46-line entry point with require_relative calls.
All files pass `ruby -c` syntax check and `--help` runs correctly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
Two runtime fixes discovered during testing:
1. CodeFormerRestore stub: CodeFormer custom node (comfyui-reactor-node)
is not installed on fresh VMs. Added a passthrough stub in smart_photo_node.py
that keeps the workflow intact while allowing the other stages to work.
Registered last so the real node from a separately installed package
takes priority if present.
2. Guided filter fallback: cv2.ximgproc.guidedFilter requires opencv-contrib
(opencv-contrib-python-headless). Added a try/except in _apply_detail that
falls back to cv2.bilateralFilter when ximgproc is unavailable. The contrib
package was also installed on the VM for full quality.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
realesr-general-x4v3.pth was referenced but only RealESRGAN_x4plus.pth is
pre-installed via the TOML config. Use the available model to avoid workflow
errors on first run. Both are 4x ESRGAN models; x4plus is the standard photo
quality option.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
grading
Addresses all five priority improvements from photo-enhance-review.md:
1. Conditional ESRGAN (ConditionalESRGANBlend node): portrait and night
scenes skip Real-ESRGAN entirely; indoor/golden_hour use a reduced
blend (0.25–0.60); landscape/beach keep full strength. Confidence-
weighted ratios from CLIP scores soften hard scene-boundary transitions.
2. Blended CLIP scene profiles: CLIPSceneDetect now outputs top-3 scene
scores as a normalised JSON string. AdaptivePhotoGrade blends the
matching grading profiles weighted by confidence instead of hard-
switching on the argmax label — a 55/35/10 landscape/golden_hour/overcast
split produces a weighted average of those three profiles.
3. Conservative depth defaults: DepthSelectiveSharpen background_blur
changed from 0.50 to 0.0 (disabled by default). Foreground sharpening
is kept; background blur can be re-enabled explicitly per shot.
4. EXIF/ICC preservation: copy_exif() in photo-enhance.rb uses exiftool
to restore capture time, camera/lens info, GPS, and ICC profile from
the source file onto the enhanced JPEG after ComfyUI strips metadata.
Embedded thumbnail/preview are excluded to avoid showing the old image.
5. Provenance logging: prompt_id is now passed into the metadata markdown
report. WritePhotoMetadata emits scene_scores and esrgan_mode into the
JSON sidecar. The markdown report now shows top-3 scene confidences,
ESRGAN mode, and ComfyUI prompt ID for reproducibility.
Workflow restructured: CLIPSceneDetect now runs on the original image
(before ESRGAN) so scene scores are available to gate the upscaler.
ConditionalESRGANBlend (node 16) inserted between ESRGAN downscale and
CodeFormer. All downstream nodes receive scene_scores via node 11 outputs.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
