summaryrefslogtreecommitdiff
path: root/docs/SERVERLESS_LARGE_FILES_ISSUE.md
diff options
context:
space:
mode:
authorPaul Buetow <paul@buetow.org>2026-07-22 23:52:52 +0300
committerPaul Buetow <paul@buetow.org>2026-07-22 23:52:52 +0300
commit3004a7100e325c006971cc2e8d0f157338c0ce5c (patch)
treeb9d2be78433b2d6e13be6344357d1f81fa9ec44b /docs/SERVERLESS_LARGE_FILES_ISSUE.md
parent17bf7e042496a4afcbf6ee7a583378adb3ec502d (diff)
docs: DTail fork — documentation, agent guide, example configsHEADmaster
Squashed development of the documentation and example configuration: - AGENTS.md / CLAUDE.md: repository guide describing build/test/benchmark/PGO workflows and the single default read/output path (formerly "turbo"). - doc/ and docs/: query-language reference, log formats, auth-key fast reconnect, journal source reads, performance analyses (dated point-in-time records kept under historical-note disclaimers), and the turbo-vs-normal benchmark report with its result CSVs. - README.md updates; examples/ config + JSON schema aligned with the current Output* server tuning fields (the removed TurboBoost* keys dropped). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Diffstat (limited to 'docs/SERVERLESS_LARGE_FILES_ISSUE.md')
-rw-r--r--docs/SERVERLESS_LARGE_FILES_ISSUE.md61
1 files changed, 61 insertions, 0 deletions
diff --git a/docs/SERVERLESS_LARGE_FILES_ISSUE.md b/docs/SERVERLESS_LARGE_FILES_ISSUE.md
new file mode 100644
index 0000000..3ff4b5f
--- /dev/null
+++ b/docs/SERVERLESS_LARGE_FILES_ISSUE.md
@@ -0,0 +1,61 @@
+# Serverless Mode Large File Issue
+
+## Summary
+While the serverless mode deadlock has been partially resolved, files larger than approximately 10KB still experience timeouts in serverless mode.
+
+## Current Status
+- ✅ Files up to 10KB work correctly
+- ❌ Files larger than 100KB timeout
+- ❌ The 72MB test_data.log used in profiling examples still hangs
+
+## Technical Details
+The current fix uses a channel-based approach to prevent deadlocks:
+- Separate goroutines for reading from client/server handlers
+- Buffered channels (100 slots) for data transfer
+- 32KB buffer size for read operations
+
+However, this approach still has limitations with larger files, possibly due to:
+1. Channel buffer exhaustion
+2. Synchronization issues between read/write operations
+3. EOF handling complexities
+4. Memory pressure from buffering large amounts of data
+
+## Workaround
+For profiling large files, avoid serverless mode by specifying a dummy server:
+```bash
+./dcat -profile -profiledir profiles -plain -cfg none -servers dummy test_data.log
+```
+
+## Proposed Solutions
+
+### Short-term
+1. Increase channel buffer sizes dynamically based on file size
+2. Implement backpressure handling
+3. Add proper flow control between readers and writers
+
+### Long-term
+1. Redesign serverless mode to avoid bidirectional copying
+2. Implement a proper streaming architecture
+3. Consider using io.Pipe with proper goroutine management
+4. Add file size detection and automatic mode switching
+
+## Testing
+Use the test_serverless.go script to verify fixes:
+```go
+// Test different file sizes
+sizes := []struct {
+ name string
+ size int
+}{
+ {"tiny", 100}, // ✅ Works
+ {"small", 1024}, // ✅ Works
+ {"medium", 10240}, // ✅ Works
+ {"large", 102400}, // ❌ Timeouts
+ {"xlarge", 1048576}, // ❌ Timeouts
+}
+```
+
+## Impact
+- Profiling benchmarks work for small to medium test files
+- Large file profiling requires non-serverless mode
+- Integration tests may need adjustment if they use large files in serverless mode \ No newline at end of file