summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
-rw-r--r--docs/SERVERLESS_LARGE_FILES_ISSUE.md61
1 files changed, 61 insertions, 0 deletions
diff --git a/docs/SERVERLESS_LARGE_FILES_ISSUE.md b/docs/SERVERLESS_LARGE_FILES_ISSUE.md
new file mode 100644
index 0000000..3ff4b5f
--- /dev/null
+++ b/docs/SERVERLESS_LARGE_FILES_ISSUE.md
@@ -0,0 +1,61 @@
+# Serverless Mode Large File Issue
+
+## Summary
+While the serverless mode deadlock has been partially resolved, files larger than approximately 10KB still experience timeouts in serverless mode.
+
+## Current Status
+- ✅ Files up to 10KB work correctly
+- ❌ Files larger than 100KB timeout
+- ❌ The 72MB test_data.log used in profiling examples still hangs
+
+## Technical Details
+The current fix uses a channel-based approach to prevent deadlocks:
+- Separate goroutines for reading from client/server handlers
+- Buffered channels (100 slots) for data transfer
+- 32KB buffer size for read operations
+
+However, this approach still has limitations with larger files, possibly due to:
+1. Channel buffer exhaustion
+2. Synchronization issues between read/write operations
+3. EOF handling complexities
+4. Memory pressure from buffering large amounts of data
+
+## Workaround
+For profiling large files, avoid serverless mode by specifying a dummy server:
+```bash
+./dcat -profile -profiledir profiles -plain -cfg none -servers dummy test_data.log
+```
+
+## Proposed Solutions
+
+### Short-term
+1. Increase channel buffer sizes dynamically based on file size
+2. Implement backpressure handling
+3. Add proper flow control between readers and writers
+
+### Long-term
+1. Redesign serverless mode to avoid bidirectional copying
+2. Implement a proper streaming architecture
+3. Consider using io.Pipe with proper goroutine management
+4. Add file size detection and automatic mode switching
+
+## Testing
+Use the test_serverless.go script to verify fixes:
+```go
+// Test different file sizes
+sizes := []struct {
+ name string
+ size int
+}{
+ {"tiny", 100}, // ✅ Works
+ {"small", 1024}, // ✅ Works
+ {"medium", 10240}, // ✅ Works
+ {"large", 102400}, // ❌ Timeouts
+ {"xlarge", 1048576}, // ❌ Timeouts
+}
+```
+
+## Impact
+- Profiling benchmarks work for small to medium test files
+- Large file profiling requires non-serverless mode
+- Integration tests may need adjustment if they use large files in serverless mode \ No newline at end of file