diff options
| author | Paul Buetow <paul@buetow.org> | 2025-06-26 18:12:00 +0300 |
|---|---|---|
| committer | Paul Buetow <paul@buetow.org> | 2025-06-26 18:12:00 +0300 |
| commit | ee5250441a2d241dc1a980dfd051a12f2db898cf (patch) | |
| tree | e9da32e65c3fe4f3664c63c9042b86ee7a043625 /docs | |
| parent | 73ca612de9289a7362993099e3de720dbbf21519 (diff) | |
docs: document remaining serverless mode large file issues
Add documentation about the current limitations of serverless mode
with files larger than 10KB. While the immediate profiling deadlock
is resolved for small files, larger files still experience timeouts.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/SERVERLESS_LARGE_FILES_ISSUE.md | 61 |
1 files changed, 61 insertions, 0 deletions
diff --git a/docs/SERVERLESS_LARGE_FILES_ISSUE.md b/docs/SERVERLESS_LARGE_FILES_ISSUE.md new file mode 100644 index 0000000..3ff4b5f --- /dev/null +++ b/docs/SERVERLESS_LARGE_FILES_ISSUE.md @@ -0,0 +1,61 @@ +# Serverless Mode Large File Issue + +## Summary +While the serverless mode deadlock has been partially resolved, files larger than approximately 10KB still experience timeouts in serverless mode. + +## Current Status +- ✅ Files up to 10KB work correctly +- ❌ Files larger than 100KB timeout +- ❌ The 72MB test_data.log used in profiling examples still hangs + +## Technical Details +The current fix uses a channel-based approach to prevent deadlocks: +- Separate goroutines for reading from client/server handlers +- Buffered channels (100 slots) for data transfer +- 32KB buffer size for read operations + +However, this approach still has limitations with larger files, possibly due to: +1. Channel buffer exhaustion +2. Synchronization issues between read/write operations +3. EOF handling complexities +4. Memory pressure from buffering large amounts of data + +## Workaround +For profiling large files, avoid serverless mode by specifying a dummy server: +```bash +./dcat -profile -profiledir profiles -plain -cfg none -servers dummy test_data.log +``` + +## Proposed Solutions + +### Short-term +1. Increase channel buffer sizes dynamically based on file size +2. Implement backpressure handling +3. Add proper flow control between readers and writers + +### Long-term +1. Redesign serverless mode to avoid bidirectional copying +2. Implement a proper streaming architecture +3. Consider using io.Pipe with proper goroutine management +4. Add file size detection and automatic mode switching + +## Testing +Use the test_serverless.go script to verify fixes: +```go +// Test different file sizes +sizes := []struct { + name string + size int +}{ + {"tiny", 100}, // ✅ Works + {"small", 1024}, // ✅ Works + {"medium", 10240}, // ✅ Works + {"large", 102400}, // ❌ Timeouts + {"xlarge", 1048576}, // ❌ Timeouts +} +``` + +## Impact +- Profiling benchmarks work for small to medium test files +- Large file profiling requires non-serverless mode +- Integration tests may need adjustment if they use large files in serverless mode
\ No newline at end of file |
