summaryrefslogtreecommitdiff
path: root/docs
diff options
context:
space:
mode:
authorPaul Buetow <paul@buetow.org>2025-06-26 17:53:23 +0300
committerPaul Buetow <paul@buetow.org>2025-06-26 17:53:23 +0300
commit0688866f81266a75c30411089cabb3896f4068bd (patch)
tree6877ea50edefd09e4cb396c33bf60112c1487bf5 /docs
parent07bcd62caff1f03602331e5e10ca2dae888dcf18 (diff)
docs: add comprehensive plan to fix serverless mode deadlock
Document multiple solution approaches for fixing the io.Copy deadlock in serverless mode, with a recommended buffered pipe implementation. Includes detailed implementation phases and success criteria. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
Diffstat (limited to 'docs')
-rw-r--r--docs/SERVERLESS_FIX_PLAN.md138
1 files changed, 138 insertions, 0 deletions
diff --git a/docs/SERVERLESS_FIX_PLAN.md b/docs/SERVERLESS_FIX_PLAN.md
new file mode 100644
index 0000000..1e28633
--- /dev/null
+++ b/docs/SERVERLESS_FIX_PLAN.md
@@ -0,0 +1,138 @@
+# Serverless Mode Deadlock Fix Plan
+
+## Problem Summary
+The serverless connector uses bidirectional `io.Copy` operations that deadlock when processing large files. This happens because:
+1. `io.Copy(serverHandler, s.handler)` reads from client, writes to server
+2. `io.Copy(s.handler, serverHandler)` reads from server, writes to client
+3. When both buffers fill up, neither can proceed, causing a deadlock
+
+## Proposed Solutions
+
+### Solution 1: Buffered Pipe with Flow Control (Recommended)
+Replace direct `io.Copy` with a buffered pipe implementation that handles backpressure properly.
+
+**Implementation steps:**
+1. Create a buffered pipe implementation with configurable buffer size
+2. Implement flow control to prevent buffer overflow
+3. Use channels for coordination between read/write operations
+4. Add timeout mechanisms to detect and break deadlocks
+
+**Pros:**
+- Maintains bidirectional communication
+- Handles backpressure gracefully
+- Can be tuned for performance
+
+**Cons:**
+- More complex implementation
+- Requires careful testing
+
+### Solution 2: Sequential Processing
+Instead of concurrent bidirectional copying, process data sequentially.
+
+**Implementation steps:**
+1. Send all commands first
+2. Wait for responses
+3. Process responses one at a time
+4. Close connection when done
+
+**Pros:**
+- Simple implementation
+- No deadlock possible
+- Easy to debug
+
+**Cons:**
+- May impact performance for interactive operations
+- Changes the communication model
+
+### Solution 3: Channel-Based Communication
+Replace `io.Copy` with channel-based message passing.
+
+**Implementation steps:**
+1. Define message types for client-server communication
+2. Use buffered channels for message passing
+3. Implement proper channel closing semantics
+4. Add message framing for proper boundaries
+
+**Pros:**
+- Go-idiomatic solution
+- Clear message boundaries
+- Easy to add features like timeouts
+
+**Cons:**
+- Requires refactoring handler interfaces
+- May need protocol changes
+
+### Solution 4: Non-Blocking I/O
+Use non-blocking I/O operations with select statements.
+
+**Implementation steps:**
+1. Set handlers to non-blocking mode
+2. Use select with timeouts for read/write operations
+3. Implement proper EOF handling
+4. Add retry logic for partial reads/writes
+
+**Pros:**
+- Fine-grained control over I/O
+- Can detect and handle deadlocks
+
+**Cons:**
+- Complex error handling
+- Platform-specific considerations
+
+## Recommended Approach
+
+Start with **Solution 1 (Buffered Pipe)** as it:
+- Maintains the current architecture
+- Provides a drop-in replacement for `io.Copy`
+- Can be implemented incrementally
+- Allows for performance tuning
+
+## Implementation Plan
+
+### Phase 1: Create Test Case
+1. Write a test that reproduces the deadlock with a known file size
+2. Ensure test fails consistently with current implementation
+3. Add benchmarks to measure performance impact
+
+### Phase 2: Implement Buffered Pipe
+1. Create `internal/io/bufferedpipe` package
+2. Implement `BufferedPipe` type with:
+ - Configurable buffer size
+ - Flow control mechanisms
+ - Timeout support
+3. Add comprehensive unit tests
+
+### Phase 3: Integrate into Serverless Connector
+1. Replace `io.Copy` calls with buffered pipe
+2. Add configuration for buffer sizes
+3. Implement graceful shutdown handling
+4. Add metrics/logging for debugging
+
+### Phase 4: Testing & Validation
+1. Verify deadlock is resolved
+2. Run performance benchmarks
+3. Test with various file sizes
+4. Ensure backward compatibility
+
+### Phase 5: Documentation & Rollout
+1. Update documentation
+2. Add configuration examples
+3. Create migration guide if needed
+4. Monitor for issues in production
+
+## Alternative Quick Fix
+
+As an immediate mitigation, we could:
+1. Detect serverless mode in profiling scenarios
+2. Automatically add a dummy server to avoid serverless mode
+3. Log a warning about the limitation
+
+This would unblock profiling work while the proper fix is implemented.
+
+## Success Criteria
+
+1. No deadlocks with files of any size in serverless mode
+2. Performance remains within 10% of current implementation
+3. All existing tests pass
+4. New tests verify the fix
+5. Clear documentation of the solution \ No newline at end of file