summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorPaul Buetow <paul@buetow.org>2025-06-26 17:52:21 +0300
committerPaul Buetow <paul@buetow.org>2025-06-26 17:52:21 +0300
commit07bcd62caff1f03602331e5e10ca2dae888dcf18 (patch)
tree2dff855f13ef40bde833f0037b0ed9e065c55b75
parentde796a4d70bc5891a6dba5e1050ae265cfddfe36 (diff)
docs: document serverless mode profiling hang issue
Add detailed analysis of the deadlock that occurs when profiling dtail commands in serverless mode with large files. The issue is caused by bidirectional io.Copy operations that deadlock when processing files larger than ~478KB. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
-rw-r--r--docs/PROFILING_HANG_ISSUE.md50
1 files changed, 50 insertions, 0 deletions
diff --git a/docs/PROFILING_HANG_ISSUE.md b/docs/PROFILING_HANG_ISSUE.md
new file mode 100644
index 0000000..637f40b
--- /dev/null
+++ b/docs/PROFILING_HANG_ISSUE.md
@@ -0,0 +1,50 @@
+# Profiling Hang Issue Analysis
+
+## Issue Description
+The dtail profiling suite hangs when processing large files in serverless mode. This occurs when running commands like `dcat`, `dgrep`, or `dmap` with `-cfg none` and no servers specified.
+
+## Root Cause
+When dtail operates in serverless mode (no servers specified), the `Serverless` connector creates bidirectional `io.Copy` operations between client and server handlers that deadlock on larger files.
+
+### Key Findings
+1. **File Size Threshold**: Small files work fine, but files over ~478KB cause hangs
+2. **Mode Specific**: The issue only occurs in serverless mode (when no servers are specified)
+3. **Deadlock Mechanism**: Two goroutines run `io.Copy` in opposite directions, creating a deadlock when buffers fill up
+4. **Profiling Impact**: The profiling example generates a 72MB test file, which triggers this issue
+
+### Code Location
+The problematic code is in `/home/paul/git/dtail/internal/clients/connectors/serverless.go:86-98`:
+
+```go
+go func() {
+ defer terminate()
+ if _, err := io.Copy(serverHandler, s.handler); err != nil {
+ dlog.Client.Trace(err)
+ }
+ dlog.Client.Trace("io.Copy(serverHandler, s.handler) => done")
+}()
+go func() {
+ defer terminate()
+ if _, err := io.Copy(s.handler, serverHandler); err != nil {
+ dlog.Client.Trace(err)
+ }
+ dlog.Client.Trace("io.Copy(s.handler, serverHandler) => done")
+}()
+```
+
+## Workaround
+Specify a dummy server to avoid serverless mode:
+```bash
+./dcat -profile -profiledir profiles -plain -cfg none -servers dummy test_data.log
+```
+
+## Symptoms
+- Command hangs indefinitely when processing large files
+- CPU profile files are created but remain at 0 KB
+- Multiple profile files are generated as the profiler attempts to write snapshots
+- Process must be killed with timeout or Ctrl+C
+
+## Impact
+- Profiling benchmarks fail to complete
+- Performance analysis of dtail tools is impaired
+- Integration tests may hang if they use serverless mode with large files \ No newline at end of file