summaryrefslogtreecommitdiff
path: root/internal/io
AgeCommit message (Collapse)Author
2026-04-10Fix known-hosts trust deadlock, host key stat, and optional nozstd buildPaul Buetow
- stdout logger: release mutex while waiting on pause resume so prompt callbacks can log (fixes hang after trusting new hosts; known_hosts was written but Resume never ran). - known hosts callback: stop borrowing the SSH dial throttle channel (could block or interact badly with parallel handshakes). - host key path: use errors.Is(..., fs.ErrNotExist) for RootedPath.Stat wrapped errors; stat errors now fail fast instead of mis-read. - public key path: same ErrNotExist check for authorized_keys miss. - Build: optional DTAIL_NO_ZSTD=yes / nozstd tag for CGO-free builds; split zstd readers into tagged files. - Docs/examples: firewalld note for port 2222, log prune timer+script, SSHBindAddress note, dserver unit disabled-by-default comment; firewalld helper script example. - Regression test for stdout pause/mutex behavior. Made-with: Cursor
2026-03-19task 260: harden ssh server key file access with OpenRootPaul Buetow
2026-03-19task 261: harden server reads with OpenRootPaul Buetow
2026-03-13Stop stale query work promptly on generation cancelPaul Buetow
2026-03-13task 682e6ae9: filter stale generation outputPaul Buetow
2026-03-10Refactor server-side config singleton readsPaul Buetow
2026-03-08task: add optimized file-reader processor tests (task 376)Paul Buetow
2026-03-08task: propagate dropped processing and flush errors (task 379)Paul Buetow
2026-03-08task: close compressed readers in file read paths (task 377)Paul Buetow
2026-03-08task: harden truncate signal lifecycle in processor readers (task 372)Paul Buetow
2026-03-03Apply go fix build tag modernizationPaul Buetow
2026-02-03Add SSH agent key selection and fix MapReduce outfile handlingPaul Buetow
This commit adds two major features and fixes: 1. SSH Agent Key Selection: - Add --agentKeyIndex flag to select specific SSH agent key (0-based) - Solves "too many authentication failures" with multiple SSH keys - Default -1 uses all keys (backwards compatible) - Available in dtail, dcat, dgrep, dmap commands 2. MapReduce Outfile Fixes: - CSV files now written at every interval, not just on exit - Proper signal handling (SIGTERM/SIGINT) with graceful shutdown - 5-second grace period for cleanup before force exit - Fixes issue where outfile remained as .tmp during execution Usage: dtail --servers host --agentKeyIndex 0 --query '...' outfile results.csv This is particularly useful with YubiKey/hardware tokens where many keys are loaded in the SSH agent, and for monitoring MapReduce results in real-time as they're computed. Co-authored-by: Cursor <cursoragent@cursor.com>
2026-01-29refactor: improve Go best practices compliancePaul Buetow
- Add explicit interface satisfaction checks (var _ Interface = (*Type)(nil)) for compile-time verification: - TurboWriter implementations (DirectTurboWriter, TurboChannelWriter) - Processor implementations (GrepLineProcessor, ChannellessLineProcessor) - Parser implementations (genericParser, csvParser, genericKVParser, custom parsers, mimecastParser) - Logger implementations (file, stdout) - Handler implementations (ServerHandler, ClientHandler) - Connector implementations (Serverless, ServerConnection) - SSH callback implementations (KnownHostsCallback) - Improve error handling with context wrapping (%w): - SSH operations: GeneratePrivateRSAKey, Agent - Query parsing: Query.parse - SSH client connections: dial, session, handle methods - Fix receiver consistency: - Convert Query.String() from value to pointer receiver - Convert Outfile.String() from value to pointer receiver - Convert all KnownHostsCallback methods to pointer receivers - Convert mapCommand.Start() to pointer receiver - Reorganize file structure for better clarity: - internal/io/dlog/dlog.go: Move type definition before public functions - internal/mapr/token.go: Reorganize helper functions after public ones - Add documentation comments: - Query.String() method - Outfile.String() method - Regex.String() method - Improve config variable documentation All unit tests and integration tests pass. Amp-Thread-ID: https://ampcode.com/threads/T-019c0b08-0eeb-705d-a1f7-31bb764b659a Co-authored-by: Amp <amp@ampcode.com>
2025-07-02perf: implement tiered buffer pooling to reduce allocationsPaul Buetow
- Add scanner_pool.go with tiered buffer pools (1MB, 64KB, 4KB) - Modify readWithProcessorOptimized to use pooled scanner buffers - Update tailWithProcessorOptimized to pool 64KB read buffers - Increase BytesBuffer pool initial capacity from 128B to 4KB - Add buffer_pool_test.go to benchmark pooling effectiveness This reduces memory allocations by ~36% in turbo mode by reusing buffers instead of allocating new ones for each file operation. All integration tests pass. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
2025-07-01perf: optimize turbo mode for 2.87x faster serverless performancePaul Buetow
Major performance improvements in turbo mode: - Fixed trace logging overhead by adding early level checks before expensive runtime.Caller() operations - Improved buffering strategy by removing forced immediate flush in serverless mode - Turbo mode now 2.87x faster (was 3-5x slower before optimization) Changes: - internal/io/dlog/dlog.go: Added early return in Trace() and Devel() when logging disabled - internal/server/handlers/turbo_writer.go: Removed serverless immediate flush condition Performance results: - Before: Turbo mode was 3-5x slower than non-turbo mode - After: Turbo mode is 2.87x faster (65% improvement) - All integration tests pass Added comprehensive benchmarking tools in benchmarks/ directory 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
2025-07-01fix: resolve turbo mode race condition and improve TestDCat2Paul Buetow
- Fixed race condition in periodicTruncateCheck by using context cancellation - Added turbo mode support to TestDCat2 server configuration - Removed problematic wait for pending files in readCommand.Start - Fixed potential panic when truncate channel is closed while goroutine is running The test now properly enables turbo mode on both client and server, preventing the timeout issues that occurred when only the client had turbo mode enabled. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
2025-06-30fix: ensure complete data transmission in turbo mode for dtail operationsPaul Buetow
This commit fixes integration test failures in turbo mode where data was not being fully transmitted before the connection closed. The main issue was that readWithTurboProcessor was returning too quickly without ensuring all data had been written to the network stream. Key changes: - Add comprehensive trace logging to track data flow in turbo mode - Fix turbo channel draining mechanism in baseHandler.Read() to wait for all data - Add proper flushing in TurboNetworkWriter with channel drain synchronization - Increase flush timeout from 10 to 100 iterations for turbo mode data volumes - Fix color formatting in serverless mode by processing lines individually - Add synchronization delays to ensure data transmission completes The fixes ensure that all data is properly transmitted before connection closure, resolving TestDcat integration test failures when DTAIL_TURBOBOOST_ENABLE is set. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
2025-06-29fix: respect MaxLineLength in turbo boost mode for integration testsPaul Buetow
The optimized line reader now properly handles the MaxLineLength configuration which is set to 1024 bytes in integration test mode. This ensures that long lines are split consistently between regular and turbo boost modes. - Cache MaxLineLength value to avoid repeated config lookups - Split lines that exceed MaxLineLength even when they contain newlines - Handle EOF cases properly when lines exceed the limit - Reset warning flag when normal lines are encountered All dcat and dgrep integration tests now pass with DTAIL_TURBOBOOST_ENABLE=yes 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
2025-06-28feat: implement channel-less grep for 62% performance improvementPaul Buetow
- Add LineProcessor interface for direct line processing without channels - Implement channel-less file reading in readfile_processor.go - Add optimized reader with 256KB buffering for efficient I/O - Create GrepLineProcessor for direct writing without intermediate channels - Fix serverless mode hanging due to stdin pipe detection - Fix base64 decoding bug (was counting characters instead of arguments) - Fix message output formatting by adding proper newline handling Performance improvements: - Channel-based: 9.00s → Channel-less: 3.42s (62% faster on 100MB files) - Removed channel synchronization overhead and context switching - Reduced memory allocations with buffer pooling Environment variables: - DTAIL_CHANNELLESS_GREP=yes - Enable channel-less implementation - DTAIL_OPTIMIZED_READER=yes - Use optimized buffered reader Known limitation: Inverted grep with context (--invert with --before/--after) not fully implemented in channel-less mode. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
2025-06-26fix: resolve package conflicts and update documentationPaul Buetow
- Move main package files to benchmarks/cmd/ to fix test failures - Update CLAUDE.md with comprehensive benchmarking and profiling instructions - Fix unused imports in serverless.go - Remove experimental buffered pipe/copy implementations - Remove outdated documentation files All integration tests now pass successfully. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
2025-06-26fix: resolve serverless mode deadlock for profilingPaul Buetow
Implement channel-based bidirectional copying in serverless connector to prevent deadlocks that occur with io.Copy when processing large files. Changes: - Replace direct io.Copy with channel-based approach in serverless.go - Add bufferedpipe and bufferedcopy utilities (for future use) - Add tests to verify deadlock prevention - Fix dmap profiling example to use absolute paths The fix successfully handles files up to ~10KB in serverless mode. Larger files still experience issues and will be addressed in a follow-up fix. Fixes profiling hang issue when using -cfg none without servers. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
2024-03-29lint warningsPaul Buetow
2022-07-15gofmt permission file headersPaul Buetow
2022-03-14a 0666 to OpenFile will respect the user's default umaskPaul Buetow
2022-02-15minor fix, add missing newlines to server messagesPaul Buetow
2021-12-21remove year from datestampPaul Buetow
2021-12-08Document exported line.New function.Paul Buetow
2021-12-06Refactor local context filter method to reduce its cognitive complexityPaul Buetow
2021-12-05buffer line.Line for performancePaul Buetow
2021-12-04refactorPaul Buetow
2021-12-04Refactor LContext handling to reduce cognitive method complexity.Paul Buetow
2021-12-04Refactor read method to reduce its Cognitive Complexity.Paul Buetow
2021-12-03satisfy sonarqubePaul Buetow
2021-12-03Use nested comments to satisfy SonarQubePaul Buetow
2021-12-03fix misuse of unbuffered channel for OS signal handlingPaul Buetow
2021-12-03comment why none logger documentation is emptyPaul Buetow
2021-11-03Add integration test for long line splitting - Also fixed a bug regarding ↵Paul Buetow
this along the way
2021-11-02Bugfix: Dealing correctly with files without newline characters, also add ↵Paul Buetow
more tests
2021-10-31add dmap integration test with stdin input pipePaul Buetow
2021-10-30add support to read from stdin pipe in serverless mode, e.g. grep foo.log | ↵Paul Buetow
dmap "select from ...."
2021-10-27Fix --shutdownAfter client switchPaul Buetow
2021-10-24Fix DCat color test.Paul Buetow
2021-10-24Fix deadlock around aggregating data + server max concurrent file read limiterPaul Buetow
2021-10-19lowercase log rotation comparisonPaul Buetow
2021-10-14add dgrep context integration testsPaul Buetow
2021-10-14Merging grep context from masterPaul Buetow
2021-10-12Update JSON-schema to reflect all recent config file changes.Paul Buetow
2021-10-10add another dmap test - reading 100 source files at oncePaul Buetow
fix a data race when reading multiple files on one server from the same session at once
2021-10-10vetting and linting and some code restylingPaul Buetow
2021-10-10add dtail health check unit test.Paul Buetow
generated by cgit v1.2.3 (git 2.25.1) at 2026-06-08 15:09:09 +0000