diff options
| author | Paul Buetow <paul@buetow.org> | 2025-06-28 19:52:43 +0300 |
|---|---|---|
| committer | Paul Buetow <paul@buetow.org> | 2025-06-28 19:52:43 +0300 |
| commit | 0aa3222cef46d527bb9437afa9ddd90f3a80a9d8 (patch) | |
| tree | aa56290ef19fa1a14b3281f2be38b04cd129db48 /CHANNELLESS_GREP_IMPLEMENTATION.md | |
| parent | c75b6595f6cb0c94f4ecc05ca7c27ec0e83de368 (diff) | |
refactor: consolidate optimization flags into DTAIL_TURBOBOOST_ENABLE
- Replace DTAIL_CHANNELLESS_GREP and DTAIL_OPTIMIZED_READER with single flag
- Rename documentation to TURBOBOOST_OPTIMIZATION.md
- Fix channel-less adapter to use blocking sends (prevent data loss)
- Update logging messages to reference "turbo boost" mode
The DTAIL_TURBOBOOST_ENABLE variable now controls all performance
optimizations and can be extended to other commands in the future.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Diffstat (limited to 'CHANNELLESS_GREP_IMPLEMENTATION.md')
| -rw-r--r-- | CHANNELLESS_GREP_IMPLEMENTATION.md | 103 |
1 files changed, 0 insertions, 103 deletions
diff --git a/CHANNELLESS_GREP_IMPLEMENTATION.md b/CHANNELLESS_GREP_IMPLEMENTATION.md deleted file mode 100644 index af79d9c..0000000 --- a/CHANNELLESS_GREP_IMPLEMENTATION.md +++ /dev/null @@ -1,103 +0,0 @@ -# Channel-less dgrep Implementation - -## Overview - -This document describes the channel-less implementation of dgrep that was created to address performance bottlenecks caused by channel overhead in the original implementation. - -## Problem Statement - -The original dgrep implementation used multiple channels in a pipeline: -- `rawLines chan *bytes.Buffer` (buffer: 100) - Raw lines read from file -- `lines chan *line.Line` (buffer: 100) - Filtered lines to send to client - -This created several performance issues: -1. Fixed channel buffer sizes causing blocking under high throughput -2. Context switching overhead between goroutines -3. Channel synchronization overhead -4. Memory allocations for channel operations - -## Solution - -The channel-less implementation replaces the channel pipeline with direct function calls using a `LineProcessor` interface. - -### Key Components - -1. **LineProcessor Interface** (`internal/io/line/processor.go`) - - Defines methods for processing lines without channels - - `ProcessLine()` - Handle a single line - - `Flush()` - Ensure buffered data is written - - `Close()` - Clean up resources - -2. **GrepLineProcessor** (`internal/server/handlers/lineprocessor.go`) - - Implements LineProcessor for grep operations - - Writes directly to the network connection - - Uses internal buffering for efficiency (64KB buffer) - - Thread-safe with mutex protection - -3. **Modified File Reading** (`internal/io/fs/readfile_processor.go`) - - `StartWithProcessor()` - Channel-less file reading - - Direct callbacks instead of channel sends - - Inline regex filtering without goroutines - -4. **Optimized File Reading** (`internal/io/fs/readfile_processor_optimized.go`) - - Uses buffered line reading instead of byte-by-byte - - Custom scanner with 256KB buffer - - Efficient handling of long lines - - Special optimization for tail mode - -### Feature Flags - -The implementation can be controlled via environment variables: -- `DTAIL_CHANNELLESS_GREP=yes` - Enable channel-less grep implementation -- `DTAIL_OPTIMIZED_READER=yes` - Use optimized buffered reader - -### Benefits - -1. **Reduced Latency**: No channel queuing delays -2. **Lower Memory Usage**: No channel buffers -3. **Better CPU Efficiency**: Fewer context switches -4. **Simpler Code Flow**: Direct processing without goroutine coordination -5. **Predictable Performance**: No channel blocking - -### Backward Compatibility - -- Original channel-based implementation remains available -- Same command-line interface -- Protocol compatibility maintained -- All integration tests pass unchanged - -### Performance Testing - -Use the provided script to compare performance: - -```bash -./test_channelless_performance.sh -``` - -This will test: -1. Original channel-based implementation -2. Channel-less implementation -3. Optimized channel-less implementation - -### Usage - -To use the channel-less implementation: - -```bash -# Enable channel-less grep -export DTAIL_CHANNELLESS_GREP=yes - -# Also enable optimized reader -export DTAIL_OPTIMIZED_READER=yes - -# Run dgrep normally -dgrep -regex "pattern" file.log -``` - -### Future Improvements - -1. Extend channel-less approach to other commands (dcat, dtail) -2. Add configurable buffer sizes -3. Implement zero-copy optimizations -4. Add performance metrics collection -5. Consider using io_uring on Linux for async I/O
\ No newline at end of file |
