summaryrefslogtreecommitdiff
path: root/internal/io/line/processor.go
diff options
context:
space:
mode:
authorPaul Buetow <paul@buetow.org>2025-06-28 19:47:10 +0300
committerPaul Buetow <paul@buetow.org>2025-06-28 19:47:10 +0300
commitc75b6595f6cb0c94f4ecc05ca7c27ec0e83de368 (patch)
treeedc815d8e0e35eaad5fbfd201852b33cd074fc6d /internal/io/line/processor.go
parent408d6365383ecca294c3260df261f08092484aef (diff)
feat: implement channel-less grep for 62% performance improvement
- Add LineProcessor interface for direct line processing without channels - Implement channel-less file reading in readfile_processor.go - Add optimized reader with 256KB buffering for efficient I/O - Create GrepLineProcessor for direct writing without intermediate channels - Fix serverless mode hanging due to stdin pipe detection - Fix base64 decoding bug (was counting characters instead of arguments) - Fix message output formatting by adding proper newline handling Performance improvements: - Channel-based: 9.00s → Channel-less: 3.42s (62% faster on 100MB files) - Removed channel synchronization overhead and context switching - Reduced memory allocations with buffer pooling Environment variables: - DTAIL_CHANNELLESS_GREP=yes - Enable channel-less implementation - DTAIL_OPTIMIZED_READER=yes - Use optimized buffered reader Known limitation: Inverted grep with context (--invert with --before/--after) not fully implemented in channel-less mode. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
Diffstat (limited to 'internal/io/line/processor.go')
-rw-r--r--internal/io/line/processor.go22
1 files changed, 22 insertions, 0 deletions
diff --git a/internal/io/line/processor.go b/internal/io/line/processor.go
new file mode 100644
index 0000000..c7bec77
--- /dev/null
+++ b/internal/io/line/processor.go
@@ -0,0 +1,22 @@
+package line
+
+import (
+ "bytes"
+)
+
+// Processor defines an interface for processing lines read from files.
+// This interface replaces the channel-based approach for better performance.
+type Processor interface {
+ // ProcessLine handles a single line read from a file.
+ // The line buffer ownership is transferred to the processor.
+ // Returns error if processing should stop.
+ ProcessLine(line *bytes.Buffer, lineNum uint64, sourceID string) error
+
+ // Flush ensures any buffered data is written out.
+ // Called when file reading completes or on periodic intervals.
+ Flush() error
+
+ // Close cleans up any resources used by the processor.
+ // Called when processing is complete.
+ Close() error
+} \ No newline at end of file