blob: 545dde828bc7c466c038e565d0f59593330244cf (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
|
# DTail Channelless Performance Benchmark Results
## Executive Summary
The new channelless architecture for DTail delivers dramatic performance improvements for grep operations, with 4-5x speedup across different scenarios while maintaining full compatibility with existing functionality.
## Test Environment
- **File**: benchmark_large.log (50MB, 698,333 lines)
- **Hardware**: 8 CPU cores, 31Gi RAM
- **Go version**: go1.24.3
- **Test date**: 2025-06-17
## Detailed Results
### 1. Basic ERROR Filtering
```
Pattern: "ERROR"
Expected matches: ~87,000 lines
OLD (channel-based):
Run 1: 0.515 seconds
Run 2: 0.524 seconds
Run 3: 0.526 seconds
Run 4: 0.530 seconds
Run 5: 0.544 seconds
Average: 0.528 seconds
NEW (channelless):
Run 1: 0.117 seconds
Run 2: 0.116 seconds
Run 3: 0.115 seconds
Run 4: 0.120 seconds
Run 5: 0.117 seconds
Average: 0.117 seconds
IMPROVEMENT: 4.5x faster (78% reduction)
```
### 2. ERROR Filtering with Context Lines
```
Pattern: "ERROR" --before 2 --after 2
Expected output: ~435,000 lines (with context)
OLD (channel-based):
Run 1: 1.238 seconds
Run 2: 1.217 seconds
Run 3: 1.185 seconds
Run 4: 1.293 seconds
Run 5: 1.189 seconds
Average: 1.224 seconds
NEW (channelless):
Run 1: 0.216 seconds
Run 2: 0.214 seconds
Run 3: 0.237 seconds
Run 4: 0.227 seconds
Run 5: 0.233 seconds
Average: 0.225 seconds
IMPROVEMENT: 5.4x faster (82% reduction)
```
### 3. Rare Pattern Filtering
```
Pattern: "connection_timeout"
Expected matches: ~87 lines
OLD (channel-based):
Run 1: 0.402 seconds
Run 2: 0.442 seconds
Run 3: 0.414 seconds
Run 4: 0.406 seconds
Run 5: 0.477 seconds
Average: 0.428 seconds
NEW (channelless):
Run 1: 0.106 seconds
Run 2: 0.099 seconds
Run 3: 0.103 seconds
Run 4: 0.104 seconds
Run 5: 0.105 seconds
Average: 0.103 seconds
IMPROVEMENT: 4.2x faster (76% reduction)
```
### 4. Full File Read (DCAT)
```
Operation: Read entire 50MB file
Expected output: 698,333 lines
OLD (channel-based):
Run 1: 1.547 seconds
Run 2: 1.523 seconds
Run 3: 1.583 seconds
Run 4: 1.550 seconds
Run 5: 1.599 seconds
Average: 1.560 seconds
NEW (channelless):
Run 1: 1.535 seconds
Run 2: 2.076 seconds
Run 3: 2.033 seconds
Run 4: 2.013 seconds
Run 5: 1.995 seconds
Average: 1.930 seconds
PERFORMANCE: 19% slower (expected due to network protocol overhead)
```
## Performance Analysis
### Key Insights
1. **DGREP shows dramatic 4-5x performance improvements** with channelless mode
2. **Context lines benefit even more** (5.4x faster) due to reduced coordination overhead
3. **Performance gains are consistent** across different pattern match rates
4. **DCAT shows slight slowdown** due to protocol formatting overhead when outputting all lines
5. **Channelless mode excels** when filtering/processing reduces output volume significantly
### Why Channelless is Faster
The channelless architecture eliminates:
- **Goroutine coordination overhead** between file readers and output writers
- **Channel communication latency** for each line processed
- **Memory allocation/deallocation** for channel message passing
- **Context switching** between concurrent goroutines
### Optimal Use Cases
Channelless mode provides the biggest benefits for operations that:
- **Filter/reduce data volume** (grep patterns, context lines)
- **Process large files** with selective output
- **Require high throughput** for log analysis workloads
### When NOT to Use Channelless
The following operations continue to use channel-based processing:
- **Tail operations** (require continuous monitoring and real-time streaming)
- **MapReduce operations** (require aggregation infrastructure)
- **Operations outputting full files** (minimal filtering benefit)
## Technical Implementation
The channelless implementation introduces:
- **DirectProcessor framework** with LineProcessor interface
- **NetworkOutputWriter** for direct network streaming
- **Command-specific processors** (Grep, Cat, Tail, Map)
- **Intelligent mode detection** to choose optimal processing method
## Conclusion
The channelless implementation successfully delivers significant performance improvements for DTail's core use cases while maintaining full compatibility with existing functionality. The 4-5x speedup for grep operations represents a substantial enhancement for log analysis workflows.
|
