| Age | Commit message (Collapse) | Author |
|---|
|
|
|
- Add comprehensive PGO module in internal/tools/pgo/
- Integrate PGO into dtail-tools command with full CLI support
- Add Makefile targets for PGO workflow:
- make pgo: Full PGO workflow
- make pgo-quick: Quick PGO with smaller datasets
- make pgo-generate: Generate profiles only
- make build-pgo: Build with existing profiles
- make install-pgo: Install optimized binaries
- Add convenience functions to data generator for PGO
- Document PGO workflow in CLAUDE.md
Performance improvements observed:
- DCat: 3.8-7.0% additional improvement over turbo mode
- DGrep: Up to 19% improvement for low hit rates
- DMap: Variable impact, up to 64% for min_max on large files
Benchmarks show total performance gains (pre-turbo → turbo+PGO):
- DCat: 14-21x faster
- DGrep: 9-15x faster
- DMap: 9-29% faster
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
|
|
- Changed environment variable from DTAIL_TURBOBOOST_ENABLE to DTAIL_TURBOBOOST_DISABLE
- Changed config field from TurboModeEnable to TurboBoostDisable
- Turbo boost is now enabled by default and must be explicitly disabled
- Updated all code references, documentation, and examples
- No change in functionality, only inverted the boolean logic
This makes turbo boost opt-out rather than opt-in, providing better
default performance for large files while allowing users to disable
it for scenarios where it adds overhead.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
|
|
- Add SafeAggregateSet wrapper with mutex protection for concurrent access
- Implement TurboAggregate for direct line processing without channels
- Fix race conditions in turbo mode MapReduce aggregation
- Add proper synchronization for batch processing completion
- Update shutdown sequence to ensure all data is serialized
- Add integration test configuration for high-concurrency scenarios
The turbo mode now correctly handles MapReduce queries with significant
performance improvements while maintaining data integrity and preventing
race conditions during concurrent aggregation.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
|
|
- Add mutex protection to prevent race conditions in nextLine()
- Implement synchronous channel put-back in turbo mode when possible
- Add timeout mechanism to prevent goroutine leaks
- Increase NextLinesCh buffer size to 1000 for better concurrency handling
- Document known limitation with turbo mode and high-concurrency MapReduce
These changes ensure TestDMap3 passes consistently without turbo mode.
With turbo mode, extreme concurrency (100+ files) may still have issues
due to the fundamental mismatch between turbo mode's speed and the
aggregate's channel rotation design. Workarounds are documented.
Co-Authored-By: Claude <noreply@anthropic.com>
|
|
- Add LineProcessor interface for direct line processing without channels
- Implement channel-less file reading in readfile_processor.go
- Add optimized reader with 256KB buffering for efficient I/O
- Create GrepLineProcessor for direct writing without intermediate channels
- Fix serverless mode hanging due to stdin pipe detection
- Fix base64 decoding bug (was counting characters instead of arguments)
- Fix message output formatting by adding proper newline handling
Performance improvements:
- Channel-based: 9.00s → Channel-less: 3.42s (62% faster on 100MB files)
- Removed channel synchronization overhead and context switching
- Reduced memory allocations with buffer pooling
Environment variables:
- DTAIL_CHANNELLESS_GREP=yes - Enable channel-less implementation
- DTAIL_OPTIMIZED_READER=yes - Use optimized buffered reader
Known limitation: Inverted grep with context (--invert with --before/--after)
not fully implemented in channel-less mode.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
|
|
- Updated 'make clean' to also remove all .tmp and .prof files in the repo
- Fixed dmap profiling scripts to let dmap complete naturally instead of
killing it after a timeout (dmap terminates when input is fully processed)
- Removed the special run_profile_dmap function as it's no longer needed
- Updated all profiling scripts to reflect that dmap has a natural exit point
Thanks for the correction - dmap does indeed terminate after processing
all data from the source file, so the timeout/kill approach was unnecessary.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
|
|
- Move main package files to benchmarks/cmd/ to fix test failures
- Update CLAUDE.md with comprehensive benchmarking and profiling instructions
- Fix unused imports in serverless.go
- Remove experimental buffered pipe/copy implementations
- Remove outdated documentation files
All integration tests now pass successfully.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
|
|
This file provides guidance to Claude Code when working with the dtail codebase, including:
- Build and test commands
- Project architecture overview
- Code organization details
- Common development tasks
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
|
