| Age | Commit message (Collapse) | Author |
|---|
|
- stdout logger: release mutex while waiting on pause resume so prompt
callbacks can log (fixes hang after trusting new hosts; known_hosts
was written but Resume never ran).
- known hosts callback: stop borrowing the SSH dial throttle channel
(could block or interact badly with parallel handshakes).
- host key path: use errors.Is(..., fs.ErrNotExist) for RootedPath.Stat
wrapped errors; stat errors now fail fast instead of mis-read.
- public key path: same ErrNotExist check for authorized_keys miss.
- Build: optional DTAIL_NO_ZSTD=yes / nozstd tag for CGO-free builds;
split zstd readers into tagged files.
- Docs/examples: firewalld note for port 2222, log prune timer+script,
SSHBindAddress note, dserver unit disabled-by-default comment;
firewalld helper script example.
- Regression test for stdout pause/mutex behavior.
Made-with: Cursor
|
|
integration test command
|
|
|
|
|
|
|
|
- Add comprehensive PGO module in internal/tools/pgo/
- Integrate PGO into dtail-tools command with full CLI support
- Add Makefile targets for PGO workflow:
- make pgo: Full PGO workflow
- make pgo-quick: Quick PGO with smaller datasets
- make pgo-generate: Generate profiles only
- make build-pgo: Build with existing profiles
- make install-pgo: Install optimized binaries
- Add convenience functions to data generator for PGO
- Document PGO workflow in CLAUDE.md
Performance improvements observed:
- DCat: 3.8-7.0% additional improvement over turbo mode
- DGrep: Up to 19% improvement for low hit rates
- DMap: Variable impact, up to 64% for min_max on large files
Benchmarks show total performance gains (pre-turbo → turbo+PGO):
- DCat: 14-21x faster
- DGrep: 9-15x faster
- DMap: 9-29% faster
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
|
|
This major refactoring replaces all bash-based profiling and benchmarking
scripts with a unified Go tool (dtail-tools) that provides:
- Better cross-platform compatibility
- Improved error handling and reliability
- Structured data generation for test files
- Consistent command-line interface
- Easier maintenance and extensibility
Key changes:
- Created dtail-tools command with profile and benchmark subcommands
- Implemented common utilities for data generation and file operations
- Updated Makefile to use the new Go-based tools
- Maintained backward compatibility with existing make targets
- Fixed ParseSize to handle single-letter suffixes (10M, 1G, etc.)
The new tool supports all previous functionality:
- profile-quick, profile-all, profile-dmap
- benchmark creation, comparison, and management
- Test data generation with multiple formats
- Profile analysis and listing
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
|
|
- Updated 'make clean' to also remove all .tmp and .prof files in the repo
- Fixed dmap profiling scripts to let dmap complete naturally instead of
killing it after a timeout (dmap terminates when input is fully processed)
- Removed the special run_profile_dmap function as it's no longer needed
- Updated all profiling scripts to reflect that dmap has a natural exit point
Thanks for the correction - dmap does indeed terminate after processing
all data from the source file, so the timeout/kill approach was unnecessary.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
|
|
- Moved profile_benchmarks.sh, profile_dmap.sh, and profile_quick.sh
to the profiling/ directory where they belong
- Updated Makefile targets to reference new locations
- Fixed profile_dmap.sh to remove outfile clauses since they're not
needed for profiling and were preventing proper execution
- Updated .gitignore to exclude generated files in profiling/
This better separates benchmarking (performance comparison) from
profiling (performance analysis).
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
|
|
Created a comprehensive profiling framework for dtail commands (dcat, dgrep, dmap)
to analyze CPU usage and memory allocations. The framework now prints all executed
commands to stdout for full transparency.
Key features:
- Integrated Go profiling (CPU, memory, allocations) into all three commands
- Created profile.sh bash script for analyzing pprof profiles
- Added multiple Makefile targets for different profiling scenarios
- Automated profiling scripts with command echoing
- Support for different data sizes (quick, normal, full)
- Special handling for dmap MapReduce format
All profiling commands are now echoed to stdout before execution, making it
easy to understand what the framework is doing and reproduce commands manually.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
|
|
- Create benchmark framework to measure performance of dcat, dgrep, and dmap
- Generate test files of 10MB, 100MB, and 1GB with configurable patterns
- Support benchmarking with gzip and zstd compressed files
- Implement tool-specific benchmarks:
* DCat: Simple reading, multiple files, compressed files
* DGrep: Pattern matching, regex complexity, context lines, inverted grep
* DMap: Aggregations, group by operations, complex queries, time intervals
- Track performance metrics: throughput (MB/sec), lines/sec, memory usage
- Save results in multiple formats: JSON, CSV, and Markdown reports
- Add Makefile targets: benchmark, benchmark-quick, benchmark-full
- Support environment variables for configuration (sizes, timeouts, etc.)
- Automatically clean up temporary .tmp files after benchmarks
The framework provides consistent performance testing across the DTail toolset
and enables tracking performance regressions between commits.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
|
|
- Tests will now stop on first failure with -failfast flag
- Speeds up test debugging when failures occur
- Applies to all test runs via make test
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
|
|
|
|
|
|
|
|
|
|
|
|
this is useful for running unit and integration tests in jenkins
|
|
|
|
|
|
|
|
|
|
|
|
|
|
file unless NONE is specified
|
|
logger package rewrite as dlog
|
|
|
|
|
|
|
|
|
|
less portable
|
|
|
|
|
|
|
|
|
|
|
|
dealing with log streams
|
|
useful for log file monitorig for example. breaking protocol change which allows to mapreduce aggreate messages containing the default field separator |. add of more unit tests. add logformat mapreduce query keyword. add set mapreduce clause support and support to evaluate built-in functions such as md5sum() and maskdigits().
|
|
|
|
|
|
|
|
|
