| Age | Commit message (Collapse) | Author |
|---|
|
integration test command
|
|
|
|
- Add comprehensive PGO documentation in doc/pgo_implementation.md
- Improve dserver profiling using HTTP pprof endpoint
- Handle empty profiles gracefully for I/O-bound operations
- Add concurrent client workloads for better server profiling
- Update .gitignore to exclude PGO-generated directories
- Document performance improvements: 3-39% depending on command
The PGO implementation now supports all dtail commands with
realistic workloads and proper handling of edge cases.
π€ Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
|
|
This major refactoring replaces all bash-based profiling and benchmarking
scripts with a unified Go tool (dtail-tools) that provides:
- Better cross-platform compatibility
- Improved error handling and reliability
- Structured data generation for test files
- Consistent command-line interface
- Easier maintenance and extensibility
Key changes:
- Created dtail-tools command with profile and benchmark subcommands
- Implemented common utilities for data generation and file operations
- Updated Makefile to use the new Go-based tools
- Maintained backward compatibility with existing make targets
- Fixed ParseSize to handle single-letter suffixes (10M, 1G, etc.)
The new tool supports all previous functionality:
- profile-quick, profile-all, profile-dmap
- benchmark creation, comparison, and management
- Test data generation with multiple formats
- Profile analysis and listing
π€ Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
|
|
- Moved profile_benchmarks.sh, profile_dmap.sh, and profile_quick.sh
to the profiling/ directory where they belong
- Updated Makefile targets to reference new locations
- Fixed profile_dmap.sh to remove outfile clauses since they're not
needed for profiling and were preventing proper execution
- Updated .gitignore to exclude generated files in profiling/
This better separates benchmarking (performance comparison) from
profiling (performance analysis).
π€ Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
|
|
Fixed multiple issues preventing dmap from being profiled correctly:
- Updated profile_dmap.sh to use DTail default log format
- Fixed MapReduce queries to use correct field syntax
- Reduced file sizes and run counts for faster profiling
- Added proper command echoing to all data generation steps
Optimizations:
- Reduced PROFILE_RUNS from 3 to 1
- Reduced test data sizes (1MB/10MB instead of 10MB/100MB/1GB)
- Commented out medium/large file tests for faster runs
- Reduced dmap test data from 1000/10000 to 100/1000 lines
The profiling framework now successfully profiles all three commands
(dcat, dgrep, dmap) with reasonable execution times.
π€ Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
|
|
Created a comprehensive profiling framework for dtail commands (dcat, dgrep, dmap)
to analyze CPU usage and memory allocations. The framework now prints all executed
commands to stdout for full transparency.
Key features:
- Integrated Go profiling (CPU, memory, allocations) into all three commands
- Created profile.sh bash script for analyzing pprof profiles
- Added multiple Makefile targets for different profiling scenarios
- Automated profiling scripts with command echoing
- Support for different data sizes (quick, normal, full)
- Special handling for dmap MapReduce format
All profiling commands are now echoed to stdout before execution, making it
easy to understand what the framework is doing and reproduce commands manually.
π€ Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
|
|
- Create benchmark framework to measure performance of dcat, dgrep, and dmap
- Generate test files of 10MB, 100MB, and 1GB with configurable patterns
- Support benchmarking with gzip and zstd compressed files
- Implement tool-specific benchmarks:
* DCat: Simple reading, multiple files, compressed files
* DGrep: Pattern matching, regex complexity, context lines, inverted grep
* DMap: Aggregations, group by operations, complex queries, time intervals
- Track performance metrics: throughput (MB/sec), lines/sec, memory usage
- Save results in multiple formats: JSON, CSV, and Markdown reports
- Add Makefile targets: benchmark, benchmark-quick, benchmark-full
- Support environment variables for configuration (sizes, timeouts, etc.)
- Automatically clean up temporary .tmp files after benchmarks
The framework provides consistent performance testing across the DTail toolset
and enables tracking performance regressions between commits.
π€ Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
|
|
|
|
- Add test logging infrastructure to track command execution and file comparisons
- Generate .log files for each test with command history and manual verification commands
- Ensure all temporary test files use .tmp suffix for consistency
- Clean up .tmp files before each test run (not after) for clean test starts
- Update .gitignore to exclude generated test artifacts (.log, .query files)
- Fix dserver test configurations to use .tmp suffix for output files
- Fix expected test outputs for dgrep context tests
This change improves test debugging and verification by providing detailed logs
of what each test does and allows manual verification of test results.
π€ Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
|
|
|
|
|
|
this is useful for running unit and integration tests in jenkins
|
|
|
|
logger package rewrite as dlog
|
|
|
|
|
|
|
|
|
