diff options
| author | Paul Buetow <paul@buetow.org> | 2025-06-29 21:55:36 +0300 |
|---|---|---|
| committer | Paul Buetow <paul@buetow.org> | 2025-06-29 21:55:36 +0300 |
| commit | a688faabdd2f1ddca1e28744eb1efe11a23f29d3 (patch) | |
| tree | 7edc0d361b22816e769bc2f2f1dde30b5048da45 /integrationtests/test_config.json | |
| parent | 2140ed9dcbd180cd5e810eaabd2f3c2fbce55a57 (diff) | |
fix: improve aggregate channel switching for MapReduce operations
- Add mutex protection to prevent race conditions in nextLine()
- Implement synchronous channel put-back in turbo mode when possible
- Add timeout mechanism to prevent goroutine leaks
- Increase NextLinesCh buffer size to 1000 for better concurrency handling
- Document known limitation with turbo mode and high-concurrency MapReduce
These changes ensure TestDMap3 passes consistently without turbo mode.
With turbo mode, extreme concurrency (100+ files) may still have issues
due to the fundamental mismatch between turbo mode's speed and the
aggregate's channel rotation design. Workarounds are documented.
Co-Authored-By: Claude <noreply@anthropic.com>
Diffstat (limited to 'integrationtests/test_config.json')
| -rw-r--r-- | integrationtests/test_config.json | 5 |
1 files changed, 5 insertions, 0 deletions
diff --git a/integrationtests/test_config.json b/integrationtests/test_config.json new file mode 100644 index 0000000..8844461 --- /dev/null +++ b/integrationtests/test_config.json @@ -0,0 +1,5 @@ +{ + "MaxConcurrentCats": 100, + "SSHBindAddress": "localhost" +} +EOF < /dev/null
\ No newline at end of file |