summaryrefslogtreecommitdiff
path: root/docs/headless-testing-framework-proposal.md
diff options
context:
space:
mode:
authorPaul Buetow <paul@buetow.org>2025-06-21 20:10:38 +0300
committerPaul Buetow <paul@buetow.org>2025-06-21 20:10:38 +0300
commit695adc1f6bfb0a0eeef4dd6c035475ea2826871f (patch)
tree945fc0552d4f7f1ef1f468f6030e9925970fa72b /docs/headless-testing-framework-proposal.md
parentd3b697218773eaa5a3dd368705184726dbc0fa38 (diff)
Complete GUI decoupling implementation for headless testing
- Implement MessageHandler pattern to decouple message sending from visualization - Add HeadlessLoader to load simulations without GUI components - Create HeadlessProtocolRunner for clean protocol test execution - Update VSInternalProcess to use MessageHandler for message routing - Add null checks in VSSimulator for headless mode compatibility - Update VSSimulatorVisualization paint() to check for headless mode - Remove obsolete test scripts and documentation - Update test-protocols.sh to remove GUI error suppression options - Consolidate testing documentation in docs/testing-guide.md All protocol tests now run cleanly in headless mode without GUI errors, enabling proper CI/CD integration and automated testing. πŸ€– Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
Diffstat (limited to 'docs/headless-testing-framework-proposal.md')
-rw-r--r--docs/headless-testing-framework-proposal.md656
1 files changed, 0 insertions, 656 deletions
diff --git a/docs/headless-testing-framework-proposal.md b/docs/headless-testing-framework-proposal.md
deleted file mode 100644
index 974453e..0000000
--- a/docs/headless-testing-framework-proposal.md
+++ /dev/null
@@ -1,656 +0,0 @@
-# Headless Protocol Testing Framework Proposal
-
-## Executive Summary
-
-This proposal outlines a comprehensive headless testing framework for DS-Sim that enables automated verification of distributed protocols by:
-1. Loading saved simulations without GUI dependencies
-2. Replaying simulations in a controlled environment
-3. Capturing and analyzing log outputs
-4. Verifying protocol behavior through log pattern matching
-
-## Architecture Overview
-
-### Core Components
-
-```
-β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
-β”‚ Headless Testing Framework β”‚
-β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
-β”‚ β”‚
-β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚
-β”‚ β”‚ Headless Runner │───▢│ Log Capturer │───▢│ Verifier β”‚ β”‚
-β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚
-β”‚ β”‚ β”‚
-β”‚ β–Ό β”‚
-β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚
-β”‚ β”‚ Simulation Mgr │───▢│ Protocol β”‚ β”‚
-β”‚ β”‚ (No Frame) β”‚ β”‚ Executor β”‚ β”‚
-β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚
-β”‚ β”‚
-β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
-```
-
-## Detailed Design
-
-### 1. HeadlessSimulationRunner
-
-The main entry point for running simulations without GUI:
-
-```java
-public class HeadlessSimulationRunner {
- private VSSimulator simulator;
- private VSSimulatorVisualization viz;
- private LogCapture logCapture;
- private boolean running;
-
- public SimulationResult runSimulation(String simulationFile, long maxTime) {
- // 1. Load simulation without frame
- // 2. Install log capture
- // 3. Run simulation steps
- // 4. Return captured logs and metrics
- }
-}
-```
-
-### 2. LogCapture System
-
-A custom logging implementation that intercepts all log messages:
-
-```java
-public class LogCapture extends VSLogging {
- private final List<LogEntry> capturedLogs;
- private final Map<Integer, List<LogEntry>> processlogs;
-
- @Override
- public synchronized void log(String message, long time) {
- LogEntry entry = new LogEntry(time, message, LogType.GLOBAL);
- capturedLogs.add(entry);
- notifyListeners(entry);
- }
-
- @Override
- public synchronized void log(VSInternalProcess process, String message) {
- LogEntry entry = new LogEntry(
- process.getTime(),
- message,
- LogType.PROCESS,
- process.getProcessNum()
- );
- capturedLogs.add(entry);
- processLogs.computeIfAbsent(process.getProcessNum(), k -> new ArrayList<>())
- .add(entry);
- }
-}
-```
-
-### 3. Protocol Verifier
-
-A flexible verification system using pattern matching and assertions:
-
-```java
-public class ProtocolVerifier {
- private final List<VerificationRule> rules;
-
- public VerificationResult verify(List<LogEntry> logs) {
- List<RuleResult> results = new ArrayList<>();
-
- for (VerificationRule rule : rules) {
- results.add(rule.verify(logs));
- }
-
- return new VerificationResult(results);
- }
-}
-
-public interface VerificationRule {
- RuleResult verify(List<LogEntry> logs);
-}
-```
-
-### 4. Test Definition Format
-
-Tests are defined using a fluent API or configuration files:
-
-```java
-@Test
-public void testRaftLeaderElection() {
- HeadlessTest test = HeadlessTest.builder()
- .withSimulation("saved-simulations/raft-working.dat")
- .runFor(5000) // milliseconds
- .expectLog().containing("CANDIDATE").atLeastOnce()
- .expectLog().matching("Elected as LEADER").exactly(1)
- .expectLog().containing("REQUEST_VOTE").atLeast(2)
- .expectSequence()
- .first("FOLLOWER")
- .then("CANDIDATE")
- .finally("LEADER")
- .withinTime(3000)
- .build();
-
- TestResult result = test.run();
- assertTrue(result.passed());
-}
-```
-
-## Implementation Classes
-
-### HeadlessSimulationRunner.java
-
-```java
-package testing;
-
-import simulator.*;
-import core.*;
-import prefs.*;
-import events.*;
-import serialize.VSSerialize;
-import java.lang.reflect.*;
-import java.util.*;
-import java.util.concurrent.*;
-
-public class HeadlessSimulationRunner {
- private final VSDefaultPrefs prefs;
- private VSSimulator simulator;
- private VSSimulatorVisualization viz;
- private LogCapture logCapture;
- private final ExecutorService executor;
-
- public HeadlessSimulationRunner() {
- this.prefs = new VSDefaultPrefs();
- this.prefs.fillWithDefaults();
- VSRegisteredEvents.init(prefs);
- this.executor = Executors.newSingleThreadExecutor();
- }
-
- public SimulationResult runSimulation(String simulationFile, long maxTime)
- throws Exception {
- // Load simulation without frame
- VSSerialize serialize = new VSSerialize();
- simulator = serialize.openSimulator(simulationFile, null);
-
- if (simulator == null) {
- throw new IllegalStateException("Failed to load simulation");
- }
-
- // Access visualization via reflection
- Field vizField = VSSimulator.class.getDeclaredField("simulatorVisualization");
- vizField.setAccessible(true);
- viz = (VSSimulatorVisualization) vizField.get(simulator);
-
- // Install log capture
- logCapture = new LogCapture();
- installLogCapture();
-
- // Run simulation
- Future<Void> runFuture = executor.submit(() -> {
- runSimulationSteps(maxTime);
- return null;
- });
-
- // Wait for completion or timeout
- try {
- runFuture.get(maxTime * 2, TimeUnit.MILLISECONDS);
- } catch (TimeoutException e) {
- runFuture.cancel(true);
- }
-
- return new SimulationResult(
- logCapture.getCapturedLogs(),
- logCapture.getProcessLogs(),
- getSimulationMetrics()
- );
- }
-
- private void runSimulationSteps(long maxTime) throws Exception {
- VSTaskManager taskManager = viz.getTaskManager();
- Field globalTimeField = VSSimulatorVisualization.class
- .getDeclaredField("globalTime");
- globalTimeField.setAccessible(true);
-
- Method runTasksMethod = VSTaskManager.class
- .getDeclaredMethod("runTasks", long.class);
- runTasksMethod.setAccessible(true);
-
- long startTime = globalTimeField.getLong(viz);
-
- while (globalTimeField.getLong(viz) - startTime < maxTime) {
- long currentTime = globalTimeField.getLong(viz);
- runTasksMethod.invoke(taskManager, currentTime);
-
- // Advance time
- globalTimeField.setLong(viz, currentTime + 1);
-
- // Sync process times
- for (int i = 0; i < viz.getNumProcesses(); i++) {
- viz.getProcess(i).syncTime(currentTime + 1);
- }
- }
- }
-
- private void installLogCapture() throws Exception {
- // Install on visualization
- Field logingField = VSSimulatorVisualization.class
- .getDeclaredField("loging");
- logingField.setAccessible(true);
- logingField.set(viz, logCapture);
-
- // Install on all processes
- for (int i = 0; i < viz.getNumProcesses(); i++) {
- VSInternalProcess process = viz.getProcess(i);
- Field processLogingField = VSAbstractProcess.class
- .getDeclaredField("loging");
- processLogingField.setAccessible(true);
- processLogingField.set(process, logCapture);
- }
- }
-
- private SimulationMetrics getSimulationMetrics() {
- return new SimulationMetrics(
- viz.getNumProcesses(),
- logCapture.getTotalLogCount(),
- logCapture.getProcessMessageCounts()
- );
- }
-
- public void shutdown() {
- executor.shutdown();
- }
-}
-```
-
-### LogCapture.java
-
-```java
-package testing;
-
-import simulator.VSLogging;
-import core.VSInternalProcess;
-import java.util.*;
-import java.util.concurrent.CopyOnWriteArrayList;
-
-public class LogCapture extends VSLogging {
- private final List<LogEntry> capturedLogs;
- private final Map<Integer, List<LogEntry>> processLogs;
- private final List<LogListener> listeners;
-
- public LogCapture() {
- super();
- this.capturedLogs = new CopyOnWriteArrayList<>();
- this.processLogs = new ConcurrentHashMap<>();
- this.listeners = new CopyOnWriteArrayList<>();
- }
-
- @Override
- public synchronized void log(String message) {
- // Call parent to maintain compatibility
- super.log(message);
-
- long time = simulatorVisualization != null ?
- simulatorVisualization.getTime() : 0;
-
- LogEntry entry = new LogEntry(time, message, LogType.GLOBAL, -1);
- capturedLogs.add(entry);
- notifyListeners(entry);
- }
-
- @Override
- public synchronized void log(String message, long time) {
- super.log(message, time);
-
- LogEntry entry = new LogEntry(time, message, LogType.GLOBAL, -1);
- capturedLogs.add(entry);
- notifyListeners(entry);
- }
-
- public synchronized void log(VSInternalProcess process, String message) {
- // Create formatted message for parent
- String formattedMessage = "Process " + process.getProcessNum() +
- ": " + message;
- super.log(formattedMessage, process.getTime());
-
- LogEntry entry = new LogEntry(
- process.getTime(),
- message,
- LogType.PROCESS,
- process.getProcessNum()
- );
-
- capturedLogs.add(entry);
- processLogs.computeIfAbsent(process.getProcessNum(),
- k -> new CopyOnWriteArrayList<>())
- .add(entry);
- notifyListeners(entry);
- }
-
- private void notifyListeners(LogEntry entry) {
- for (LogListener listener : listeners) {
- listener.onLogEntry(entry);
- }
- }
-
- public List<LogEntry> getCapturedLogs() {
- return new ArrayList<>(capturedLogs);
- }
-
- public Map<Integer, List<LogEntry>> getProcessLogs() {
- Map<Integer, List<LogEntry>> result = new HashMap<>();
- for (Map.Entry<Integer, List<LogEntry>> entry : processLogs.entrySet()) {
- result.put(entry.getKey(), new ArrayList<>(entry.getValue()));
- }
- return result;
- }
-
- public int getTotalLogCount() {
- return capturedLogs.size();
- }
-
- public Map<Integer, Integer> getProcessMessageCounts() {
- Map<Integer, Integer> counts = new HashMap<>();
- for (Map.Entry<Integer, List<LogEntry>> entry : processLogs.entrySet()) {
- counts.put(entry.getKey(), entry.getValue().size());
- }
- return counts;
- }
-
- public void addListener(LogListener listener) {
- listeners.add(listener);
- }
-
- public void removeListener(LogListener listener) {
- listeners.remove(listener);
- }
-}
-```
-
-### ProtocolVerifier.java
-
-```java
-package testing;
-
-import java.util.*;
-import java.util.regex.*;
-import java.util.function.Predicate;
-
-public class ProtocolVerifier {
- private final List<VerificationRule> rules;
-
- public ProtocolVerifier() {
- this.rules = new ArrayList<>();
- }
-
- public ProtocolVerifier withRule(VerificationRule rule) {
- rules.add(rule);
- return this;
- }
-
- public ProtocolVerifier expectLog(String pattern) {
- rules.add(new PatternRule(pattern, 1, Integer.MAX_VALUE));
- return this;
- }
-
- public ProtocolVerifier expectLogExactly(String pattern, int count) {
- rules.add(new PatternRule(pattern, count, count));
- return this;
- }
-
- public ProtocolVerifier expectLogAtLeast(String pattern, int minCount) {
- rules.add(new PatternRule(pattern, minCount, Integer.MAX_VALUE));
- return this;
- }
-
- public ProtocolVerifier expectSequence(String... patterns) {
- rules.add(new SequenceRule(Arrays.asList(patterns)));
- return this;
- }
-
- public ProtocolVerifier expectNoLog(String pattern) {
- rules.add(new PatternRule(pattern, 0, 0));
- return this;
- }
-
- public VerificationResult verify(List<LogEntry> logs) {
- List<RuleResult> results = new ArrayList<>();
-
- for (VerificationRule rule : rules) {
- results.add(rule.verify(logs));
- }
-
- return new VerificationResult(results);
- }
-
- // Rule implementations
-
- private static class PatternRule implements VerificationRule {
- private final Pattern pattern;
- private final int minCount;
- private final int maxCount;
- private final String description;
-
- public PatternRule(String pattern, int minCount, int maxCount) {
- this.pattern = Pattern.compile(pattern);
- this.minCount = minCount;
- this.maxCount = maxCount;
- this.description = String.format(
- "Pattern '%s' should appear %s times",
- pattern,
- minCount == maxCount ?
- String.valueOf(minCount) :
- minCount + "-" + (maxCount == Integer.MAX_VALUE ? "∞" : maxCount)
- );
- }
-
- @Override
- public RuleResult verify(List<LogEntry> logs) {
- int count = 0;
- List<LogEntry> matches = new ArrayList<>();
-
- for (LogEntry log : logs) {
- if (pattern.matcher(log.getMessage()).find()) {
- count++;
- matches.add(log);
- }
- }
-
- boolean passed = count >= minCount && count <= maxCount;
- String message = String.format(
- "%s (found %d occurrences)",
- description, count
- );
-
- return new RuleResult(passed, message, matches);
- }
- }
-
- private static class SequenceRule implements VerificationRule {
- private final List<Pattern> patterns;
- private final String description;
-
- public SequenceRule(List<String> patterns) {
- this.patterns = patterns.stream()
- .map(Pattern::compile)
- .toList();
- this.description = "Sequence: " + String.join(" β†’ ", patterns);
- }
-
- @Override
- public RuleResult verify(List<LogEntry> logs) {
- int patternIndex = 0;
- List<LogEntry> matches = new ArrayList<>();
-
- for (LogEntry log : logs) {
- if (patternIndex < patterns.size() &&
- patterns.get(patternIndex).matcher(log.getMessage()).find()) {
- matches.add(log);
- patternIndex++;
- }
- }
-
- boolean passed = patternIndex == patterns.size();
- String message = String.format(
- "%s (%d/%d patterns matched)",
- description, patternIndex, patterns.size()
- );
-
- return new RuleResult(passed, message, matches);
- }
- }
-}
-```
-
-### Test Example: RaftProtocolTest.java
-
-```java
-package testing.protocols;
-
-import testing.*;
-import org.junit.jupiter.api.*;
-import static org.junit.jupiter.api.Assertions.*;
-
-public class RaftProtocolTest {
- private HeadlessSimulationRunner runner;
-
- @BeforeEach
- public void setup() {
- runner = new HeadlessSimulationRunner();
- }
-
- @AfterEach
- public void teardown() {
- runner.shutdown();
- }
-
- @Test
- @DisplayName("Test Raft leader election completes within timeout")
- public void testLeaderElection() throws Exception {
- // Run simulation
- SimulationResult result = runner.runSimulation(
- "saved-simulations/raft-working.dat",
- 3000 // 3 seconds
- );
-
- // Verify leader election
- ProtocolVerifier verifier = new ProtocolVerifier()
- .expectLog("Starting election")
- .expectLog("Elected as LEADER").expectLogExactly(1)
- .expectLog("REQUEST_VOTE").expectLogAtLeast(2)
- .expectSequence("FOLLOWER", "CANDIDATE", "LEADER");
-
- VerificationResult verification = verifier.verify(result.getAllLogs());
-
- assertTrue(verification.passed(), verification.getFailureMessage());
-
- // Additional assertions
- assertTrue(result.getMetrics().getTotalLogCount() > 10,
- "Should have substantial log activity");
- }
-
- @Test
- @DisplayName("Test Raft handles server crashes correctly")
- public void testCrashRecovery() throws Exception {
- SimulationResult result = runner.runSimulation(
- "saved-simulations/raft-working.dat",
- 6000 // Include crash/recovery events
- );
-
- ProtocolVerifier verifier = new ProtocolVerifier()
- .expectLog("Process.*crashed")
- .expectLog("Process.*recovered")
- .expectLog("Starting new election.*timeout");
-
- VerificationResult verification = verifier.verify(result.getAllLogs());
- assertTrue(verification.passed(), verification.getFailureMessage());
- }
-
- @Test
- @DisplayName("Test Raft client requests are handled")
- public void testClientRequests() throws Exception {
- SimulationResult result = runner.runSimulation(
- "saved-simulations/raft-working.dat",
- 2000
- );
-
- // Check client activation and requests
- ProtocolVerifier verifier = new ProtocolVerifier()
- .expectLog("Client.*activated")
- .expectLog("CLIENT_REQUEST|Sending request");
-
- VerificationResult verification = verifier.verify(result.getAllLogs());
-
- // May not have client requests if no leader elected yet
- if (!verification.passed()) {
- System.out.println("Note: " + verification.getFailureMessage());
- }
- }
-}
-```
-
-## Usage Examples
-
-### 1. Simple Test Case
-
-```java
-@Test
-public void testBasicProtocol() throws Exception {
- HeadlessTest.runAndVerify("simulation.dat")
- .expectLog("Protocol started")
- .expectLog("Message sent")
- .expectLog("Message received");
-}
-```
-
-### 2. Performance Test
-
-```java
-@Test
-public void testProtocolPerformance() throws Exception {
- SimulationResult result = runner.runSimulation("perf-test.dat", 10000);
-
- // Verify throughput
- int messageCount = result.countLogs("Message processed");
- double throughput = messageCount / 10.0; // messages per second
-
- assertTrue(throughput > 100, "Throughput too low: " + throughput);
-}
-```
-
-### 3. Regression Test
-
-```java
-@Test
-public void testKnownScenario() throws Exception {
- SimulationResult result = runner.runSimulation("regression-test.dat", 5000);
-
- // Compare with golden output
- List<String> golden = Files.readAllLines(Paths.get("golden-output.txt"));
- List<String> actual = result.getAllLogs().stream()
- .map(LogEntry::getMessage)
- .collect(Collectors.toList());
-
- assertEquals(golden, actual, "Output differs from golden file");
-}
-```
-
-## Benefits
-
-1. **Automated Testing**: No manual GUI interaction required
-2. **Reproducible**: Same simulation produces same results
-3. **Fast**: No GUI overhead, can run many tests quickly
-4. **CI/CD Ready**: Can be integrated into build pipelines
-5. **Comprehensive**: Can verify complex protocol behaviors
-6. **Debugging**: Failed tests provide detailed log traces
-
-## Implementation Timeline
-
-1. **Phase 1** (1-2 days): Core headless runner and log capture
-2. **Phase 2** (1-2 days): Verification framework and rules
-3. **Phase 3** (1 day): Test utilities and helpers
-4. **Phase 4** (1 day): Example tests for existing protocols
-5. **Phase 5** (1 day): Documentation and integration guides
-
-## Next Steps
-
-1. Review and approve the design
-2. Implement core components
-3. Create test suite for Raft protocol
-4. Extend to other protocols
-5. Integrate with Maven test lifecycle \ No newline at end of file