We talked a little about measuring Soar's performance in today's meeting. I mentioned a simple utility called TestSoarPerformance, and several people couldn't find it. Now that I've looked back at the releases, I can see why. It no longer exists!
I think I've reconstructed that memory now and can explain why. If I recall correctly, TestSoarPerformance was limited, so I replaced it with a slightly more elaborate utility. (This happened a few years ago between 9.5 and the current version.)
Arguments:
These are the 24 agents that come with the test utility:
I think I've reconstructed that memory now and can explain why. If I recall correctly, TestSoarPerformance was limited, so I replaced it with a slightly more elaborate utility. (This happened a few years ago between 9.5 and the current version.)
- The executable is called "PerformanceTests"
- Note that this utility is in the repository but is not in the binary release.
- It looks for agents in PerformanceTests/TestAgents.
- You can add your own agents if you'd like to use them for testing.
- The utility will set the random seed to the same value each time to control some variability.
- There is a script called do_performance_tests.sh that calls the above executable for a variety of agents, with and without learning on.
- Odds are this script is not generally useful. It's something I made for testing performance changes related to my EBBS research, so most of the agents are ones that can learn new rules. The script runs them with and without learning, which is something most people probably won't need to do.
- If you run it with Soar 9.4, it will execute a similar test but only with the agents that work in 9.4.
PerformanceTest <agent name> <numtrials> <num_decisions> <num_init_and_rerun>
Arguments:
- agent_name: The file that loads your agent
- num_trials: How many times it runs your agent
- num_decisions: Maximum number of decision cycles to let agent run in each trial
- 0 lets agent run until it stops. Note that setting it to 0 can cause the program to hang if your agent doesn't stop.
- num_init_and_rerun: After agent is run (to max decisions or stopping), the utility will perform init-soar and re-run the agent this number of times.
- Most people won't need to use this option.
These are the 24 agents that come with the test utility:
- arithmetic94_learning.soar
- arithmetic94.soar
- arithmetic96_learning.soar
- arithmetic96.soar
- count-test-5000_learning.soar ( this is the agent that the original TestSoarPerformance would run)
- count-test-5000.soar
- count-test-5000.soarx
- FactorizationStressTest_learning.soar
- FactorizationStressTest.soar
- fifteen94_learning.soar
- fifteen94.soar
- fifteen96_learning.soar
- fifteen96.soar
- mac-planning94_learning.soar
- mac-planning94.soar
- mac-planning96_learning.soar
- mac-planning96.soar
- Teach_Soar_90_Games.soar
- wait_learning.soar
- wait.soar
- water-jug-lookahead94_learning.soar
- water-jug-lookahead94.soar
- water-jug-lookahead96_learning.soar
- water-jug-lookahead96.soar
Comment