Announcement

Collapse
No announcement yet.

Utility to test Soar's performance with an agent

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Utility to test Soar's performance with an agent

    We talked a little about measuring Soar's performance in today's meeting. I mentioned a simple utility called TestSoarPerformance, and several people couldn't find it. Now that I've looked back at the releases, I can see why. It no longer exists!

    I think I've reconstructed that memory now and can explain why. If I recall correctly, TestSoarPerformance was limited, so I replaced it with a slightly more elaborate utility. (This happened a few years ago between 9.5 and the current version.)
    • The executable is called "PerformanceTests"
      • Note that this utility is in the repository but is not in the binary release.
    • It looks for agents in PerformanceTests/TestAgents.
      • You can add your own agents if you'd like to use them for testing.
    • ​​​​​​​The utility will set the random seed to the same value each time to control some variability.
    • There is a script called do_performance_tests.sh that calls the above executable for a variety of agents, with and without learning on.
      • Odds are this script is not generally useful. It's something I made for testing performance changes related to my EBBS research, so most of the agents are ones that can learn new rules. The script runs them with and without learning, which is something most people probably won't need to do.
      • If you run it with Soar 9.4, it will execute a similar test but only with the agents that work in 9.4.
    If you'd like to use the utility yourself, here's the usage:
    PerformanceTest <agent name> <numtrials> <num_decisions> <num_init_and_rerun>

    Arguments:
    • agent_name: The file that loads your agent
    • num_trials: How many times it runs your agent
    • num_decisions: Maximum number of decision cycles to let agent run in each trial
      • 0 lets agent run until it stops. Note that setting it to 0 can cause the program to hang if your agent doesn't stop.
    • num_init_and_rerun: After agent is run (to max decisions or stopping), the utility will perform init-soar and re-run the agent this number of times.
      • Most people won't need to use this option.

    These are the 24 agents that come with the test utility:
    1. arithmetic94_learning.soar
    2. arithmetic94.soar
    3. arithmetic96_learning.soar
    4. arithmetic96.soar
    5. count-test-5000_learning.soar ( this is the agent that the original TestSoarPerformance would run)
    6. count-test-5000.soar
    7. count-test-5000.soarx
    8. FactorizationStressTest_learning.soar
    9. FactorizationStressTest.soar
    10. fifteen94_learning.soar
    11. fifteen94.soar
    12. fifteen96_learning.soar
    13. fifteen96.soar
    14. mac-planning94_learning.soar
    15. mac-planning94.soar
    16. mac-planning96_learning.soar
    17. mac-planning96.soar
    18. Teach_Soar_90_Games.soar
    19. wait_learning.soar
    20. wait.soar
    21. water-jug-lookahead94_learning.soar
    22. water-jug-lookahead94.soar
    23. water-jug-lookahead96_learning.soar
    24. water-jug-lookahead96.soar
    If you have any questions, let me know.

  • #2
    FYI, the performance test utility won't build if you just run the scons script without any arguments. Use either the performance_tests target or the all target:

    python scons/scons.py performance_tests

    python scons/scons.py all

    Comment


    • #3
      I tried running the program with various parameter values, and got the following results:

      PS C:\Users\plind> cd C:\Users\plind\Documents\GitHub\Soar\PerformanceTe sts
      PS C:\Users\plind\Documents\GitHub\Soar\PerformanceTe sts> PerformanceTests
      e[1;31mcount-test-5000e[0;37m: single run, run forever
      1 ✅
      e[1;34mcount-test-5000 e[0;37m:e[0;93m 1.7898 1.7898 1.7898 0.0%e[0;37m
      ---------------------------------------------------------------------------------
      PS C:\Users\plind\Documents\GitHub\Soar\PerformanceTe sts> PerformanceTests wait
      e[1;31mwaite[0;37m: single run, run forever
      1
      PS C:\Users\plind\Documents\GitHub\Soar\PerformanceTe sts> PerformanceTests wait 3
      e[1;31mwaite[0;37m: 3 trials, run forever
      1
      PS C:\Users\plind\Documents\GitHub\Soar\PerformanceTe sts> PerformanceTests wait 3 1000
      e[1;31mwaite[0;37m: 3 trials, 1000 DCs
      1 ✅ 2 ✅ 3 ✅
      e[1;34mwait e[0;37m:e[0;93m 0.0025 0.0016 0.0031 49.8%e[0;37m
      ---------------------------------------------------------------------------------
      PS C:\Users\plind\Documents\GitHub\Soar\PerformanceTe sts> PerformanceTests wait 3 1000 5
      e[1;31mwaite[0;37m: 3 trials, 1000 DCs, 5 extra init-soar/runs
      1 .....✅
      2 .....✅
      3 .....✅

      e[1;34mwait e[0;37m:e[0;93m 0.0022 0.0010 0.0039 73.8%e[0;37m
      ---------------------------------------------------------------------------------
      PS C:\Users\plind\Documents\GitHub\Soar\PerformanceTe sts> PerformanceTests arithmetic94 3 1000 5
      e[1;31marithmetic94e[0;37m: 3 trials, 1000 DCs, 5 extra init-soar/runs
      1 .....✅
      2 .....✅
      3 .....✅

      e[1;34marithmetic94 e[0;37m:e[0;93m 0.0213 0.0151 0.0390 61.3%e[0;37m
      ---------------------------------------------------------------------------------
      PS C:\Users\plind\Documents\GitHub\Soar\PerformanceTe sts>


      That all seems like a lot of gibberish to me. How can we interpret the output?

      Comment


      • #4
        If that’s how it looks on your system (and not a cut and paste artifact), I think the problem is that you’re seeing the ANSI color codes. Maybe there’s a setting in your terminal program that lets it understand those?

        Comment


        • #5
          It's not a cut and past artifact, that's exactly how it looks in the Windows PowerShell window. Besides the gibberish, there are also some numbers that might mean something, but there's nothing to tell us what they mean.

          Comment


          • #6
            Yeah, then you're seeing the ANSI color codes. Unfortunately, Windows has its own color escape codes. Sorry, but this was a small, simple personal utility (~100 lines of code) that wasn't meant for public release, and I have only needed to use it on OSX and Ubuntu.

            For each run, it outputs four values.
            1. Value returned by the "time run" command.
            2. Real time in seconds (parsed from the "stats" command after the run)
            3. Kernel CPU Time (parsed from the "stats" command after the run)
            4. Total CPU Time (parsed from the "stats" command after the run)

            P.S. If you really want to use it on Windows, you could just delete the color strings from PerformanceTests.cpp and recompile. There's only 2 or 3 of them. Just look for anything that starts with "\e. Or, you could try to get Windows or your terminal program to read those. I saw something about Windows 10 being able to accept them standard escape seq (https://stackoverflow.com/questions/...ows-10-console), and I also saw this: https://www.codeproject.com/Tips/525...indows-Console. FYI, I didn't look into either of those options closely.

            Comment


            • #7
              By the way, I don't use this utility any more. You're welcome to take out those color codes from the development branch and check it in.

              Comment

              kusadasi escort
              alanya escort
              free sex porn hd porno
              ankara escort
              replica rolex
              xxx oiling anally full
              pornvideoshd.net
              huluhub.com
              Working...
              X