Track LPNF vs. OPNF performance #65
Description
The goal of these is to determine if we can get rid of the LegacyPropNetFactory entirely in favor of the OptimizingPropNetFactory.
Results from a recent round of testing that focused on time taken:
The LPNF is faster on smaller games, where the exact time for computation doesn't really matter. The OPNF consistently takes at least 200 ms to run, while the LPNF can finish in under 20 ms. Instead, we look for games where the OPNF takes several seconds to run (at least) or fails while the LPNF is significantly faster.
Games where the OPNF takes at least 5 seconds to build a propnet and the LPNF is significantly faster:
Dresden/ggp-course2013_jordi [256] (6 vs. 1 seconds)
GGP.org/pancakes6 [70] (10 vs. <1 seconds)
GGP.org/pancakes [215] (10 vs. <1 seconds)
Dresden/knightstour [260] (40 vs. <1 seconds)
Dresden/ggp-course2013_a1 [396] (73 vs. 4 seconds)
Games where the OPNF times out (120 second limit) or fails and the LPNF does not:
GGP-Base/test_case_5a [483]
Stanford/tictictoe [429] (fails StaticValidator)
GGP.org/knightsTourLarge [21]
GGP.org/factoringImpossibleTurtleBrain [18]
Dresden/ggp-course2013_michal13 [373]
GGP.org/knightwar [33]
GGP.org/pancakes88 [171]
Dresden/ggp-course2013_nicolai [338]
GGP.org/god [66]
Note that games listed above may be earlier versions of games that have since been modified. (For some of these games, at least, adding bases and inputs makes the OPNF faster.) I need to go through and weed these out. Numbers in square brackets are game IDs from a personal database, for my reference (and so my output code doesn't conflate different versions of a game).