Performance goals & tracking?

[gavinp]

Hi,

Throughout reviewing the code, I see a lot of focus on performance. For instance, there's the separation of parsing of the Wube prototype data and the normal running of the library. The 2.0 branch TODOs even contemplates a rewrite into a compiled curly brace language (see https://github.com/redruin1/factorio-draftsman/blob/2.0/TODO.md#investigate-a-cythonrust-rewrite-in-efforts-to-make-the-library-as-performant-as-possible )

I'm curious: where's the current state of performance captured? Which tests? What's the benchmark?

Most important: what are the use cases? What operations will they be doing over and over? Is startup cost essential? This guides which operations are targets for improvement.

For the complex optimizations in the current code, do we track their cost/benefit? How much startup cost is draftsman update saving us?

[redruin1]

Unfortunately, it's basically no on all fronts currently. Originally performance was not a concern, expressive API was, but as the scope of the project grew operations that should be fast became slow, so I decided to focus more on performance concerns. However, since performance is not really a primary goal of the project, it always gets pushed to the wayside, especially now with all of the new features that need to be implemented and tested for correctness.

I think most performance concerns can be handled by keeping Draftsman as modular as possible going forward. For example, Draftsman has always had a level of blueprint validation or linting, in the sense that it checks for correctness of attributes and their values. This constant checking can eat a significant overhead if done at all times, which if the user does not want, is entirely wasteful. Draftsman 2.0 now gives users the option to disable runtime validation entirely, and only run validation when they want to. In this way, even if the validation function itself is slow, it won't be done when users don't want it, and how long it takes doesn't really matter to the person who wants a detailed report of mistakes in their blueprint strings.

That being said, simple operations should be fast. Setting signals in constant combinators, adding entities to a blueprint, setting descriptions, icons, etc. Complex operations (like searching a blueprint) can take arbitrarily longer, within reason.

Draftsman currently emulates the Factorio load process offline and stores the data to a set of pickle files, which are then (always) loaded when the module is imported. Loading from these pickle files is faster than dynamically running the emulation every time the module is run (I don't have explicit tests for this but know it to be true), but if you want you can manually call the update function from your script on init if desired. Perhaps if the emulation could be sped up by a magnitude or two you could get rid of the intermediary step altogether, but that's assuming you actually want all of the data from the game, instead of just the information that Draftsman needs to perform blueprint string validation.