Learning from research - what can we learn from Sampy

[jackiekazil]

This paper cited Mesa and built a model in Python but didn't use Mesa citing speed reasons.

I haven't combed through the paper yet but plan on it. I would love to consider what we can learn from this --

Here is the project https://github.com/sampy-project

Citation of paper: [PDF] SamPy: A New Python Library for Stochastic Spatial Agent-Based Modeling in Epidemiology of Infectious Diseases
F Viard, E Acheson, A Allibert, C Sauve, P Leighton - 2022
Agent-based models (ABMs) are computational models for simulating the actions and interactions of autonomous agents in time and space. These models allow users to simulate the complex interactions between individual agents and the landscapes they inhabit and are increasingly used in epidemiology to understand complex phenomena and make predictions. However, as the complexity of the simulated systems increases, notably when disease control interventions are considered …
https://www.preprints.org/manuscript/202211.0556/download/final_file&hl=en&sa=X&d=4467816798941415421&ei=UCyLY_TjGIvymgHU24bYBw&scisig=AAGBfm3Q6RX39kD9kvvuYNDIWIovLRI4VA&oi=scholaralrt&hist=oI0CgqgAAAAJ:6732058387213389799:AAGBfm3OH--jYN-B3iSfQ930GzLBIoeHLw&html=&pos=0&folt=cit&fols=

[jackiekazil]

@mrgoodfish @sampy-project
Would love to get insights, benchmarks, passing thoughts on where Mesa failed for future feature/improvements.

[rht]

I haven't gone through the detail, but from reading https://github.com/sampy-project/sampy-paper/blob/385fea6046fd20ffb6d5e84dfdc97da48e4dc0d3/sampy/agent/base.py#L117-L175, it seems that agents are bulk-added, each as a row in a pandas DataFrame. The agent action seems to be a pandas operation. This is definitely be fast for millions of agents. i.e. the agents are not defined as Python objects.

[mrgoodfish]

@jackiekazil
Sorry, I just saw the notification today.

Mesa did not fail in any way on our end. It's actually tricky to compare Mesa and SamPy, since their designs and goals are very different. For instance, SamPy's scope is very specialized.

I think the best way to explain my point is to give some context on why we ended up making SamPy. Initially, people in my lab were working with the Ontario Rabies Model (ORM). This is an ABM, coded from scratch in C# and sadly not open-source (so I can't share the code), specially designed to modelize rabies dynamics within raccoons and whose structure is quite classical (for instance agents are objects, etc...). People in the team wanted to expand the scope of this model, by studying different species and increasing the size of the landscapes/populations (for instance, one project involves modelling the entirety of the population of arctic foxes in the arctic circle, including genetics and sea-ice dynamic). However, we faced several difficulties: (1) the code was hard to modify (the documentation was not up to date) and (2) performance was a problem at those scales (a single simulation could take weeks).

So we decided to create a new ABM, and we obviously considered the "normal" options like Mesa or Netlogo. Given that ORM structure is pretty much compatible with those frameworks, it should be possible to "translate ORM" into any of those. This would necessarily lead to a speed up, at the very least because this translation implies to clean up the code. However, given the state of ORM's code this would take a considerable ammount of work. Furthermore, given the similarity in structure between ORM and ABMs developped with usual frameworks, we couldn't expect drastic differences in terms of performance. That's why we finally decided to try something really different, and the result is SamPy.

As mentionned by @rht , one of the core difference is that there is no longer any object for individual agent. Instead, Sampy has population objects, and users create their models using those populations. The details are explained in the preprint (but I can provide more explanations if you wish), but this choice allowed us to have a great control over memory layout and memory allocation. It also allowed us to use a lot of compiled code for intensive computations. In the end, we got the speed boost we wanted, as simulations that took 11 days with ORM now take 30 minutes with similar outputs (for our needs at least).

However, the catch is that our new models are quite different from usual ABMs (and obviously more restricted in terms of what can be accurately modelized with them), and I really don't know if it even make sense to try to compare them. For instance, if I tried to reproduce an ABM created with Mesa (or with netlogo) in SamPy, I expect SamPy's memory organisation to actually create a lot of problems which will likely result in catastrophic performances (since the CPU would need to fetch data scattered around everywhere in the RAM each time a new agent is considered). Conversely, I don't know if it would make sense to reproduce an ABM from Sampy with Mesa, if at all possible without a lot of work.

[rht]

I think the idea of agent groups in SamPy can be abstracted out into a general ABM library. I'd imagine reproducing this paper would be easier using SamPy's method?