DataCollector for Mesa-Frames

[Ben-geo]

I’m working on a proposal for GSoC 2025 focused on improving data collection in Mesa-Frames, especially for large-scale or memory-constrained simulations. I’ve submitted my proposal that includes early implementations and prototype code for a DataCollector design
Since many of you have much more experience running large-scale agent-based models, I’d love to hear about the kinds of data you’ve found most useful to collect — or any pain points you’ve hit. Any feedback, edge cases, or thoughts would be really appreciated and could help guide improvements.

@EwoutH I noticed you’ve worked on some large-scale simulations mesa-frames discussion - would love to hear any insights you might have

Thanks!

DATA COLLECTOR in Mesa : projectmesa/mesa#1944
this is the latest proposed api :

# Illustrative Example API
from mesa.datacollection import DataCollector, collect
import numpy as np

# Assume Wolf, Sheep, Citizen Agent classes and calculate_gini defined elsewhere
class MyModel(mesa.Model):
    def __init__(self, ...):
        super().__init__()
        # ... model setup ...

        self.datacollector = DataCollector([
            # --- Model Level ---
            collect(name="step_count", target=self, attributes=["schedule.steps"]),
            collect(name="total_wealth", target=self,
                    function=lambda m: sum(a.wealth for a in m.agents)),

            # --- Agent Type Level (Resolved at runtime) ---
            collect(name="wolf_data", target=Wolf, attributes=["energy", "kills"]),
            # Example: apply function per agent (output stored per agent ID)
            collect(name="sheep_age_category", target=Sheep,
                    function=lambda agent: "lamb" if agent.age < 2 else "adult",
                    apply_level="agent"), # Hint needed for per-agent application

            # --- Dynamic AgentSet Level (Lambda evaluated at runtime) ---
            collect(name="starving_agents_pos",
                    target=lambda m: m.agents.select(lambda a: a.energy < 10),
                    attributes=["pos"]), # Collects pos attribute for matching agents
            collect(name="avg_starving_energy",
                    target=lambda m: m.agents.select(lambda a: a.energy < 10),
                    function=lambda agent_set: agent_set.agg("energy", np.mean)), # Aggregate over dynamic set

            # --- Aggregation Focused ---
            collect(name="energy_stats", target=self.agents, # Target can be an AgentSet
                    attributes=["energy"], # Base attribute(s) to collect first
                    aggregates={ # Dictionary: output_name -> func(list_of_values)
                        "mean_energy": np.mean,
                        "median_energy": np.median,
                        "energy_gini": calculate_gini # Assumes calculate_gini takes a list
                    }),
            # Direct aggregation on an AgentSet (e.g., count) without collecting individuals first
             collect(name="quiescent_count", target=Citizen, # Agent Type target
                    function=lambda agentset: agentset.select(lambda a: a.condition == "Quiescent").count), # agent_set passed to function

            # --- Conditional Collection (Addresses @Ben-geo's point) ---
            collect(name="periodic_wolf_aggression", target=Wolf, attributes=["aggression"],
                    trigger=lambda model: model.schedule.steps % 10 == 0) # Only collect every 10 steps
        ])

    def step(self):
        # ... step logic ...
        self.datacollector.collect(self) # Pass model instance to process collectors

[EwoutH]

Thanks for opening the discussion. This is a complicated one. Some background from Mesa:

• Multi-Agent data collection mesa#348
• The future of data collection mesa#1944

Don't read then all at once, touch some grass in between
I know these issue numbers from the top of my head. That's the state we're in.

I feel I'm close with #1944. A few full engineering days are needed, but with a full time job those are difficult to find.

@adamamer20 might have some insights on the Mesa-frames specifics to this.

[Ben-geo]

#1944 felt like fighting through the trenches trying to piece it all together, awesome read @EwoutH! Loved seeing how the discussions gradually shaped into a full API.

I will explore how I can incorporate whatever’s possible into mesa-frames as well , will share my thoughts as I map things out.

also hoping to see what others think, when it comes to data collection for mesa-frames I think storing the data is a major concern. the memory issue is much more of a concern here.