Workload gap filling

[BillGrieser-Antithesis]

The workload skills CAN leave gaps in what it covers in a SUT. I have been trying out various strategies to address this. The point of this issue is to record this as something we should spend time on to flesh out.

Things I tried:

1. I took a report of coverage edges missed and fed it to claude, asking it "what did we miss". It found some great things and we used this to decide what workload scripts to write next. We did in fact see the number of unique edges increase. The downside is that this coverage report is not something customer can request on their own (today).

Possible actions on this item: Productize some kind of coverage report; create a skill to do something with it.

This led to something that customers COULD do:

2. I also asked claude to insert Reachable assertions into the source code of the SUT for major functions. It was aware of the coverage reports so I asked it to insert them into places we hadn't reached already. If we don't want to make this require a coverage report, we should think about providing a skill to do this. We should provide guidance on what a good guidepost assertion is. Once these were in place they became very useful -- as we expanded the workload we could see new reachable assertions being reached which provided confidence the workload was expanding in interesting ways.

3. For a different approach, I asked claude to take a bottom-up approach and look at all the operations we COULD do through the API we were using to get to the system. I asked to to make a state transition graph showing what we could do at each state based on what we did before. (E.g., for a DB, after install state the only thing you can do is create a database. Once you have a database you can create a table. When you have a table, you can add records, create a view, modify the schema. Once you have records you can add more records, delete records, etc.) The workload became traversing the state transition graph and trying to visit all nodes. This is in progress and we will see if it is effective at reaching the reachability assertions we have not yet reached. This approach would be a candidate for a skill if effective.

[SeanTAllen]

Been thinking about this from a different angle than the three experiments on their own. Posting where I've landed so anyone picking this up cold can follow along.

The reframe: our current approach is top-down, and that's where the gaps come from

The research skill drives workload design from "what does this system say it does, and what properties should we assert about it?" That's useful work and it stays. But the output is bounded by what we looked for. Behavior the research pass didn't notice, the workload won't exercise. The three experiments in the issue are all, in different forms, attempts to supplement the top-down view with something bottom-up.

Watch the word "meaningful"

My first instinct was to frame bottom-up value as "helping us find meaningful bugs we'd otherwise miss." That framing is wrong. Worth pulling apart why. "Meaningful" smuggles in a claim we don't actually have: that we know in advance which bugs matter. We don't. What we have is a guess about where the worst problems might be. Importance of a bug is a post-hoc property, set by what we actually break. So the value of bottom-up approaches isn't "find more important bugs." It's "reduce blind spots in what the workload actually drives through." Narrower claim, more honest.

The interesting gap isn't catalog-to-workload

If we have 25 properties cataloged and 5 workloads implemented, the 20 unimplemented properties aren't an interesting gap. They're a backlog. The interesting gap is what's happening in the SUT that no property in the catalog ever described. Call that crevice search: finding the corners of the system that no workload touches because nothing in the catalog pointed there.

Mechanism: Reachable assertions as guideposts

#2 in the issue already hits this. Drop Reachable assertions into the SUT at places a thorough workload should reach, run what we have, and watch which ones don't fire. Unfired Reachables mean either "haven't written that workload yet" or "this area was missed entirely." Either way: cheap, customer-accessible today, no coverage-report dependency.

Timing: it works at any ratio

The 5-of-25 question has a clean answer under this framing. Crevice search is useful at 5/25, 25/25, or anywhere in between. The instrument stays the same: the set of Reachables you haven't hit. What changes is the interpretation. Early in implementation, an unfired guidepost probably means "haven't gotten there yet." Late, an unfired guidepost points at something the research pass never surfaced. Same tool, different reading.

This is a new skill, not a fold-in to the research skill

Crevice search is research-flavored work. It produces understanding of the SUT. But it isn't the same activity as building the property catalog, and folding it into the existing research skill would take a focused skill and turn it into a less-focused one. So: separate skill. Name TBD, but something in the "crevice search" or "guideposts" neighborhood.

Still open

• Placement heuristics. Where do Reachables go so they form useful guideposts? Entry points, state-changing operations, error branches are plausible starting points, but "plausible" isn't a skill. This has to get concrete.
• The feedback loop. When an unfired Reachable points at a crevice, what does the customer do with it? Update the property catalog? Write a workload directly for the crevice? Both? This shapes whether the skill just emits findings or has a handoff story to the rest of the suite.
• Static vs. runtime more broadly. We landed on starting static here and deferring the runtime/iterative story until it can be addressed across the whole skill suite rather than in isolation. Not crevice-search-specific, but worth flagging.
• Coverage-report productization. Separate track from this skill. add podman compose shenanigans to bootstrap skill #1 in the issue (feed missed coverage edges to Claude and ask what we missed) is the highest-signal version of all of this, but it depends on customers having a coverage report they can pull. Product work, not skill work. Parking here so we don't lose it.
• Antithesis triage skill #3 (state transition graph) isn't dropped. The design started around enumeration and gap-check, then I realized the interesting gap wasn't catalog-to-workload, which pulled me toward Reachables as the first move. State transition work may resurface as a placement heuristic ("put Reachables at every node and every legal transition") or as its own skill later. Not dropped, just not first.