Dataset & tags filtering DSL

[pudo]

Goal: we want to be able to filter datasets and collections in a way that is less confusing than the current yente API endpoints offer.

Idea: let's implement a DSL for defining subsets of the data collection using logical operators and #tag filters. A typical pattern may be: take this collection and remove a,b,c - or: just add 1, 2, 3.

Sketches:

datasets=(#issuer.eu|#list.sanction)-lt_fiu_sanctions
datasets=(#issuer.west|#list.sanction|#list.debarment)-lt_fiu_sanctions-#issuer.ru

This could be used:

• In yente for /match and /search.
• In zavod for scoped CLI commands like zavod xref, or even as a way to dynamically define collections.
• To dynamically generate custom-scoped exports for customers:

https://delivery.opensanctions.com/entities.ftm.json?datasets=(#issuer.west&#list.sanction&#list.debarment)-lt_fiu_sanctions-#issuer.ru

[pudo]

Claude doing some good thinking here ;)

The minimal complete set is surprisingly small:

Theoretical minimum: 1 operator

NAND (or NOR) alone is functionally complete—you can build everything else from it. But nobody wants to write NAND(NAND(A, NAND(A, B)), NAND(B, NAND(A, B))) for A | B.

Practical minimum: 2 operators

NOT + AND or NOT + OR (De Morgan equivalents):

A | B  =  NOT(NOT(A) AND NOT(B))
A - B  =  A AND NOT(B)

Or with NOT + OR:

A & B  =  NOT(NOT(A) OR NOT(B))
A - B  =  NOT(NOT(A) OR B)

For your DSL specifically

You're dealing with set operations, not boolean logic per se. The minimal useful set:

Operators	What you lose	Practical?
& -	Union requires ALL - ((ALL - A) & (ALL - B))	Awkward
| -	Intersection requires A - (A - B) or double subtraction	Workable
| &	No way to exclude specific datasets	Incomplete for your use case
| - !	Can derive & via A & B = A - (A - B)	Complete, readable

My recommendation for OpenSanctions

Keep three operators: | & -

Reasons:

1. - (subtraction) is your most common operation: "take this collection, remove X"
2. & is essential for "sanctions AND from EU issuers"
3. | is essential for "sanctions OR debarments"
4. Adding ! (complement) requires defining a universal set, which gets messy with dynamic collections

The current grammar is already minimal for practical use. You could drop & since A & B = A - (A - B), but that's hostile to users writing #issuer.eu&#list.sanction.

[leonhandreke]

Thought that I threw out in the eng meeting today: we should skip writing a parser and just spec the AST in JSON, like ES or Mongo does for example.

[pudo]

I like it a ton. Basically:

{
  "and": [
    "sanctions",
    "#issuer.west",
    "not": {
       "#issuer.gb"
    }
  ]
}

^ need to polish this :)