Help needed: fixing non-standard `resource.path` in vega-datasets

[dsmedia]

What is your suggestion?

I'm working on improving Data Package compliance in vega-datasets (PR #755). One blocker is that the current datapackage.json uses non-standard paths that fail validation.

This is my design oversight in vega-datasets, not an Altair issue — but I need Altair's help to fix it without breaking things.

The Problem

vega-datasets/
├── datapackage.json      # resource.path = "airports.csv"  ← wrong
└── data/
    └── airports.csv

Current datapackage.json:

{
  "name": "vega-datasets",
  "resources": [
    {
      "name": "airports",
      "path": "airports.csv",
      "format": "csv",
      "bytes": 21176,
      "hash": "sha256:608ba6d..."
    }
  ]
}

Per the Data Package spec, paths must be relative to the descriptor. Since datapackage.json is at the repo root and files live in data/, the correct path should be data/airports.csv.

Altair correctly compensates for this oversight by hardcoding data/ in the base URL (npm.py:62):

def dataset_base_url(self, version: BranchOrTag, /) -> LiteralString:
    return f"{self._prefix(version)}data/"

If I fix vega-datasets now, Altair's URL construction breaks (double data/data/).

Request

Could Altair handle both path formats? This would let me fix vega-datasets without a coordinated release.

Format	resource.path	Expected URL
Current	airports.csv	.../data/airports.csv
Fixed	data/airports.csv	.../data/airports.csv

Suggested Approach

# npm.py - remove hardcoded "data/"
def dataset_base_url(self, version: BranchOrTag, /) -> LiteralString:
    return f"{self._prefix(version)}"

# datapackage.py - normalize paths for backwards compatibility
@property
def _url(self) -> Column:
    path_col = col("path")
    normalized = pl.when(path_col.str.starts_with("data/")).then(
        path_col
    ).otherwise(
        pl.concat_str(pl.lit("data/"), path_col)
    )
    expr = pl.concat_str(pl.lit(self._base_url), normalized)
    return Column("url", expr, "Remote url used to access dataset.")

Also file_name would need to extract just the filename:

Column("file_name", col("path").str.split("/").list.last(), ...)

Migration

1. Altair adds backwards-compatible path handling
2. vega-datasets fixes paths to data/airports.csv
3. Standard validators pass, Altair continues working

Happy to help with the implementation if useful.

Related

• feat: improve Data Package metadata compliance with CKAN licenses and field schemas vega-datasets#755: feat: improve Data Package metadata compliance — blocked by this
• Data Package path spec: https://datapackage.org/standard/data-resource/#path

Have you considered any alternative solutions?

No response

[mattijn]

I think we are using a fixed version of vega-datasets if I remember correctly.

So this issue is a heads-up once we want to bump to a newer version of vega-datasets? Or is this an issue directly after you release a new data package specification?

[dsmedia]

If you use a fixed version, then it's not an immediate issue. That said, I would feel more comfortable implementing the fix in vega-datasets knowing (and being able to test) that it will be compatible with Altair.

I think we are using a fixed version of vega-datasets if I remember correctly.

So this issue is a heads-up once we want to bump to a newer version of vega-datasets? Or is this an issue directly after you release a new data package specification?

[mattijn]

Altair side of things for the vega datasets generation is started here:

altair/tools/generate_schema_wrapper.py

Line 1409 in 64e0d68

datasets.app.refresh(VERSIONS["vega-datasets"], include_typing=True)

This is done using versions as written here (currently tag v3.2.1):

altair/altair/utils/schemapi.py

Lines 1690 to 1701 in 64e0d68

	VERSIONS: Mapping[
	Literal[
	"vega-datasets", "vega-embed", "vega-lite", "vegafusion", "vl-convert-python"
	],
	str,
	] = {
	"vega-datasets": "v3.2.1",
	"vega-embed": "v7",
	"vega-lite": "v6.1.0",
	"vegafusion": "2.0.3",
	"vl-convert-python": "1.8.0",
	}