Zarr chunk corruption in Level1 magnetics data for 11xxx/12xxx series shots

[RHH-CQER]

We are conducting disruption prediction research using MAST Mirnov coil data (see also #211 re: Level2 quantization).

Summary

Approximately 18 shots from the 11xxx and 12xxx series have zarr chunk size mismatches in their Level1 magnetics data, making them unreadable. The corruption is in the archive itself — re-downloading does not resolve it. Shot 46943 is also missing magnetics data entirely.

Affected Shots

From a batch of 23 shots we attempted to download and process, these failed:

11xxx series (chunk size mismatch):
11779, 11848, 11849, 11852, 11855, 11860, 11897, 11957

12xxx series (similar corruption):
12144, 12167, 12185, 12230, 12291, 12441, 12442, 12444, 12457

Missing magnetics entirely:
46943

Successfully loaded:
12091, 12438, 12455, 29975, 29980, 29981

Failure Mode

When opening the zarr stores, the declared chunk shape in .zarray metadata does not match the actual stored chunk sizes. This causes zarr to raise errors on read. The issue persists across fresh downloads, confirming it is upstream.

Reproduction

# Download a known-affected shot
s5cmd --no-sign-request --endpoint-url https://s3.echo.stfc.ac.uk \
  cp 's3://mast/level1/shots/11779.zarr/amb/ccbv01/*' 11779_level1/amb/ccbv01/

# Attempt to read
python -c "
import zarr
store = zarr.open('11779_level1/amb/ccbv01', 'r')
print(store[:])  # Expected: chunk mismatch error
"

Impact

We were attempting to validate a disruption prediction method across a broad shot range. The 18 failed shots reduced our usable dataset from 24 to 6, limiting statistical power. The 11xxx series shots would have been particularly valuable as they span a different operational period.

Questions

1. Is the chunk corruption a known issue for these shot ranges?
2. Are there plans to rebuild/repair these zarr stores?
3. Is there an alternative path to access the underlying data for these shots (e.g. different archive format or direct request)?
4. Is shot 46943 expected to have no magnetics data, or is that also corruption?

Environment

• Access method: s5cmd v2.x via s3.echo.stfc.ac.uk
• Reader: Python zarr library
• Date of access: December 2025 – January 2026

Related

• Level2 magnetics data severely quantized (~6 effective bits despite float64 storage) #211 — Level2 quantization issue (separate problem, same diagnostic)

[samueljackson92]

@NathanCummings @LariDG can you check, but I think this is the same as #202 and should be fixed as high priority.

[RHH-CQER]

Correction to "Successfully loaded" list

On re-auditing our local data, I need to correct the "Successfully loaded" section:

• 12091, 12438, 12455: Confirmed — these have Level1 Mirnov coil data (amb/ccbvNN, 37–38 coils each). ✅
• 29975, 29981: Our download log reported 35 coils each, but the _level1 directories no longer exist locally and I cannot verify. Other 29xxx Level1 paths only contain amc/plasma_current, not Mirnov coils. Moving to unverified.
• 29980: This shot was not included in our Level1 bulk download. Should not have been listed. ❌

Corrected list:

Confirmed: 12091, 12438, 12455
Unverified: 29975, 29981
Incorrectly listed: 29980

This is consistent with @samueljackson92's observation in #211 that raw L1 XMC data may not have been pushed publicly for the 29xxx range. Thank you for linking this to #202 — helpful to know it's being tracked.

[LariDG]

Hi @RHH-CQER ,

Is this still a continued issue or was it resolved in #211 ?
Just trying to reproduce your errors but am unable to. The shots that you have listed do not give me a chunk mismatch error, although we have seen this in corrupted data in level2 shots like in issue #202 mentioned above.

Example of your given shots:

store = zarr.open('https://s3.echo.stfc.ac.uk/mast/level1/shots/11779.zarr/amb/ccbv01', 'r')
store[:]
array([ 1.57755392e-04,  1.58644791e-04,  1.59534204e-04, ...,
       -1.41875125e-05, -1.32981031e-05, -1.24086937e-05],
      shape=(7500,), dtype=float32)

store = zarr.open('https://s3.echo.stfc.ac.uk/mast/level1/shots/12144.zarr/amb/ccbv01', 'r')
store[:]

I have also downloaded this locally and get the same result.

For the missing mirnov, is this in regards to xmc as answered by Sam? I see that amb and amc are present in the 29xxx shots for level1.

[RHH-CQER]

Hi @LariDG,

Thank you for checking. I've just re-tested all the originally reported shots directly against S3 and can confirm:

• 11779, 11808, 11848, 11862, 11885: all now load without chunk mismatch errors
• 12144, 12167, 12168, 12233, 12291, 12401, 12462, 12444, 12457: all load correctly
• 11960, 11987, 11997: these no longer exist on S3 (amb/ccbv01 path not found) — so it's possible these were removed rather than repaired
• 46943: still not present on S3

So the chunk corruption issue does appear to be resolved for all shots that remain in the archive. It's likely this was fixed during the same maintenance period that addressed #202 / #211. Apologies for not re-checking sooner.

On the missing Mirnov: yes, resolved. We can confirm amb data (36 ccbv coils) is present in Level1 for 29xxx shots including 29975, 29980, and 29981. Our original report was based on incomplete local downloads.

Happy for this issue to be closed. Thank you for following up.

[LariDG]

Hi @RHH-CQER,

I am glad to hear that the shots now all load correctly. To clarify, we have not pushed any changes/done maintenance to any of the data. A change will be pushed to the level2 data shortly that will fix #202.

I presume that the errors you were finding before will have likely been a corruption on download.

• 11960, 11987, and 11997: are you sure that you were able to pull these shots before? There is no amb signal for these shots at the source so hence also not in the s3 bucket.

• 46943: is not on our system as it is a MAST-U shot and hence not open publicly at the moment. The max shot currently available is 30471.