|
| 1 | +--- |
| 2 | +layout: default |
| 3 | +title: 1st December |
| 4 | +description: ZEPs Meeting Notes for 2022-12-01 |
| 5 | +parent: ZEP meetings |
| 6 | +nav_order: 7 |
| 7 | +--- |
| 8 | + |
| 9 | +# 2022-12-01 |
| 10 | + |
| 11 | +**Attending:** Sanket Verma (SV), Josh Moore (JM), Jonathan Striebel (JS), Jeremy Maitin-Shepard (JMS), Ward Fisher (WF), Dennis Heimbigner (DH), Ryan Abernathey (RA) |
| 12 | + |
| 13 | +## TL;DR: |
| 14 | + |
| 15 | +RA started a discussion on drop `/meta` prefix. See the issue [here](https://github.com/zarr-developers/zarr-specs/issues/177), which basically led to chain reaction of several conversations around topics related to each other. These discussions are mostly around some lingering issues around the finalisation of Zarr V3 spec. |
| 16 | + |
| 17 | +RA, JMS and JS took some action items which can be seen at the bottom |
| 18 | + |
| 19 | +**Updates:** |
| 20 | + |
| 21 | +- Conversations (issues and feedback) on ZEP1 [PR](https://github.com/zarr-developers/zarr-specs/pull/149) are now resolved. Check [this](https://github.com/zarr-developers/zarr-specs/pull/149#issuecomment-1327605570). Thanks to Jonathan Striebel! 🙌🏻 |
| 22 | + - The conversations which needs additional input have been moved to separate issues |
| 23 | +- Jeremy Maitin-Shepard promoted as one of the authors for [ZEP0001](https://zarr.dev/zeps/draft/ZEP0001.html) |
| 24 | +- Current status of ZEP1 can be viewed [here](https://github.com/orgs/zarr-developers/projects/2) |
| 25 | + |
| 26 | +**Meeting minutes:** |
| 27 | + |
| 28 | +- RA: suggest focusing on the meta/ prefix discussion |
| 29 | + - [https://github.com/zarr-developers/zarr-specs/issues/177](https://github.com/zarr-developers/zarr-specs/issues/177) |
| 30 | + - JMS: not sure it's solving a problem (optimally). nice feature of v2 is copying out an array |
| 31 | + - JS: was for performance, use exclusion mechanism |
| 32 | + - RA: never need to list chunks (even if implementations do...) |
| 33 | + - NB: don't like trying to open files to know things (404-based) |
| 34 | + - JM: so we all agree? Yes. But what's the default? |
| 35 | + - RA: suggest: drop meta, use .json on the array |
| 36 | + - Can then drop the root metadata? |
| 37 | + - DH: there is dataset-level metadata (superblock) |
| 38 | + - JS: discuss those separately? |
| 39 | + - Agreed |
| 40 | + - JS: so to that suggestion, how do you list all metadata? |
| 41 | + - RA: don't think we should plan for discovering all metadata (millions of arrays) |
| 42 | + - JS: |
| 43 | + - RA: listing recursively isn't ok? |
| 44 | + - JS: not with implicit groups |
| 45 | + - RA: use storage transformer to get the previous behavior |
| 46 | + - RA: data is out there so need to provide a mechanism |
| 47 | + - DH: don't think that's fair |
| 48 | + - RA: nice feature of this proposal if we could keep it. |
| 49 | + - JM: how is conslidated metadata related? |
| 50 | + - RA: that's another problem. |
| 51 | + - RA: had thought about explicitly list the children (stac catalog) |
| 52 | + - DH: nczarr does that as well. |
| 53 | + - RA: downside is the concurrency issue |
| 54 | + - JS: good extension for groups (listing children) |
| 55 | + - JS: but could also have consolidated per group |
| 56 | + - JM: different commnunities here. some are definitely asking for listability |
| 57 | + - DH: not lots of formats that are listable without tools. They are asking for something powerful. |
| 58 | + - RA: so that's the root feature of the separate hierarchies |
| 59 | + - RA: we should look at some data. offer to write a script |
| 60 | + - JS one other alternative: chunks in an extra subfolder |
| 61 | + - JMS: k/v versus directory based will have different performance behavior |
| 62 | + - RA: does this require us to give up on implicit groups? |
| 63 | + - summary: foo/array.json and foo/chunks/0/0 |
| 64 | + - JMS: re: dropping of root metadata |
| 65 | + - solution is perhaps storage transformer would need to write something in _array_ metadata |
| 66 | + - JS: but then everything in the array metadata and you'd need be able to read it |
| 67 | + - JMS: don't _always_ need to access the metadata directly. more like a safety measure. |
| 68 | + - JMS: could copy "global" metadata into each array |
| 69 | + - JM: that works to the design goal of being able to freely copy an array |
| 70 | + - DH: that assumes no global extensions. |
| 71 | + - RA: need to think through this separately: |
| 72 | + - portability ("invariance" of v2 that any group is standalone) |
| 73 | + - RA: danger is that you can build zarrs through the command-line (need no library) |
| 74 | + - DH: sure you want to do that? |
| 75 | + - RA: it was at least part of the design of V2 |
| 76 | + - DH: as a principle, if that's what you want it's really critical to the v3 process |
| 77 | + - DH: cf. intellij -- fiddling in text file then going back (bypassing the tool) |
| 78 | + - JM: perhaps CLI manipulations are inherently "extension-less" and therefore this is "safe" |
| 79 | + - DH: one tool being a verification tool? |
| 80 | + - JS: consolidated metadata as an example |
| 81 | + - RA: primarily used as a way to allow easy listing |
| 82 | + - but you know that you can't touch the store. |
| 83 | + JS: you need a root to do fancy things |
| 84 | +- **action items:** |
| 85 | + - RA to do some performance benchmarking |
| 86 | + - JMS to propose a new storage layout for v3 |
| 87 | + - next time: root metadata discussion issue (JS) |
0 commit comments