Usage of `VarName` as keys

[torfjelde]

This is a bit of a drive-by comment, but I've so far failed to wrap my head around how we use VarNames as keys in containers. I feel like sometimes we use subsumes, sometimes we don't, and maybe sometimes we also do some ad hoc half way thing. This is complicated further by having e.g. multiple backend storage structures for (Simple)VarInfo, and having getters and setters for them defined in a few different places (#654). Having to do string representations with Chains is also a big contributor to confusion.

I don't really have a proposal for how to change this, but for many cases like your above hasvalue examples it is not clear to me what I should expect to happen. A more systematic approach, with [mumblemumble interfacesomething design doc mumblemumble] might help solidify this.

Originally posted by @mhauru in #710 (comment)

Opened an issue here @mhauru to avoid cluttering the PR. I'll reply in comments.

[torfjelde]

I feel like sometimes we use subsumes, sometimes we don't, and maybe sometimes we also do some ad hoc half way thing.

Part of this is simply because how you can use VarName as a key depends on the underlying storage of the values and the contextual information you have at the call-site.

If you give me a dict with keys like

@varname(x[:][1] => 0), @varname(x[:][2])

and ask me for the value of @varname(x), the proper answer is "I don't know 🤷 " becuase I don't have the original shape of x; I only know what x[:] looks like.

This is the flavour of issue we're encountering in #710 (comment).

Even if we did know the shape, e.g. we instead had

@varname(x[1][1] => 0), @varname(x[1][2])

and you ask for @varname(x), we can in theory reconstruct x as needed, but man is that going to be annoying to implement + won't work in complete generality (we'll have to reconstruct the array based on keys which might contain negative indices, keys when using component arrays, etc.)

So, a lot of this confusion, IMO, is caused by the fact that VarName isn't perfect information but requires contextual information, and hence it's utility depends on the context in which it's being used.

[penelopeysm]

Even if we did know the shape, e.g. we instead had @varname(x[1][1] => 0), @varname(x[1][2]) and you ask for @varname(x), we can in theory reconstruct x as needed

And even that assumes that x[1][1] and x[1][2] are the only sub-varnames of x, for all we know, x might actually a 10-by-10 matrix and we can't reconstruct the other 98 entries. 😩

[penelopeysm]

(For example, that's why contextual_isfixed can't really catch what happens when only part of an array is fixed – it checks the symbol x and the varnames passed to fix(), but there's no way to tell whether the varnames cover all of x unless (1) the varnames include x itself, or (2) we have runtime information about the shape of x.)

I've been toying with the idea of having varnames that carry information about the shape of the parent symbol, but I'm finding it really hard to tell whether that will solve anything without actually attempting to code it up.

[torfjelde]

I've been toying with the idea of having varnames that carry information about the shape of the parent symbol, but I'm finding it really hard to tell whether that will solve anything without actually attempting to code it up.

This might work nicely for standard Base.Array, buuuut once you start working with stuff like OffsetArrays.jl or ComponentArrays.jl, things will get real messy I think.

[yebai]

I've been toying with the idea of having varnames that carry information about the shape of the parent symbol, but I'm finding it really hard to tell whether that will solve anything without actually attempting to code it up.

This might work nicely for standard Base.Array, buuuut once you start working with stuff like OffsetArrays.jl or ComponentArrays.jl, things will get real messy I think.

I thought of this problem a bit myself some time ago. I think we could make this work by

• carry the type information, e.g. Array
• carry the shape information as @penelopeysm suggested above
• provide a default reconstruction implementation for Array types. Users could extend this by implementing new reconstruction functions for additional types if needed, but by default, we can throw an error for unknown types, e.g. OffsetArrays