Clarifying the `express` interface

[akirakyle]

I've been thinking a lot about interfaces as I keep working on #40 where among many other things, I've been trying to figure out the best interface for conversion among superoperators. I realized the design issues there are similar to the design issues for express, which is also in some sense, an interface for conversions between types. The julia docs also document conversion patterns which either use a type's constructor T(x) or Base.convert. I think the main issue in conversion is how to specify "additional" information necessary for conversion. For example the fock cutoff, or in the superoperator case, the precision with which the spectrum of a superoperator is obtained to convert to the kraus representation.

So I wanted to ask a few questions about how the express interface is or isn't supposed to handle the following hypothetical scenarios:

• Say I have a ket = QuantumSymbolics.CoherentState(α) where α is a symbolic number. Then I want a symbolic matrix representation of ket in the fock basis with a cutoff, so I want Ket(FockBasis(cutoff), exp(-{|α|^2/2}) * [1,α, α^2/sqrt(2),..., α^cutoff/sqrt(cutoff!)]. Would this be some hypothetical express(ket, SymbolicMatrixRepr(cutoff))? What if I explicitly want dense versus sparse representations?
• What if I want this ket expressed in PositionBasis or MomentumBasis? It seems like just adding the xmin, xmax, N or pmin, pmax, N fields to QuantumOpticsRepr won't specify which of FockBasis, PositionBasis, or MomentumBasis one would want?

Perhaps I'm totally misunderstanding the intended interface for express, but as it is, I'm not sure how it will capture all the types of conversions and the parameters they might need. In order to do this, it seems like combinations of AbstractRepresentation and AbstractUse will end up just being stand ins for a T(x) type constructor and it might be better just to use that pattern? For example Ket(FockBasis(cutoff), ket) where ket is as defined as above.

On the other hand, the type T(x) constructor method wouldn't seem to be able to handle the other use of express which converts expressions containing multiple terms. This is mainly relevant for QuantumSymbolics expressions and for Lazy* that can actually represent multiple terms. For example conversion of the symbolic ket⊗ket to LazyKet(FockBasis(cutoff)⊗FockBasis(cutoff), [ket,ket]). However, I think a more flexible design might be to have AbstractRepresentation be essentially just a set of conversion rules for how to call T(x) type constructors. Sensible sets can be provided but it should be easily to define one's own rules depending on what exactly on wants. For example, something schematically like

QuantumOpticsRepr(cutoff) = Dict(
CoherentState => ket -> Ket(FockBasis(cutoff), ket),
STensor => LazyTensor
)

Hopefully this would make it easier for a user to modify existing sets if they largely cover what one wants but want slight differences like only using LazyProduct or something. I'm imagining that, somewhat like SymbolicUtils.toexpr which build_function uses, we could traverse the expression tree starting from leaves and calling the appropriate T(x) conversion to build up the converted expression.

Something I'm also worried about is that express has preferred defaults, for example the current fock cutoff of 2 is way to low for any coherent state and won't give normalized states (which seems like another option express ought to have).

[akirakyle]

Some other relevant issues:

• Plans for Symbolic expressions in QuantumOptics QuantumOpticsBase.jl#70 (comment)
• Migrate express and AbstractRepresentation type from QuantumSymbolics #46 (comment)
• Convert from QuantumCumulants to QuantumOptics QuantumSavory/QuantumSymbolics.jl#111

[Krastanov]

I do not think express is particularly well defined outside of QuantumSymbolics yet.

Reading the base docs for convert, it seems semantically it is meant only for "safe and unsurprising conversions". I am leaning towards saying that most of the conversions considered here are not safe and unsurprising, so I am leaning towards using convert only as an inspiration in these discussion, but early on deciding that it will not be an overall part of solutions to be considered.

In those same docs (https://docs.julialang.org/en/v1/manual/conversion-and-promotion/) convert is compare to conversion through a type constructor, and it makes me comfortable with how we are already using that type of conversion. E.g. QuantumOptics.Ket(::QuantumClifford.Stabilizer) works as expected and there are a lot of other examples.

express was created specifically for conversion from QuantumSymbolics to QuantumOptics/QuantumClifford/others. Initially it was never meant to do more, but @apkille is finding more usecases (for similar reasons that you are describing).

express is separate from "constructor" conversions for two reasons:

• extra config options as you already mentioned
• there is a lot of custom tree-traversal to do with symbolics
• caching of conversion results was pretty important for even remotely good performance

The two examples you give are definitely far beyond what express has been designed for up to now, but if this functionality is useful it would be cool to start prototyping it. I do not have good initial design suggestions.

A few thoughts though:

• Using constructors as much as possible before using express. In Gabs.jl for instance, there are reasons to consider using express, but as much as possible should be done just with constructors.
• Having the representation objects (<:AbstractRepresentation) be immutable seems pretty useful and important to me -- it is restrictive too, which I like.
• What is the use case in which it makes sense to set up a particularly complicated representation instead of just doing the complicated conversion directly (with appropriate case-specific conversion tools)? Maybe in many cases we just do not need something as general as express.

[akirakyle]

Thanks for the thoughts!

I am leaning towards using convert only as an inspiration in these discussion, but early on deciding that it will not be an overall part of solutions to be considered.

I completely agree, especially after seeing this discourse comment. Also this is a relevant issue to keep in mind regarding convert and type T(x) constructors.

E.g. QuantumOptics.Ket(::QuantumClifford.Stabilizer) works as expected and there are a lot of other examples.

Ah great!

• Using constructors as much as possible before using express. In Gabs.jl for instance, there are reasons to consider using express, but as much as possible should be done just with constructors.

I also agree. Constructors seem to me to be cleaner for "single object" conversion since they indicate what the converted object will be along with requiring only the options necessary for conversion of that object. Perhaps @apkille can comment on usecases in Gabs.jl where express would be better than constructors?

• Having the representation objects (<:AbstractRepresentation) be immutable seems pretty useful and important to me -- it is restrictive too, which I like.

Yes this is definitely important for caching. The Dict based example was more to use a datastructure that makes explicit the direction of conversion. Immutable dictionaries woud still have the advantage of being able to derive new express rulesets easily from existing ones.

• What is the use case in which it makes sense to set up a particularly complicated representation instead of just doing the complicated conversion directly (with appropriate case-specific conversion tools)? Maybe in many cases we just do not need something as general as express.

The main use case I have in mind, although admittedly it is maybe a bit vague, is trying to generalize the Lazy* system in QuantumOpticsBase by relying on QuantumSymbolics. I probably need to study the Lazy* system more, but initially it seems like there's overlap between the two, but the key difference is that the Lazy system system seems to be mainly focused getting better performance than doing linear algebra in a "naive" order of operations fixed by the user's code. But care still has to be taken in how Lazy operators are nested together to get these algorithmic speedups. The symbolic approach is much more general since the whole expression tree can be represented and then analyzed for the best order of operations (this is what Finch.jl is trying to tackle for sparse linear algebra). So I'm imaging that one can, say, use STensor in lieu of LazyTensor to build up an expression. Then to ensure the fastest order of linear algebra operations are taken, the AST could be normalized to a form so that when the AST is traversed with express, the order of operations for numerical evaluation would be optimal.

[Krastanov]

Adding sophistication to Lazy* and how it interacts with QuantumSymbolics would be awesome (and quite valuable). Besides Finch, entities that are working on similar ideas include the ITensor folks. Also, the SciMLOperators.jl https://github.com/SciML/SciMLOperators.jl are related as discussed in qojulia/QuantumOpticsBase.jl#99