AST Parsing and Python 3.11

What's Changed

• Support multi-line lambda functions which are valid before complete by @gordonwatts in #103
• Test to make sure multi-line lambda with funny breaks works by @gordonwatts in #105
• Add as_awkward and as_pandas, etc by @gordonwatts in #106
• Python 3.11 support by @gordonwatts in #107

Full Changelog: 3.0...3.1b1

Types & Metadata

This is a major release. Old queries against old backends should work without any changes. However, you'll need a new backend to take advantage of most of the new features.

Features marked with a ** require the new backend.

New Features in the language

• C++ Code injection can be done via a decorator
• List comprehension and Generator expressions are now supported, automatically turned into func_adl statements. This is particularly helpful when doing 2-object matching.
• Local and global variables are properly captured in lambda's.

New Features in the infrastructure

• Support for Query Metadata. This metadata is sent to the backend without being touched **
• Support for in-query Metadata. This metadata is not sent to the backend, but can be extracted from the query. The in-query metadata can be overridden by setting a value more than once. **
• A Fully typed ObjectStream is supported via Generics. Various editors that type-follow will be able to correctly predict methods, etc., that are available. You'll need a complete object model for this to work (see the func_adl_servicex_xaodr21 package) **
• Python 10 is supported.

The PR's

• Add a function to allow MetaData over the wire by @gordonwatts in #78
• Added typing into ObjectStream by @gordonwatts in #80
• Support Python 10 by @gordonwatts in #86
• Extensability via the type system by @gordonwatts in #82
• Pythonic Decorators for Code Injection by @gordonwatts in #87
• Support Lambda Capture by @gordonwatts in #88
• Turning on black as a formatter by @gordonwatts in #89
• Parameterized Method Calls by @gordonwatts in #90
• Pr_query_metadata by @gordonwatts in #91
• Support ListComp and Generator expressions by @gordonwatts in #92
• Fixes to get a version that runs in production by @gordonwatts in #94
• More robust parsing of ast's from callables by @gordonwatts in #97
• Regression (untested) for lambdas continuing accross lines by @gordonwatts in #98
• Calls to AsXXX with column names as single items or single arrays should be the same by @gordonwatts in #99
• Bug Fix: argument names should override captured variables by @gordonwatts in #100

Full Changelog: 2.3...3.0

Fix multi-line ast parsing regression

Regression (untested) for lambdas continuing accross lines (#98)

* Fix up multi-line function parsing

Add better ast parsing

• More robust in finding and parsing source code in methods
• Ability to provide the finder hints

Allow IfExpr to see Any as a float

Minor bug when any's get used in IfExpr. I get the feeling for untyped stuff, we'll be finding a lot of things like this.

Production Running

Can run the complete notebook that demos xaod_usage

• Do not require clients to install the func-adl.ast package.
• Fix bug in how QMetaData updates items in its dict (overriden items were ignored!)
• Add a warning if there is a lambda that isn't scanned for type forwarding
• Fix bug where function names were being captured during lambda resolution
• Move to a single package rather than two (no more .ast). This is certianly a breaking change!

Bug Fixes

• incorrect __init__ file - made meta data clearing for the unreleased version of func_adl_servicex fail.
• QMetaData was not dealing with updated items properly.

Remove the ast package - everything as one now

• There is no ast package now - everything comes down at once. This will cause knock-on effects in everyone's configuration, sadly.

Move metadata into client part of library

Another cross-include into the ast part of the library fixed.

Fix bug introduced

Moved the wrong file