Releases | Open Source Science

func-adl.ast - 3.5.0 - `in` operator, dictionary merging, bug fixes

See below for full change list

You can now use the in operator (p.absPdgId() in [31, 51])
You can now use dictionary combinations, even with if expressions as long as those can be evaluated as the func adl query is being assembled. This is designed to make MC vs Data ntuple extraction more straight forward.
A few bug fixes (sqrt no longer produced the undeclared warning, some specific code formats are now parsed correctly).

What's Changed

Fix parsing of multiline lambda dictionaries by @gordonwatts in https://github.com/iris-hep/func_adl/pull/196
Implement the in operator by @gordonwatts in https://github.com/iris-hep/func_adl/pull/197
Add sqrt as builtin function by @gordonwatts in https://github.com/iris-hep/func_adl/pull/198
Support dict unpacking with conditional expression by @gordonwatts in https://github.com/iris-hep/func_adl/pull/199

Full Changelog: https://github.com/iris-hep/func_adl/compare/3.4.2...3.5.0

- Python
Published by gordonwatts 8 months ago

func-adl.ast - 3.5.0b1 - beta of dictionary and in operator

The big changes are you can now combine dictionaries in your Select statement. This allows you to dynamically alter the returned ntuple depending on mc/data.

What's Changed

Fix parsing of multiline lambda dictionaries by @gordonwatts in https://github.com/iris-hep/func_adl/pull/196
Implement the in operator by @gordonwatts in https://github.com/iris-hep/func_adl/pull/197
Add sqrt as builtin function by @gordonwatts in https://github.com/iris-hep/func_adl/pull/198
Support dict unpacking with conditional expression by @gordonwatts in https://github.com/iris-hep/func_adl/pull/199

Full Changelog: https://github.com/iris-hep/func_adl/compare/3.4.2...3.5.0b1

- Python
Published by gordonwatts 8 months ago

func-adl.ast - 3.4.2 - Fix bug in how Range is interpreted

Fixed a bug in how Range was called.

What's Changed

Make sure Range isn't Replaced by @gordonwatts in https://github.com/iris-hep/func_adl/pull/190

Full Changelog: https://github.com/iris-hep/func_adl/compare/3.4.1...3.4.2

- Python
Published by gordonwatts 11 months ago

func-adl.ast - 3.4.1 - Correctly parse nested functions

This is a small bug fix to make sure nested functions can be used in our code.

What's Changed

Feat/nestedfunctions by @gordonwatts in https://github.com/iris-hep/funcadl/pull/188

Full Changelog: https://github.com/iris-hep/func_adl/compare/3.4.0...3.4.1

- Python
Published by gordonwatts 11 months ago

func-adl.ast - 3.4.0 - enums, data classes, 3.13, lambda arguments

Lots of new features in this code. See the list of pull requests below. In particular:

dataclasses and named ntuples are now supported, allowing for the breakdown of large queries to be much easier.
enums are now much more carefully handled, and correctly passed down to the backend
outter arguments in a nested lambda function can now be correctly referenced.

Lots of other things, like official 3.13 python support, the start of removing 3.8 support, documentation, citation files, etc.

There is some possible backwards incompatibility - in particular in the way parameterized calls are processed. If you have any, you should test!

What's Changed

Add preferred citation and CITAITON.cff by @gordonwatts in https://github.com/iris-hep/func_adl/pull/157
fix: Add citation information for repository by @matthewfeickert in https://github.com/iris-hep/func_adl/pull/162
docs: Add conda-forge badge to README by @matthewfeickert in https://github.com/iris-hep/func_adl/pull/153
build: Support Python 3.9+ by @matthewfeickert in https://github.com/iris-hep/func_adl/pull/159
Support 3.13 and also pyproject.toml by @gordonwatts in https://github.com/iris-hep/func_adl/pull/166
Fixup project distriubtion files by @gordonwatts in https://github.com/iris-hep/func_adl/pull/168
Add dataclasses and named tuples as syntatic sugar by @gordonwatts in https://github.com/iris-hep/func_adl/pull/176
Add docs for named tuples and data classes by @gordonwatts in https://github.com/iris-hep/func_adl/pull/177
Lambda arguments can be captured by @gordonwatts in https://github.com/iris-hep/func_adl/pull/182
deep copy for deref tuple, list, dict by @gordonwatts in https://github.com/iris-hep/func_adl/pull/184
Improve error message when you forget to parameterize a call with [cppxxx] by @gordonwatts in https://github.com/iris-hep/funcadl/pull/185
Enum pass through by @gordonwatts in https://github.com/iris-hep/func_adl/pull/186

Full Changelog: https://github.com/iris-hep/func_adl/compare/3.3.3...3.4.0

- Python
Published by gordonwatts 11 months ago

func-adl.ast - 3.4.0b7 - enum and other fixes

Specific fixes below - but lots of details making code in the wild work.

What's Changed

Lambda arguments can be captured by @gordonwatts in https://github.com/iris-hep/func_adl/pull/182
deep copy for deref tuple, list, dict by @gordonwatts in https://github.com/iris-hep/func_adl/pull/184
Improve error message when you forget to parameterize a call with [cppxxx] by @gordonwatts in https://github.com/iris-hep/funcadl/pull/185
Enum pass through by @gordonwatts in https://github.com/iris-hep/func_adl/pull/186

Full Changelog: https://github.com/iris-hep/func_adl/compare/3.4.0b6...3.4.0b7

- Python
Published by gordonwatts 11 months ago

func-adl.ast - 3.4.0b6 - typed (named) data classes and tuples

What's Changed

Add preferred citation and CITAITON.cff by @gordonwatts in https://github.com/iris-hep/func_adl/pull/157
fix: Add citation information for repository by @matthewfeickert in https://github.com/iris-hep/func_adl/pull/162
docs: Add conda-forge badge to README by @matthewfeickert in https://github.com/iris-hep/func_adl/pull/153
build: Support Python 3.9+ by @matthewfeickert in https://github.com/iris-hep/func_adl/pull/159
Support 3.13 and also pyproject.toml by @gordonwatts in https://github.com/iris-hep/func_adl/pull/166
Fixup project distriubtion files by @gordonwatts in https://github.com/iris-hep/func_adl/pull/168
Add dataclasses and named tuples as syntatic sugar by @gordonwatts in https://github.com/iris-hep/func_adl/pull/176
Add docs for named tuples and data classes by @gordonwatts in https://github.com/iris-hep/func_adl/pull/177

Full Changelog: https://github.com/iris-hep/func_adl/compare/3.3.3...3.4.0b6

- Python
Published by gordonwatts 12 months ago

func-adl.ast - 3.4.0b5 - and again

Full Changelog: https://github.com/iris-hep/func_adl/compare/3.4.0b4...3.4.0b5

- Python
Published by gordonwatts 12 months ago

func-adl.ast - 3.4.0b4 - another attempt

Full Changelog: https://github.com/iris-hep/func_adl/compare/3.4.0b3...3.4.0b4

- Python
Published by gordonwatts 12 months ago

func-adl.ast - 3.4.0b3 - another try at python publishing

Full Changelog: https://github.com/iris-hep/func_adl/compare/3.4.0b2...3.4.0b3

- Python
Published by gordonwatts 12 months ago

func-adl.ast - 3.4.0b2- see if the pypi publishing pipeline works

Full Changelog: https://github.com/iris-hep/func_adl/compare/3.4.0b1...3.4.0b2

- Python
Published by gordonwatts 12 months ago

func-adl.ast - 3.4.0b1 - Test Release

Making a move to python 3.13 and upgrading build system. This is to test the pypi push mechanism to make sure it is still working. You can use this, but it doesn't have any functional changes.

What's Changed

Add preferred citation and CITAITON.cff by @gordonwatts in https://github.com/iris-hep/func_adl/pull/157
fix: Add citation information for repository by @matthewfeickert in https://github.com/iris-hep/func_adl/pull/162
docs: Add conda-forge badge to README by @matthewfeickert in https://github.com/iris-hep/func_adl/pull/153
build: Support Python 3.9+ by @matthewfeickert in https://github.com/iris-hep/func_adl/pull/159

Full Changelog: https://github.com/iris-hep/func_adl/compare/3.3.3...3.4.0b1

- Python
Published by gordonwatts 12 months ago

func-adl.ast - Bug fix: Zip operator crash and `len` warning

Fixed one crash and one extra warning message:

Zip now won't crash when trying to do the internal type checking. It does not yet return the proper result, but that should be safe for most uses (see #150).
len operator no longer produces a warning and type checks as a float.

What's Changed

Type-Checking the Zip operator on Dictionary should not crash by @gordonwatts in https://github.com/iris-hep/func_adl/pull/149
Test that triggers the len error by @gordonwatts in https://github.com/iris-hep/func_adl/pull/151

Full Changelog: https://github.com/iris-hep/func_adl/compare/3.3.2...3.3.3

- Python
Published by gordonwatts over 1 year ago

func-adl.ast - Add license

Apparently a blocker for conda is not having explicit license... Here is the fix.

What's Changed

feat: Add LICENSE by @matthewfeickert in https://github.com/iris-hep/func_adl/pull/144

Full Changelog: https://github.com/iris-hep/func_adl/compare/3.3.1...3.3.2

- Python
Published by gordonwatts over 1 year ago

func-adl.ast - Dict type-following

Prior to this, when you accessed methods that were part of a dictionary the type following wasn't done correctly. This release will update to code that does this correctly.

Note this is still now working correctly for tuples!

What's Changed

Type propagation through dict and tuples by @gordonwatts in https://github.com/iris-hep/func_adl/pull/140
Deep-Type Dictionaries by @gordonwatts in https://github.com/iris-hep/func_adl/pull/142

Full Changelog: https://github.com/iris-hep/func_adl/compare/3.3.0...3.3.1

- Python
Published by gordonwatts almost 2 years ago

func-adl.ast - Bug fix for deref

bug fix for released code.

- Python
Published by gordonwatts almost 2 years ago

func-adl.ast - Deep-type following for dictionaries

Prior to this, type information was lost when we used dictionaries in Select statements. This fixes that.

- Python
Published by gordonwatts almost 2 years ago

func-adl.ast - Fix dict/tuple type following bug

Correctly follow types through tuple and dict references.

- Python
Published by gordonwatts almost 2 years ago

func-adl.ast - Fix type following for tuples an dicts

Fix up type following when there are dict and tuple references

Full Changelog: https://github.com/iris-hep/func_adl/compare/3.3.0...3.3.1b1

- Python
Published by gordonwatts almost 2 years ago

This release adds explicit support for enums in the type libraries and fixes a regression that was preventing parameterized functions from working correctly. Otherwise, there should be no changes in how this operates and it should be backward compatible.

What's Changed

(feature) Enable Enums by @gordonwatts in https://github.com/iris-hep/func_adl/pull/133
(bug) Allow types for parameterized functions by @gordonwatts in https://github.com/iris-hep/func_adl/pull/138

Full Changelog: https://github.com/iris-hep/func_adl/compare/3.2.8...3.3.0

- Python
Published by gordonwatts almost 2 years ago

func-adl.ast - Relax library requirements

Thanks to @matthewfeickert for this!

Support 3.7+, without having a cap on the release version.
Start the migration away from the old distutils

There should be no functional difference.

What's Changed

build: Support Python 3.7+ by @matthewfeickert in https://github.com/iris-hep/func_adl/pull/134

Full Changelog: https://github.com/iris-hep/func_adl/compare/3.2.7...3.2.8

- Python
Published by gordonwatts almost 2 years ago

func-adl.ast - More uniform `lambda` capture

What's Changed

Support Python 3.12 by @gordonwatts in https://github.com/iris-hep/func_adl/pull/129
Imported module names/function should be correctly passed to qastle. by @gordonwatts in https://github.com/iris-hep/func_adl/pull/131

Full Changelog: https://github.com/iris-hep/func_adl/compare/3.2.6...3.2.7

- Python
Published by gordonwatts over 2 years ago

func-adl.ast - Don't add a null-length MetaData to stream

Minor bug fix

What's Changed

Do not add an empty MetaData call by @gordonwatts in https://github.com/iris-hep/func_adl/pull/126

Full Changelog: https://github.com/iris-hep/func_adl/compare/3.2.5...3.2.6

- Python
Published by gordonwatts over 2 years ago

func-adl.ast - 3.2.5 - Retain original instance of ObjectStream

- Python
Published by BenGalewsky over 2 years ago

func-adl.ast - Support for new ServiceXClient

Maintain the identity of the object stream subclass and don't downcast to ObjectStream

- Python
Published by BenGalewsky over 2 years ago

func-adl.ast - Bug Fix: Do not alter python run-time types

This fixes a bug discovered by Haoran and Callum - if you ask for two different typed collections one after the other, the type propagation system gets very confused because I was accidentally modifying a Python runtime type. This fixes that.

What's Changed

Do not alter runtime types by @gordonwatts in https://github.com/iris-hep/func_adl/pull/120

Full Changelog: https://github.com/iris-hep/func_adl/compare/3.2.3...3.2.4

- Python
Published by gordonwatts almost 3 years ago

func-adl.ast - Do not alter python runtime types at... runtime

Testing the fix for runtime type alterations.

Full Changelog: https://github.com/iris-hep/func_adl/compare/3.2.2...3.2.4b1

- Python
Published by gordonwatts almost 3 years ago

func-adl.ast - Use function names as clues for parsing

This is a bug-fix release to address a regression. The following should have parsed correctly but was failing with a "multiple-lambdas on one line" error:

ds.Where(lambda e: e.met > 10).Select(lambda e: e.met)

What's Changed

Use Function Names as Clues Parsing Lambda's by @gordonwatts in https://github.com/iris-hep/func_adl/pull/116

Full Changelog: https://github.com/iris-hep/func_adl/compare/3.2.2...3.2.3

- Python
Published by gordonwatts about 3 years ago

func-adl.ast - Update package metadata

Minor updates to incorrect package metadata for pypi.

Full Changes

Full Changelog: https://github.com/iris-hep/func_adl/compare/3.2...3.2.2

- Python
Published by gordonwatts about 3 years ago

func-adl.ast - Release

There is something very broken about the last release, trying again.

- Python
Published by gordonwatts about 3 years ago

func-adl.ast - Lambda Parsing Bug Fix

The engine that finds lambda's for translation in source code has received a major upgrade/change. It now uses the builtin python tokenize module.

This makes it much more robust when dealing with comments, new lines, and other oddities.
More safely checks are built in to catch mistakes in matching lambda expressions to the intended code.

There is one regression - which is caused because this code is more rigorous in how it treats the source code: Using the python \ continuation character means that the line is really treated as a single line. This was not the case with the previous version of the parser. This may cause some expressions that work now to fail upon upgrade. You can just change the argument name, use ( and ) around the whole expression to get rid of the continuation characters, etc. to get your code quickly working again. You'll get an exception if this happens suggesting one of these two approaches if your code is affected.

What's Changed

Spelling update and flake8 control by @gordonwatts in https://github.com/iris-hep/func_adl/pull/112
Use builtin Python tokenizer to find lambdas by @gordonwatts in https://github.com/iris-hep/func_adl/pull/114

Full Changelog: https://github.com/iris-hep/func_adl/compare/3.1.2...3.2

- Python
Published by gordonwatts about 3 years ago

func-adl.ast - Lambda Parsing Upgrade

Moving to a new tokenizer (the builtin one!!) for finding where lambda' expressions are!

What's Changed

Spelling update and flake8 control by @gordonwatts in https://github.com/iris-hep/func_adl/pull/112

Full Changelog: https://github.com/iris-hep/func_adl/compare/3.1.2...3.2b1

- Python
Published by gordonwatts about 3 years ago

func-adl.ast - Lambda Parsing Bug Fix

Found a black inspired reformat that tickled a bug in the func_adl parser. Updated.

What's Changed

Infinite loop if this lambda is parsed in Jupyter by @gordonwatts in https://github.com/iris-hep/func_adl/pull/111

Full Changelog: https://github.com/iris-hep/func_adl/compare/3.1.1...3.1.2

- Python
Published by gordonwatts about 3 years ago

func-adl.ast - Lambda Parsing Upgrade

Correctly parse a lambda formatting that black will often impose:

python my_obj().do_it( lambda e: e.Jets("AntiKt4EMTopoJets").do_it( lambda j: j.Jets("AntiKt4EMTopoJets").do_it( lambda j1: j1.pt() / 1000.0 ) ) )

- Python
Published by gordonwatts over 3 years ago

func-adl.ast - Lambda Parsing Upgrade

This point release is focused on correctly parsing lambda's - increasing the range of styles of code we can correctly detect inside a Select, SelectMany and Where statement.

For example, the following two statements now work (and didn't properly work before):

python my_obj().Select(lambda x: x + 1 + 2 + 20 )

and

python my_obj().do_it(lambda event: event + 1 ).do_it(lambda event: event)

Since this is using heuristics, likely there are cases we've missed. Please do not hesitate to submit a bug report (or PR).

What's Changed

Support multi-line lambda functions which are valid before complete by @gordonwatts in https://github.com/iris-hep/func_adl/pull/103
Test to make sure multi-line lambda with funny breaks works by @gordonwatts in https://github.com/iris-hep/func_adl/pull/105
Add as_awkward and as_pandas, etc by @gordonwatts in https://github.com/iris-hep/func_adl/pull/106
Python 3.11 support by @gordonwatts in https://github.com/iris-hep/func_adl/pull/107

Full Changelog: https://github.com/iris-hep/func_adl/compare/3.0...3.1

- Python
Published by gordonwatts over 3 years ago

func-adl.ast - AST Parsing and Python 3.11

What's Changed

Support multi-line lambda functions which are valid before complete by @gordonwatts in https://github.com/iris-hep/func_adl/pull/103
Test to make sure multi-line lambda with funny breaks works by @gordonwatts in https://github.com/iris-hep/func_adl/pull/105
Add as_awkward and as_pandas, etc by @gordonwatts in https://github.com/iris-hep/func_adl/pull/106
Python 3.11 support by @gordonwatts in https://github.com/iris-hep/func_adl/pull/107

Full Changelog: https://github.com/iris-hep/func_adl/compare/3.0...3.1b1

- Python
Published by gordonwatts over 3 years ago

func-adl.ast - Types & Metadata

This is a major release. Old queries against old backends should work without any changes. However, you'll need a new backend to take advantage of most of the new features.

Features marked with a ** require the new backend.

New Features in the language

C++ Code injection can be done via a decorator
List comprehension and Generator expressions are now supported, automatically turned into func_adl statements. This is particularly helpful when doing 2-object matching.
Local and global variables are properly captured in lambda's.

New Features in the infrastructure

Support for Query Metadata. This metadata is sent to the backend without being touched **
Support for in-query Metadata. This metadata is not sent to the backend, but can be extracted from the query. The in-query metadata can be overridden by setting a value more than once. **
A Fully typed ObjectStream is supported via Generics. Various editors that type-follow will be able to correctly predict methods, etc., that are available. You'll need a complete object model for this to work (see the func_adl_servicex_xaodr21 package) **
Python 10 is supported.

The PR's

Add a function to allow MetaData over the wire by @gordonwatts in https://github.com/iris-hep/func_adl/pull/78
Added typing into ObjectStream by @gordonwatts in https://github.com/iris-hep/func_adl/pull/80
Support Python 10 by @gordonwatts in https://github.com/iris-hep/func_adl/pull/86
Extensability via the type system by @gordonwatts in https://github.com/iris-hep/func_adl/pull/82
Pythonic Decorators for Code Injection by @gordonwatts in https://github.com/iris-hep/func_adl/pull/87
Support Lambda Capture by @gordonwatts in https://github.com/iris-hep/func_adl/pull/88
Turning on black as a formatter by @gordonwatts in https://github.com/iris-hep/func_adl/pull/89
Parameterized Method Calls by @gordonwatts in https://github.com/iris-hep/func_adl/pull/90
Prquerymetadata by @gordonwatts in https://github.com/iris-hep/func_adl/pull/91
Support ListComp and Generator expressions by @gordonwatts in https://github.com/iris-hep/func_adl/pull/92
Fixes to get a version that runs in production by @gordonwatts in https://github.com/iris-hep/func_adl/pull/94
More robust parsing of ast's from callables by @gordonwatts in https://github.com/iris-hep/func_adl/pull/97
Regression (untested) for lambdas continuing accross lines by @gordonwatts in https://github.com/iris-hep/func_adl/pull/98
Calls to AsXXX with column names as single items or single arrays should be the same by @gordonwatts in https://github.com/iris-hep/func_adl/pull/99
Bug Fix: argument names should override captured variables by @gordonwatts in https://github.com/iris-hep/func_adl/pull/100

Full Changelog: https://github.com/iris-hep/func_adl/compare/2.3...3.0

- Python
Published by gordonwatts almost 4 years ago

func-adl.ast - Fix multi-line ast parsing regression

- Python
Published by gordonwatts almost 4 years ago

func-adl.ast - Add better ast parsing

More robust in finding and parsing source code in methods
Ability to provide the finder hints

- Python
Published by gordonwatts almost 4 years ago

func-adl.ast - Allow IfExpr to see Any as a float

Minor bug when any's get used in IfExpr. I get the feeling for untyped stuff, we'll be finding a lot of things like this.

- Python
Published by gordonwatts almost 4 years ago

func-adl.ast - Production Running

Can run the complete notebook that demos xaod_usage

Do not require clients to install the func-adl.ast package.
Fix bug in how QMetaData updates items in its dict (overriden items were ignored!)
Add a warning if there is a lambda that isn't scanned for type forwarding
Fix bug where function names were being captured during lambda resolution
Move to a single package rather than two (no more .ast). This is certianly a breaking change!

- Python
Published by gordonwatts almost 4 years ago

func-adl.ast - Bug Fixes

incorrect __init__ file - made meta data clearing for the unreleased version of func_adl_servicex fail.
QMetaData was not dealing with updated items properly.

- Python
Published by gordonwatts almost 4 years ago

func-adl.ast - Remove the ast package - everything as one now

There is no ast package now - everything comes down at once. This will cause knock-on effects in everyone's configuration, sadly.

- Python
Published by gordonwatts almost 4 years ago

func-adl.ast - Move metadata into client part of library

Another cross-include into the ast part of the library fixed.

- Python
Published by gordonwatts almost 4 years ago

func-adl.ast - Fix bug introduced

Moved the wrong file

- Python
Published by gordonwatts almost 4 years ago

func-adl.ast - Do not require func-adl.ast in client installs

I'd accidentally added a dependency!

- Python
Published by gordonwatts almost 4 years ago

func-adl.ast - Type Hint Support, MetaData, Syntactic Sugar

Lots of big features. This should be backwards compatible, but there are major upgrades. Hence the change in version number.

New Features:

It is possible to properly type-hint so that a good editor will be able to understand typing in func_adl queries. This makes heavy use of python's Generic type as we are often working with streams of objects (like events or jets, etc.).
Python list comprehensions and generator expressions inside lambda's are now converted to func_adl automatically. Ifs and multiple generators are supported.
Variables referenced inside lambda expressions are now properly captured. So you can parameterize a cut some external variables. Method and attribute and function calls on captured variables are not supported.

New Extensibility Points

Parameterized method calls are supported. You can write something like [j.getAttribute[cpp_float]('EMF') for j in jets] and with the proper support it will generate a C++ template function. The func_adl library provides none of the support, but you can use the decorator func_adl_parameterized_call
You can now use the @func_adl_callback decorator to decorate a class or a method in your type hints file to invoke custom python code as the query is built. This allows extensions, default argument processing, etc. The C++ xAOD backend heavily uses this to implement things like calibration, etc.
Query Metadata can be attached to an expression. It is stripped out before being sent down to ServiceX. This can be used to cache information that needs to be tracked in a query. The C++ xAOD type system uses this to track calibration configuration.
MetaData can be sent down to the backend at arbitrary points in the query. This can be used, for example, to declare types to the C++ backend.

Technical Debt:

Numerous clean ups to the type hinting in the library
Test coverage improvements, which uncovered some python version incompatibilities (which were fixed).
Using black as the formatted for the repo
README documentation improvements
Supporting 3.7 is getting to be work. Python 3.10 is supported.
Added full support for the type bool.

- Python
Published by gordonwatts almost 4 years ago

func-adl.ast - Add Python 3.10 Support

Python 3.10 is out. This has now been tested with func_adl.

- Python
Published by gordonwatts about 4 years ago