tfs-pandas - Release 4.0.0 - MAD-NG Compatibility

Version 4.0 of tfs-pandas is a major release bringing compatibility with MAD-NG features in TFS files and tables, apart from the most exotic ones (more on that later).

We also bring new documentation pages regarding the TFS format, code compatibilities, and the new features. Please have a look at the documentation should you intent to use tfs-pandas 4.0, as there are a few (now documented) caveats.

Important:

• Support for Python 3.9 has been dropped. The minimum required Python version is now 3.10.
• DataFrame validation is now OFF by default, both when reading from and writing to file. It is up to the user to ask for a given validation mode.
• Minimum supported MAD-NG version is 1.0.0, due to synchronized development of some feature details. The corresponding pymadng version is 0.6.0.

Added:

• Handling of boolean-type and complex-type header values (MAD-NG feature).
• Handling of bolean-type and complex-type columns (MAD-NG feature).
• Handling of nullable-type nil values in headers and columns (MAD-NG feature).
• Compatibility modes for dataframe validation. The tfs.frame.validate function now expects this, and valid choices are MADX, MAD-X, MADNG and MAD-NG (case-insensitive, see API documentation).
• Many new exceptions have been created for raised errors, which will be more specific. They all inherit from the previously raised TfsFormatError, so user code that was catching it will still work.

Changed:

• By default, TfsDataFrame validation is now skipped on reading.
• By default, TfsDataFrame validation is now skipped on writing.
• By default, TfsDataFrame validation in MAD-X requires headers to be present in the dataframe.

Fixed:

• Writing a dataframe with no headers (not empty headers), e.g. a pandas.DataFrame - now works correctly.

Documentation:

• The documentation has been updated to reflect the new features and changes.
• The documentation now includes a new page on the TFS format itself.
• The documentation now includes a new page on compatibility modes for TfsDataFrame validation.
• A great deal of internal documentation has been added to the codebase.

Relevant Changes

• MAD-NG Features and Compatibility Modes by @fsoubelet in https://github.com/pylhc/tfs/pull/135

Full Changelog: https://github.com/pylhc/tfs/compare/v3.9.0...v4.0.0

- Python
Published by fsoubelet over 1 year ago

tfs-pandas - Release 3.9.0

Release 3.9.0 brings a helper function into the public API for testing purposes.

Additions

• A module, tfs.testing, has been added and made publicly available. It provides an assert function to compare TfsDataFrame similar to that provided by pandas, destined for unit tests.

What's Changed

• Unit Testing Helper by @fsoubelet in https://github.com/pylhc/tfs/pull/136

Full Changelog: https://github.com/pylhc/tfs/compare/v3.8.2...v3.9.0

- Python
Published by fsoubelet over 1 year ago

tfs-pandas - Release 3.8.2

Release 3.8.2 brings two minor changes, which are fully transparent to the user.

Changes

• The headers of a TfsDataFrame are now stored as a dict and no longer an OrderedDict. This should save some memory.
• Removed a workaround function for some pandas behaviour which is no longer necessary due to the higher minimum pandas version.

What's Changed

• Remove unnecessary function and lots of linting by @fsoubelet in https://github.com/pylhc/tfs/pull/132
• Use dicts instead of OrderedDicts by @fsoubelet in https://github.com/pylhc/tfs/pull/133

Full Changelog: https://github.com/pylhc/tfs/compare/v3.8.1...v3.8.2

- Python
Published by fsoubelet almost 2 years ago

tfs-pandas - Release 3.8.1

Release 3.8.1 is a patch version fixing compatibility with numpy 2.x. Please note there is a caveat (read below) on Python 3.9 when installing tfs-pandas with the hdf5 extra.

Important Changes:

• The minimum required numpy version is now numpy 1.24.

Fixes

• The package's HDF functionality is fully compatible with numpy 2.x on Python >= 3.10 thanks to a pytables compatibility release.
• The package's HDF functionality limits to numpy < 2 on Python 3.9 due to the lack of compatibility from pytables on this versions.

What's Changed

• Migrate to pyproject.toml by @fsoubelet in https://github.com/pylhc/tfs/pull/131

Full Changelog: https://github.com/pylhc/tfs/compare/v3.8.0...v3.8.1

- Python
Published by fsoubelet almost 2 years ago

tfs-pandas - Release 3.8.0

Release 3.8.0 is a new minor version fixing warnings, updating to newer versions of our dependencies, and changing some minimum requirements.

Important Changes:

• The minimum required pandas version is now pandas 2.1.
• Support for Python 3.8 has been dropped. The minimum required Python version is now 3.9.

Fixes

• Solved a DeprecationWarning appearing when writing a TfsDataFrame to disk due to the use of .applymap, byusing the now recommended .map method.
• Solved a DeprecationWarning appearing when reading a file from disk due to the use of delim_whitespace in our reader, by using the now recommended sep option.
• Solved a FutureWarning appearing when validating a TfsDataFrame due to the use of the pd.option_context('mode.use_inf_as_na', True) context manager during validation by explicitely casting infinite values to NaNs.
• Solved a FutureWarning appearing when validating a TfsDataFrame due to object downcasting happening during validation by explicitely infering dtypes first.

What's Changed

• Deprecations and FutureWarnings Fixes by @fsoubelet in https://github.com/pylhc/tfs/pull/128

Full Changelog: https://github.com/pylhc/tfs/compare/v3.7.3...v3.8.0

- Python
Published by fsoubelet almost 2 years ago

tfs-pandas - Release 3.7.3

This is a patch release which fixes a regression introduced in a previous release.

• Fixed:
  • Fixed a regression where the writing of a pd.Series-like object to disk was raising an error. It is now possible again.

What's Changed

• CI Fix by @fsoubelet in https://github.com/pylhc/tfs/pull/127
• [Fix] Series-like writing by @fsoubelet in https://github.com/pylhc/tfs/pull/130

Full Changelog: https://github.com/pylhc/tfs/compare/v3.7.2...v3.7.3

- Python
Published by fsoubelet about 2 years ago

tfs-pandas - Release 3.7.2

This is a patch release, assuring compatibility again with pandas >= 2.1.0.

• Fixed:
  • fixing the issues with pandas >= v2.1.0 (see tfs-pandas v3.7.1) by overwriting the _constructor_from_mgr function.

What's Changed

• Fixing pandas v2.1.0 compatibility by @JoschD in https://github.com/pylhc/tfs/pull/126

Full Changelog: https://github.com/pylhc/tfs/compare/v3.7.1...v3.7.2

- Python
Published by JoschD over 2 years ago

tfs-pandas - Release 3.7.1

Release 3.7.1 is a patch release to circumvent an issue with the latest pandas release, 2.1.0, which would cause an infinite recursion . This version and above are now prohibited in the dependencies while we investigate with the pandas devs.

What's Changed

• Migrate to common workflows by @fsoubelet in https://github.com/pylhc/tfs/pull/123
• Pandas dependency version restriction by @fsoubelet in https://github.com/pylhc/tfs/pull/125

Full Changelog: https://github.com/pylhc/tfs/compare/v3.7.0...v3.7.1

- Python
Published by fsoubelet over 2 years ago

tfs-pandas - Release 3.7.0

Release 3.7.0 adds some functionality with a reworked TfsCollection class.

• Changed:

  • The old write_to and get_filename of the TfsCollection class have been renamed to _write_to and _get_filename as they could only be accessed internally (due to the input parameters not available to the user).
  • The column which is set as index can now also be defined manually by overwriting the INDEX attribute, which defaults to "NAME".
  • The define properties functions have been moved into the Tfs-attribute marker class.
  • The MaybeCall class returns None in case of attribute not found, instead of previously an empty function.

• Added:

  • A get_filename(name) function to TfsCollection which returns the associated filename to the property with name name.
  • A get_path(name) function to TfsCollection which returns the actual file path of the property name.
  • A flush() function to TfsCollection which writes the current state of the TfsDataFrames into their respective files.
  • A write_tfs(filename, data_frame) function to TfsCollection which writes the data_frame to self.directory with the given filename.
  • A defined_properties property to TfsCollection which returns a tuple of strings of the defined properties on this instance.
  • A filenames property to TfsCollection which is a convenience wrapper for get_filename():

For details see the API documentation.

What's Changed

• TFSCollections cleaned and with paths by @JoschD in https://github.com/pylhc/tfs/pull/122

Full Changelog: https://github.com/pylhc/tfs/compare/v3.6.0...v3.7.0

- Python
Published by fsoubelet about 3 years ago

tfs-pandas - Release 3.6.0

Release 3.6.0 is a version compatibility release with pandas 2.0.

• Removed:

  • The append and join methods of TfsDataFrame have been removed.

• Changed:

  • The dependency version on pandas has been restored to >=1.0.0 as the above removal restores compatibility with pandas 2.0.

What's Changed

• Release 3.6.0 and removal of TfsDataFrame methods by @fsoubelet in https://github.com/pylhc/tfs/pull/120

Full Changelog: https://github.com/pylhc/tfs/compare/v3.5.3...v3.6.0

- Python
Published by fsoubelet about 3 years ago

tfs-pandas - Release 3.5.3

This is a patch fixing a mistake in the previous patch.

• Changed:
  • Fixed a wrong deprecation of the .merge method of TfsDataFrames.

What's Changed

• 3.5.3 with no warning for merge method by @fsoubelet in https://github.com/pylhc/tfs/pull/121

Full Changelog: https://github.com/pylhc/tfs/compare/v3.5.2...v3.5.3

- Python
Published by fsoubelet about 3 years ago

tfs-pandas - Release 3.5.2

Release 3.5.2 is a dependency compatibility patch.

As of pandas 2.0 the append and join methods have been removed from pandas.DataFrame. This fails the corresponding compatibility methods of TfsDataFrames.

This patch version forces pandas<2.0 to guarantee the compatibility methods provided for TfsDataFrames work. They will be removed in the next version to enable compatibility with pandas 2.x.

In the future users should use the compatibility tfs.frame.concat instead.

• Changed:
  • The pandas version dependency is pinned to <2.0
  • A warning was added to the documentation of the append and join methods of TfsDataFrames.
  • A warning is now logged when the append or join methods of TfsDataFrames are used, redirecting the users to tfs.frame.concat.

What's Changed

• Pandas 2.x restraint by @fsoubelet in https://github.com/pylhc/tfs/pull/119

Full Changelog: https://github.com/pylhc/tfs/compare/v3.5.1...v3.5.2

- Python
Published by fsoubelet about 3 years ago

tfs-pandas - Release 3.5.1

Release 3.5.1 patches a single bug

• Fixed:
  • Allow reading of empty lines in headers again.

What's Changed

• Bug/fix empty lines skipping by @JoschD in https://github.com/pylhc/tfs/pull/118

Full Changelog: https://github.com/pylhc/tfs/compare/v3.5.0...v3.5.1

- Python
Published by JoschD about 3 years ago

tfs-pandas - Release 3.5.0

Release 3.5.0 adds some functionality and a bugfix.

• Fixed:

  • Any empty strings ("") in a file's columns will now properly be read as such and not converted to NaN.

• Added:

  • It is now possible to only read the headers of a file by using a new function, read_headers. The function API is not exported at the top level of the package but is available to import from tfs.reader.

What's Changed

• [Fix]: Properly read empty strings and allow reading headers only by @fsoubelet in https://github.com/pylhc/tfs/pull/116

Full Changelog: https://github.com/pylhc/tfs/compare/v3.4.0...v3.5.0

- Python
Published by fsoubelet about 3 years ago

tfs-pandas - Release 3.4.0

Release 3.4.0 adds the functionality to read and write compressed files to tfs-pandas. Just use tfs.read/tfs.write normally with compressed files and enjoy.

What's Changed

• Python 3.11 in Github Workflows and Readme additions by @fsoubelet in https://github.com/pylhc/tfs/pull/111
• [Feature]: reading and writing compressed files by @fsoubelet in https://github.com/pylhc/tfs/pull/113

Full Changelog: https://github.com/pylhc/tfs/compare/v3.3.1...v3.4.0

- Python
Published by fsoubelet over 3 years ago

tfs-pandas - Release 3.3.1

Release 3.3.1 brings a slight performance improvement in file loading.

• Changed:
  • The column types are assigned at read time and not later on, which should speed up loading for large data frames and/or slower machines.

What's Changed

• Assign column types at read time by @fsoubelet in https://github.com/pylhc/tfs/pull/110

Full Changelog: https://github.com/pylhc/tfs/compare/v3.3.0...v3.3.1

- Python
Published by fsoubelet over 3 years ago

tfs-pandas - Release 3.3.0

Release 3.3.0 adds features regarding data frame validation and quality of life improvements.

• Added:

  • The option is now given to the user to skip data frame validation after reading from file / before writing to file. Validation is left on by default, but can be turned off with a boolean argument.

• Changes:

  • The tfs.frame.validate function has seen its internal logic reworked to be more efficient and users performing validation on large data frames should notice a significant performance improvement.
  • The documentation has been expanded and improved, with notably the addition of example code snippets.

Related PRs

• Optional validation step (and documentation improvements) by @fsoubelet in https://github.com/pylhc/tfs/pull/107
• Faster dataframe validation by @fsoubelet in https://github.com/pylhc/tfs/pull/109

Full Changelog: https://github.com/pylhc/tfs/compare/3.2.1...v3.3.0

- Python
Published by fsoubelet over 3 years ago

tfs-pandas - Release 3.2.1

• removed check for spaces in headers

- Python
Published by JoschD over 3 years ago

tfs-pandas - Release 3.2.0

Release 3.2.0 added a new feature:

• added hdf5 read/write by @JoschD in https://github.com/pylhc/tfs/pull/101
• fully dropped Python 3.6 support

Full Changelog: https://github.com/pylhc/tfs/compare/3.1.0...3.2.0

- Python
Published by JoschD about 4 years ago

tfs-pandas - Release 3.1.0

Release 3.1.0 is a change of behavior.

• Fixed:

  • Removed dependency on deprecated numpy.str in favor of the built-in str.

• Changed:

  • When an error occurs while reading files and checking dataframes, the error information is now either logged with a debug level (instead of previously error) or simply held into the raised exception.

- Python
Published by fsoubelet over 4 years ago

tfs-pandas - Release 3.0.2

Release 3.0.2 is a patch release.

• Fixed:
  • The string representation of empty headers used to wrongly print None, and will now correctly be an empty string.

- Python
Published by fsoubelet over 4 years ago

tfs-pandas - Release 3.0.1

Release 3.0.1 is a patch release.

• Fixed:
  • Merging functionality from TfsDataFrame.append, TfsDataFrame.join, TfsDataFrame.merge and tfs.concat do not crash anymore when encountering a pandas.DataFrame (or more for tfs.concat) in their input. Signatures have been updated and tests were added for this behavior.

- Python
Published by fsoubelet over 4 years ago

tfs-pandas - Release 3.0.0

Release 3.0.0: Breaking changes and long-term bug fix.

A long-standing issue where merging functionality used on TfsDataFrame (through .merge or pandas.concat for instance) would cause them to be cast back to pandas.DataFrame and lose their headers has been patched.

• Breaking changes:

  • The internal API has been reworked for clarity and consistency. Note that anyone previously using the high-level exports tfs.read, tfs.write and tfs.TfsDataFrame will not be affected.

• Added:

  • The TfsDataFrame class now has new .append, .join and .merge methods wrapping the inherited methods of the same name and fixing the aforementioned issue.
  • A tfs.frame.concat function, exported as tfs.concat, has been added to wrap pandas.concat and fix the aforementioned issue.
  • A tfs.frame.merge_headers function has been added.
  • Top level exports are now: tfs.TfsDataFrame, tfs.read, tfs.write and tfs.concat.

• Changes:

  • The tfs.frame.validate function is now a public-facing documented API and may be used stably.
  • The write_tfs function now appends an EOL (\n) at the end of the file when writing out for visual clarity and readability. This is a purely cosmetic and does not change functionality / compatibility of the files.
  • Documentation and README have been updated and cleared up.

Please do refer to the documentation for the use of the new merging functionality to be aware of caveats, especially when merging headers.

- Python
Published by fsoubelet over 4 years ago

tfs-pandas - Release 2.1.0

Release 2.1.0 is a QoL change, bringing much faster loading of TFS files for users.

• Changes:
  • The parsing in read_tfs has been reworked to make use of pandas's C engine, resulting in drastic performance improvements when loading files. No functionality was lost or changed.

- Python
Published by fsoubelet almost 5 years ago

tfs-pandas - Release 2.0.3

• Fixed:

  • Took care of a numpy deprecation warning when using np.str, which should not appear anymore for users.

• Changes:

  • Prior to version 2.0.3, reading and writing would raise a TfsFormatError in case of non-unique indices or columns. From now on, this behavior is an option in read_tfs and write_tfscalled non_unique_behavior which by default is set to log a warning. If explicitely asked by the user, the failed check will raise a TfsFormatError.

- Python
Published by fsoubelet almost 5 years ago

tfs-pandas - Release 2.0.2

• Fixed:
  • Proper error on non-string columns
  • Writing numeric-only mixed type dataframes bug

- Python
Published by JoschD over 5 years ago

tfs-pandas - Release 2.0.1

• Fixed:
  • No longer warns on MAD-X styled string column types (%[num]s).
  • Documentation is up-to-date, and plays nicely with Sphinx's parsing.

- Python
Published by fsoubelet over 5 years ago

tfs-pandas - v2.0.0

Release 2.0.0: Breaking changes, fixes and more

Breaking changes: - FixedColumn, FixedColumnCollection and FixedTfs have been removed from the package - Objects are not converted to strings upon read anymore, and will raise an error - Minimum pandas version is 1.0

Fixes: - No longer writes an empty line to file in case of empty headers - "Planed" dataframes capitalize plane key attributes to be consistent with other pylhc packages, however they can be accessed with and without capitalizing your query

Changes: - Minimum required numpy version is now 1.19 - Full tests across supported Python versions and operating systems have been implemented - Significant digits function can return the result as floats if ask to

Miscellaneous: - Removed unused test dependencies - Removed setup_requires as it is deprecated - Reworked setup.py to be consistent with that of pylhc/omc3 - Improved testing and test coverage - Improved logging - Type hinting, docstrings and formatting

- Python
Published by fsoubelet over 5 years ago

tfs-pandas -

- Python
Published by JoschD about 6 years ago

tfs-pandas -

- Python
Published by JoschD about 6 years ago

tfs-pandas -

- Python
Published by JoschD almost 7 years ago

tfs-pandas -

- Python
Published by JoschD almost 7 years ago

tfs-pandas -

- Python
Published by JoschD almost 7 years ago