Releases | Open Source Science

pysat - v3.2.2

pysat 3.2.2 is a bug fix and maintenance release. Primarily, it fixes an issue with running the unit tests from the pypi-installed version elsewhere in the PyHC ecosystem. It also updates some documentation issues.

[3.2.2] - 2025-03-20

Bug Fix
- Include test data in tar.gz so unit test suite can be run from a pip-install
Maintenance
- Update .gitignore
- Update zenodo affiliations and references
- Clarified controlled information distribution statement
- Updated Ops tests to new lower limit of Python 3.9 and Ubuntu 22.04

- Python
Published by jklenzing 11 months ago

pysat - v3.2.1

pysat 3.2.1 is a bug fix and maintenance release. In particular, it fixes the unit test suite which applies throughout the ecosystem, and it fixes the file frequency calculations for instrument objects. This was causing an issue where pysatCDAAC would not import if any files were previously downloaded.

Changes

Bug Fix
- Fixed a bug for identifying clean warnings to test
- Imposed minimum frequency of 1D on Instrument.bounds to enable data loading for instruments with inter-file time intervals of less than one day
Maintenance
- Update usage of dims to be consistent with future versions of xarray
- Update frequency strings for pandas
- Update usage of getitem for pds.Series
- Updates usage of dt.datetime.utcnow() to dt.datetime.now(dt.timezone.utc)
- Implement SPEC0000 testing limits for pandas, scipy, xarray.
- Update pip rc install workflow to test against multiple python versions
- Implement coveralls app in GitHub Actions
- Updated deprecated useage of step.delta to pds.Timedelta(step)
- Updated rationale and usage for export_pysat_info in docstrings.
- Update lower limits of core dependencies based on operational tests

- Python
Published by jklenzing over 1 year ago

pysat - v3.2.0

The Python Satellite Data Analysis Toolkit (pysat) v3.2.0 adds multiple new features for data support, including custom data set concatenation and improved metadata support.

New Features
- Added tests for warnings, logging messages, and errors in the Instrument clean method.
- Added Instrument loading test with padding and for times without data.
- Allow Instruments to define custom concat_data methods.
- Added include kwarg to Instrument.concat_data to expand allowed inputs.
- Added data kwarg to the Instrument class __getitem__ method and reduced memory usage in the load method.
- Added a hidden method the Instrument class _get_epoch_name_from_data to reduce code duplication.
- Added __delitem__ to Meta and drop to MetaHeader and MetaLabels classes.
- Modified Meta to allow MetaHeader attribute access directly from Meta.
- Expanded Meta.drop to traverse attached MetaLabel and MetaHeader data.
- Added __delitem__ and drop to Instrument and Constellation classes.
- Added options to customize pysat_ndtesting instrument with sample rate, shift in time.
- Added orbit number to pysat_ndtesting.
- Added the overwrite kwarg to utils.registry.register_by_module.
- Added unit tests for all functions in utils.files.
- Reduced code duplication in the utils.files.parse_fixed_width_filenames and utils.files.parse_delimited_filenames functions
- Added ability to set Meta data using meta['data_var', 'label'] = value structure.
- Added test for loading multiple days of data.
- Added user-friendly warning when trying to load data when there are no files at all to load. Situation currently raises an IndexError.
- Expanded eval_warnings to support testing against multiple warning types.
Bug Fix
- Fixed utils.files.parse_fixed_width_filenames output for empty file list
- Updated the parsing functions in utils.files to consider type specifiers when identifying appropriate files in a directory
Maintenance
- Update link redirects in docs.
- Improved Instrument ValueError messages.
- Updated Constellation.to_inst method definition of coords, using dims to combine common dimensions instead.
- Implement pyproject to manage metadata
- Updated docstring references to pysat.utils.files in other modules.
- Remove Sphinx cap
- Update usage of whitespace and if statements (E275)
- Remove hacking cap
- Removed deprecated pysat_testing2d instrument
- Removed deprecated meta children info
- Removed deprecated pysat_testing_xarray instrument
- Removed deprecated pysat_testing2d_xarray instrument
- Removed deprecated instrument_test_class
- Removed deprecated malformed_index kwarg in test instrumennts
- Removed deprecated convert_timestamp_to_datetime function
- Removed deprecated _test_download_travis flag
- Removed deprecated freq kwarg from download
- Removed deprecated use_header kwarg from load and changed default behaviour to use_header=True
- Use temporary directories for files created during test_utils.py
- Updated code file headers to be consistent and include NRL pub release
- Added acknowledgements.md which includes full institutional funding list
- Removed deprecated labels kwarg for pysat.Instrument()
- Removed deprecated utils.load_netcdf4 method
- Removed deprecated _filter_netcdf4_metadata method
- Removed deprecated usage of None for tag and inst_id
- Removed deprecated kwarg behaviour for 'fname' in to_netCDF4
- Added verion cap for sphinxrtdtheme
- Used line specific noqa statements for imports
- Add _new_tests flag for packages to ignore select new tests
- Add CI testing for python 3.12

- Python
Published by jklenzing almost 2 years ago

pysat - v3.1.0

The Python Satellite Data Analysis Toolkit (pysat) v3.1.0 is a minor release with significant improvements in the base classes and data I/O. This includes improvements in Constellations, data stored in the xarray Dataset format, and saving Instrument data in netCDF files.

[3.1.0] - 2023-05-31

New Features
- Added a utility to ensure two xarray Datasets can be concatonated, and incorporated this utility into the Instrument file loading
- Added unit tests for different file cadences in the Instrument class
- Added to_inst method to the Constellation class
- Added export_pysat_info kwarg to to_netcdf routines to select whether pysat instrument info is written to files
- Added Constellation class examples to the docs tutorial
- Added links to the project standards repository to the docs
- Improved formatting of custom kwargs when running print on an instrument
- Added a core utility to update fill values consistently in the data and metadata
- Adapted check_and_make_path to treat an empty path as the current dir
- Added meta_kwargs attribute and kwarg to Instrument, allowing full custom specification of the Meta class on instantiation
- Expanded MetaLabels type defaults for 'maxval', 'minval', and 'fill_val' to include more common data types
- Added data_types input to Meta and certain MetaLabels methods, allowing the default values to be set to the specified data type when multiple types are allowed, ensure these are updated when adding new data to an Instrument
- Added _update_label_types to MetaLabels, expanding the Python float/int types to include all numpy float/int types
- Added strict_dim_check for loading xarray Datasets through netCDF
- Added combine_by_coords kwarg to io.load_netcdf for use on multi-file xarray Datasets
Deprecations
- Deprecated the Instrument kwarg labels in favor of meta_kwargs and replaced the meta_labels attribute with the meta_kwargs attribute
- Deprecated the labels keyword arg in favor of meta_kwargs in the netCDF I/O functions and Instrument sub-module
- Deprecated the malformed_index kwarg in the test instruments. This is replaced by non_monotonic_index and non_unique_index
Bug Fix
- Allow pysat.instruments.methods.general.list_files to handle file cadences other than daily or monthly
- Allow equality assessments if optional kwargs are used in Instrument
- Fixed an issue with setting single variables in xarray coords (#988)
- Fixed pysat.Instrument.bounds to handle all input types for step and width regardless of start and stop time. Also fixed seasonal bounds specified using file names.
- Fixed pysat.utils.io.apply_table_translation_to_file check for duplicates in the meta translation table
- Fixed an issue when passing dates through loadremotefiles (#1022)
- Fixed a bug where data may not have any times, but still not be empty
- Fixed a bug where metadata with values of None are assigned as useful attributes when attaching metadata to xarray objects
- Fixed a bug where a multifileday non-monotonic xarray index failed to merge datasets (#1005)
- Fixed a bug in testing for setting multiple optional load kwargs (#1097)
- Fixed a bug when setting xarray data as a tuple
- Fixed a bug when loading constellations for partially empty instrument lists
- Fixed a bug when cleaning up temporary directories on windows during testing
- Fixed a bug in Instrument loading with a pad, where RangeIndex slicing no longer works on an empty series
Maintenance
- Added roadmap to readthedocs
- Improved the documentation in pysat.utils.files
- Clarified documentation and tests for name slicing support in pandas
- Clarified documentation for adding new instruments
- Fixed broken links in docs
- Updated docstring header underline lengths and addressed documentation build errors and warnings
- Additional unit tests for data padding when a data index is non-monotonic.
- Deprecated the malformed_index kwarg in the test instruments. This is replaced by non_monotonic_index and non_unique_index
- Set the instruments.pysat_testing tag='no_download' to return an empty pandas DataFrame for load
- Added constellations.testing_partial, which loads a partially empty constellation dataset
- Reduced default num_samples for constellation test objects
- Improved consistency in metadata for test instruments

- Python
Published by jklenzing over 2 years ago

pysat - v3.0.6

The Python Satellite Data Analysis Toolkit (pysat) v3.0.6 is a bug fix release to improve compatibility with Python 3.6 and appropriate versions of pandas and xarray.

[3.0.6] - 2022-12-21

Bug Fix
- Ensure pysat tests do not store temporary directory to file
- Updated links for supported instruments in documentation
Deprecations
- pysat_testing2d_xarray instrument deprecated and replaced by pysat_ndtesting
Maintenance
- Updated pytest syntax to remove nose backwards-compatibility
- Fixed NEP29 compliance
- Updated Github Actions versions
- Improved docstrings
- Fixed a bug where a deprecation warning was thrown incorrectly
- Add pysatModels to pysat instrument count
- Update contributing document
- Added support for testing with python 3.6, updated reported minimum supported Python version to 3.6

- Python
Published by rstoneback about 3 years ago

pysat - v3.0.5

The Python Satellite Data Analysis Toolkit (pysat) v3.0.5 is a bug fix release, including improved compatibility with pandas v1.5.

[3.0.5] - 2022-10-07

Maintenance
- Update usage of logger throughout code.
- Update NEP29 minimum CI numpy version to 1.20
Bug Fix
- Updated meta.py so that internal assignment of None to children is not converted to NaN when using pandas>=1.5.0

- Python
Published by rstoneback over 3 years ago

pysat - v3.0.4

The Python Satellite Data Analysis Toolkit (pysat) v3.0.4 is a bug fix release that improves compatibility with xarray when accessing data via slices.

Changelog

Maintenance
- Removed version cap on xarray
Bug Fix
- Improved compatibility with xarray 2022.06 when accessing data via slices
- Fixed a bug for fractional seconds in methods.testing.generate_times.

- Python
Published by rstoneback over 3 years ago

pysat - v3.0.3

The Python Satellite Data Analysis Toolkit (pysat) v3.0.3 is a maintenance release that includes a temporary version limit on xarray due to changes in indexing in v2022.06.0.

Changelog

Maintenance
- Added temporary version limit to xarray due to change in indexing

- Python
Published by rstoneback over 3 years ago

pysat - v3.0.2

The Python Satellite Data Analysis Toolkit (pysat) v3.0.2 is primarily intended as a bugfix and maintenance release. However, this release does include expanded support for writing and reading netCDF4 files for user customizable file standards.

Changelog

New Features
- Added the property empty_partial to the Constellation class
- Added the option to apply custom functions at the Constellation or Instrument level within the Constellation class
- Added option to load Constellation for registered Instruments using lists of platforms, names, tags, and/or inst_ids, which are new attributes
- Added hidden Constellation methods to determine unique attribute elements and set Instrument attributes across all instruments
- Added hidden Constellation method to sequentially call Instrument methods
- Extended Constellation unit tests
- Added more quality checks to the Constellation initialization
- Standardized Instrument instantiation to always define inst_module
- Extended testing options for pysat.utils.testing functions
- Added start_time keyword for test instruments
- Added max_latitude keyword for non-model test instruments
- Added _test_download_ci as a standard attribute for pysat.Instrument
- Added a testing model similar to TIEGCM to pysat.instruments.pysat_testmodel as tag='pressure_levels'.
- Added the capability to test loading with optional kwargs through _test_load_opt instrument attribute
- Added rename and add_epoch_metadata methods to the Meta class
- Added I/O utilities to the utility module, extracting existing methods from the Instrument class and core utility module.
- Added a warning evaluation and directory creation functions to pysat.utils.testing.
- Added the ability to only download new files if remote file listing capabilities are not available for the Instrument.
- Added kwargs for epoch units and origin in pysat.utils.io.load_netcdf.
- Vectorized Meta.var_case_name and Meta.attr_case_name to support list of str as input as well as str.
- Added a time function to calculate decimal year from datetime.
- Allow Instrument.rename to take a fuction or mapping dict as input, after adapting routine to use Meta.rename
- Added an update method and type evaluation method to MetaLabels.
- Added a new MetaHeader class for holding global metadata.
- Added a pysat Instrument for loading pysat-written NetCDF4 files.
- Allowed MetaLabels to be expanded through setting new Meta data values.
- Added support for user variables when parsing template filenames in pysat.utils.files.
- Improved robustness of parsing delimited files.
- Updated pysat.Instrument.pandas_format so that changes to setting are immediately reflected by changes to Instrument object.
- Added a test function for bad class/function/method input to reduce duplicate code and improve message test combliance
- Added support for filtering basic netCDF4 metadata when loading via pysat.utils.io.load_netcdf.
- Added support for user developed functions to filter metadata as it loaded from disk but before assignment to a pysat.Meta instance.
- Added meta.to_dict to support creating more compliant netCDF files.
- Added meta.default_to_netcdf_translation_table, default_from_netcdf_translation_table and apply_table_translation to pysat.utils.io to support improved compatability of pysat netcdf files without user code changes.
- Added drop_meta_labels kwarg to pysat.utils.io.load_netcdf to support easy removal of unwanted metadata during loading.
- Added support for meta_processor in pysat.utils.io.load_netcdf and pysat.utils.io.inst_to_netcdf enabling developers to easily modify metadata before it is loaded from/written to storage.
- Intermediate missing directories are now created as needed when writing files using pysat.utils.io.inst_to_netcdf.
- Condensed code present in both pandas and xarray into pysat.utils.io.meta_array_expander.
- Non-default Meta.labels are now retained by Instrument if defined by underlying support module.
- Made test data more consistent across pysat testing instruments.
- Added pysat.Instrument.vars_no_time function to return data variables excluding the main time index.
- Added support for additional metdata and other file handling options to general 'pysat', 'netcdf' instrument.
- Added keyword decode_times to flag using xarray or pysat processing of times when loading a file via pysat.utils.io.load_netcdf.
- Added pysat.utils.io.xarray_all_vars which returns a list of all variables, coordinates, and dimensions.
- Added more volume units to pysat.utils.scale_units
- Added assert_isinstance and assert_hasattr to pysat.utils.testing
Deprecations
- Removed freq as a standard kwarg for pysat.Instruments.download
- Deprecated _test_download_travis as a standard attribute for pysat.Instrument. The function is replaced by _test_download_ci
- Deprecated pysat.tests.instrument_test_class.initialize_test_inst_and_date, which is moved to pysat.tests.classes.cls_instrument_library.
- Deprecated pysat.tests.instrument_test_class.InstTestClass, which is replaced by pysat.tests.classes.cls_instrument_library.InstLibTests.
- Deprecated pysat.Instrument._filter_netcdf4_metadata and replaced it with pysat.utils.io.filter_netcdf4_metadata.
- Changed fname from a kwarg to an arg in pysat.Instruments.to_netcdf4
- Deprecated support for 2D pandas datasets
- Deprecated pysat.instruments.pysat_testing_xarray
- Deprecated usage of higher order metadata
- Deprecated pysat.Instrument.generic_meta_translator which is replaced by pysat.utils.io.apply_table_translation_to_file.
- Deprecated the general instrument method, convert_timestamp_to_datetime, which is replaced by new functionality in load_netcdf.
Documentation
- Moved logo to 'docs\images'
- Improved consistency of headers throughout documentation
- Added more reference labels to documentation
- Implemented use of Python formatting in documentation examples
- Fixed bad section labels in docstrings
- Fixed noindex warning
- Fixed redirecting links
- Made changelog style and line length consistent
- Added a paragraph describing how to access Instrument sub-module docstrings from an instantiated Instrument in an interactive session
- Updated examples in docstrings
- Improved docstring style and consistency
- Added docstring linting via flake-docstrings and hacking to CI tests
- Added clarification to the tutorials
- Added limitations to the listify docstring
- Updated image link in README.md to display in more environments.
- Updated tag and inst_id keywords and docstrings to new standards
- Added example for version checking when writing tests for packages that rely on pysat
Bug Fix
- Fixed default MetaLabel specification in pysat.utils.load_netcdf4
- Fixed parse_delimited_filename output consistency and ability to handle leading and trailing non-parsed text in filenames (e.g., file extensions)
- Added decode_timedelta=False for loading xarray from netcdf4 (#823)
- Closed links to open files when loading data through xarray (#887)
- Fixed an issue in generating filenames for pysat.Instrument._iter_list
- Allow tag and inst_id to be specified as None (#892)
- Fixed a bug in pysat.utils.time.create_datetime_index (#906)
- Fixed a bug in pysat.utils.scale_units and ensured '/cc' compatibility.
- Added a warning if inst_module and platform/name are used to instantiate an instrument (#850). In case of this, inst_module takes priority.
- Fixed a bug when instantiating empty pysat.Instrument() (#851)
- Fixed IndexError encountered when loading a multi-day file by filename with no data for that day.
- Added logging output when metadata not assigned due to a lack of instrument data (#924)
- Fixed a bug where empty check for xarray instruments fail when time not present. (#922)
- Improved feedback when users try to set meta with an array.
- Fixed a bug that expected special treatment by Instrument.data type in utils.coords.update_longitude
- Fixed pysat_testmodel Instrument longitude range
- Corrected link to testing badge in README.md
- from_os now always returns a sorted Series.
- Moved function call for Instrument modules init function to end of pysat.Instrument instantiation to ensure Instrument is complete when passed to init.
- Refactored pysat.Instrument.generic_meta_translator to use pysat.Meta.to_dict and the latest metadata label standards.
- Prevent stale data paths stored by pysat from being reassigned if path is no longer in pysat.params['data_dirs'].
- Added missing metadata in testing instruments.
- Corrected identification of string data for xarray in pysat.Instrument._get_data_info.
Maintenance
- Added unit tests for deprecation warnings related to io_utils reorg.
- Added missing unit tests for pysat.utils.time
- Added missing unit tests for pysat.utils.file.parse_delimited_filename
- Streamlined unit tests for test_orbits
- Moved metadata generation for test instruments to methods.testing
- Added integration tests for test instrument kwargs
- Updated class declaration to be consistent with python 3 standards
- Update usage of caplog and capsys in unit tests
- Reorganized tests for the pysat.Instrument class into multiple files
- Updated unit tests for pysat.Instrument with pytest.mark.parametrize
- Update minimum numpy in CI tests to 1.19 following NEP29
- Made InstLibTests more portable to streamline user implementation of the standard end-to-end instrument tests.
- Moved test classes to pysat.tests.classes
- Reorganized the core test_instrument.py as a series of classes
- Separated MetaLabels tests from Meta test class
- Organized and reduced duplication in the Meta test class
- Added CI reports for supported data products
- Added a cap on coveralls to ensure success of continuous integration
- Updated tests in test_meta to search all warnings, not just the first
- Updated pandas syntax to be compatible with pandas 2.0 (pandas 1.4 deprecations)
- Cleaned up excess variables upon import
- Removed data_path check from pysat.instruments.methods.general.list_files
- Compatible with netCDF v1.6.0

- Python
Published by rstoneback over 3 years ago

pysat - v3.0.1

The Python Satellite Data Analysis Toolkit (pysat) v3.0.1 is primarily intended as a bugfix and maintenance release. Most significantly, pysat passes online unit tests on the Windows platform.

Changelog

New Features
- Added a routine for loading CSV files into a pandas DataFrame from a list of filenames.
- Added check for supported tag and inst_id at pysat.Instrument instantiation. (#784)
- Expanded Constellation utility by:
- Adding common properties: empty, index, date, today, yesterday, tomorrow, and variables (#764)
- Improving the printed output to inform user of the Constellation contents (#764)
- Added methods to download data and create a common time index. (#764)
- Added utils.listify, a function that returns a list of whatever is input. (#766)
- Added a warning for download requests that result in an empty date range.
Deprecations
Documentation
- Added missing information about date formatting to file listing docstrings.
- Added example for registering external instrument support (#837)
Bug Fix
- Changed pysat.Instruments.orbits iteration to return a copy of the Instrument rather than the Instrument itself. Provides robustness against garbage collection. (#770)
- Improved error messages for cases where slice of data may not exist (#761)
- Improved windows compatibility (#57, #790)
- Fixed Instrument.load bug that prevented use of instrument specific kwargs
- Added pytest as a package requirement (#819)
- Fixed pysat.utils.listify for empty list inputs (#830)
Maintenance
- Changed pysat.Instrument from treating all support functions as partial functions to retaining the original form provided by developer
- Migrates CI testing to Github Actions
- Improved maintenance of documentation build
- Added a check for use of reserved keywords at instantiation
- Tests compatible with pysatSpaceWeather 0.0.4 (#782)
- Improved pandas 1.3.0 compliance

- Python
Published by rstoneback over 4 years ago

pysat - v3.0.0

The Python Satellite Data Analysis Toolkit (pysat) v3.0.0 release is a substantive release intended to support pysat in operational environments as well as expand community oriented capabilities for space science data analysis. Core features have been retained within pysat while supporting instruments and analysis codes have been moved to individual packages. While v3.0.0 is not backwards compatible due to a reorganization of some functions and the data storage structure, the user interface is largely unchanged.

Changelog

[3.0.0] - 2021-04-01

New Features
- Added registry module for registering custom external instruments
- Added Meta.mutable flag to control attribute mutability
- Added MetaLabels class to manage metadata labeling
- Unit tests are now pytest compatible, use parametrize, and have improved messages when failures are encountered
- Added altitudes to test instruments
- New flags added to instruments to streamline unit testing: _test_download, _test_download_travis, _password_req
- strict_time_flag now defaults to True
- Use of start/stop notation in remotefilelist
- Added variable rename method to Instrument object (#91)
- Added support for loading more than one day/file (#56)
- Added support for iterating over a dataset a with a loaded data width and stepsize larger than a single day/file
- Added check for inconsistent inputs when loading data via Instrument
- Added file locking for thread-safe behavior (#304)
- Allow the Instrument object to be initialized with optional kwargs for any of the standard methods (not just load).
- Added support for 'cycle' in addition to 'version' and 'revision' for filename conventions.
- Integrated Custom class directly into Instrument to support mixed use of Constellation/Instrument objects in code (#540)
- Made underlying custom data structures visible (#529)
- Updated datamode method name to customattach for the Constellation object (#540)
- Added a display utility for discovering pysat Instrument data sets.
- Added testing utility functions.
- Added support for multiple pysat data directories
- Reorganized .pysat directory to store instrument information under .pysat/instruments and .pysat/instruments/archive
- Added pysat.params, a central location to store and modify pysat default parameters. File stored at .pysat/pysat_settings.json
- Added warn_empty_file_list to pysat.params to provide a warning if no Instrument files are found. Default is False.
- Updated default pysat directory organization template
- Added support for dual specification of Instruments to include in a Constellation
- Added function to transition pysat managed data files to a user provided template.
- Allow directory_format input to Instrument to be a function
- Adopted standard for bounds. stop is an inclusive bound, end is exclusive
- Added support for SLT calculations outside [0, 24)
- Added support for continuous SLT calculations when loading multiple days
- Instrument support functions now respond to local changes in Instrument.kwargs
- Added support for pysat.Instrument, Files, and Orbits equality comparisons
- Added .copy function to Instrument, Files, and Orbits classes
- Updated Parameters to ensure paths provided for pysat.params['data_dirs'] are created if they don't already exist
Deprecations
- Migraged instruments to pysatMadrigal, pysatNASA, pysatSpaceWeather, pysatIncubator, pysatModels, pysatCDAAC, and pysatMissions
- Migrated file methods to pysat.utils.files (#336)
- Renamed sat_id Instrument keyword argument to inst_id
- Removed ssnl
- Removed utils.stats
- Removed model_utils
- Removed deprecated pandas.Panel
- Import datetime class from datetime module, not pandas
- Import DataFrame and Series classes directly from pandas, not pysat
- Removed coords.scale_units
- Removed time.seasondaterange
- Moved methods.nasacdaweb.listfiles to methods.general
- DeprecationWarning for stricttimeflag only triggered if sloppy data is found
- Changed the custom function attachment input to allow keyword argument use when additional function input is required
- Removed python 2.7 syntax
- Removed utils.coords.geodetictogeocentric
- Removed utils.coords.geodetictogeocentric_horizontal
- Removed utils.coords.sphericaltocartesian
- Removed utils.coords.globaltolocal_cartesian
- Removed utils.coords.localhorizontaltoglobalgeo
- Addressed several Warnings raised by incorrect use of dependent packages
- Deprecated use of inst_id for number of simulated samples for test instruments
- Removed writing of custom Meta attributes when producing netCDF4 files
- Removed unneeded description.txt file, using README instead
- Changed pysat.instruments.methods.general.list_files kwarg fake_monthly_files_from_daily to file_cadence
- Changed name of Instrument method default to preprocess
- Removed pysat.data_dir. Information now at pysat.params['data_dirs'].
- Moved pysat.Instrument._filter_datetime_input to pysat.utils.time.filter_datetime_input
- Deprecated pysat.utils.set_data_dir
- Changed name kwarg in Constellation to const_module
- Removed unnecessary Instrument attribute labels
- Removed unnecessary Instrument kwargs
- Removed the Custom class, incorporating it into Instrument
- Removed deprecated calls to 'modify' type custom functions
Documentation
- Added info on how to register new instruments
- Fixed description of tag and inst_id behaviour in testing instruments
- Added a tutorial for developers of instrument libraries for pysat
- Updated instrument templates
- Added .zenodo.json file, to improve specification of authors in citation
- Improved __str__ and __repr__ functions for basic classes
- Improved docstring readability and consistency
- Added Travis-CI testing for the documentation
- Added a style guide for developers
- Updated loading data flow figure
- Scrubbed the documentation, ensuring examples, tutorials, and descriptions match the current implementation and code structure
Bug Fix
- Updated Instrument.concat_func to behave as described in the docstring
- Fixed custom instrument attribute persistence upon load
- Improved string handling robustness when writing netCDF4 files in Python 3
- Improved pandas 1.1.0 compatibility in tests
- Fixed coupling of twodigityearbreak keyword to underlying method in methods.general.listfiles
- Fixed additional file date range for monthly data with gaps
- Fixed custom Meta attributes removal when transferred to instrument (#615)
- Corrected iteration over Instrument within list comprehension
- Removed unused input arguments
- Corrects Instrument today, yesterday, and tomorrow methods by implementing datetime.datetime.utcnow
- Ensured pysat.Meta instances are immutable at Instrument instantiation
- Removed weak reference back to Instrument within Files class
- Fixed access of xarray data with more than one dimension (#471)
- Improved robustness of eval(inst.__repr__()) (#636)
- Fixed calc_solar_local_time for data sets with longitude coordinates
- Fixed .copy() when pysat.Instrument instantiated with inst_module (#728)
- Modified storage of Instrument.kwargs to include all methods so that eval(Instrument.__repr__()) works in more cases
- Modified storage of Instrument.kwargs to only include user supplied keywords
- Improved robustness when working with file dates that aren't centered on midnight
- Added basic valid path check to pysat.utils.files.check_and_make_path
Maintenance
- nose dependency removed from unit tests
- Specified dtype for empty pandas.Series for forward compatibility
- Removed wildcard imports, relative imports
- Include flake8 check as part of testing suites
- Improved unit testing coverage of instrument functions and instrument object
- Added tests for Instrument attributes acknowledgements and references
- Removed implicit conversion to integers in methods.general.converttimestampto_datetime
- Simplified internal logic in Instrument class
- Moved setup metadata to setup.cfg
- Improved instrument tests for files
- Used dt.timedelta instead of pds.DateOffSet where possible
- Reduced code duplication throughout package
- Reduced unused code snippets throughout
- Ensured download start time is used
- Condensed testing support functions into methods/testing.py
- Fixed a bug with usage of numpy.dtype for numpy 1.20 compatibility
- Updated usage of pds.index.tonativetypes() to pds.index.astype(str) for pandas 2.0 compatibility (#737)
- Check type as float rather than np.float for future numpy compatibility (#740)
- Verified usage of inst.loc[slice, keyword] will continue to work in pandas 2.0 (#738)

- Python
Published by rstoneback almost 5 years ago

pysat - v2.3.0

The Python Satellite Data Analysis Toolkit (pysat) v2.3.0 release is intended to be the last release of the v2.x series and the last release to include support for Python 2.x. pysat v2.3.0 primarily features DeprecationWarnings for pysat functions that are changing as part of the 3.0.0 release.

[2.3.0] - 2021-04-01

Allow use of new Instrument kwarg, inst_id (replaces sat_id)
Allow limited use of new listfiles kwarg, `filecadance`
Added support for pysat.params['data_dirs'], the replacement for pysat.data_dir in pysat 3.0.0
Deprecation warnings added to:
- Instrument class (old meta labels, sat_id, default, multi_file_day, manual_org, and _filter_datetime_input, data_dir)
- pysat.instruments.methods.general.listfiles kwarg `fakemontlyfilesfrom_daily`
- pysat.instruments.methods.testing.generatetimes kwarg `satid`
- Constellation class kwarg name
- Custom class
- functions from _files class
- functions from ssnl module
- Instrument modules:
  - the pysatCDAAC: cosmic_gps
  - the pysatIncubator: champstar, demeteriap, superdarngrdex, supermagmagnetometer
  - the pysatMadrigal: jroisr, dmspivm
  - the pysatSpaceWeather: swdst, swf107, sw_kp
  - the pysatModels: ucar_tiegcm
  - the pysatNASA: cnofsivm, cnofsplp, cnofsvefi, de2lang, de2nacs, de2rpa, de2wats, iconeuv, iconfuv, iconivm, iconmight, issfpmu, omnihro, rocsat1ivm, sportivm, timedsaber, timed_see
- The usage of a numeric string for sat_id to specify number of points in test instruments
- SpaceWeather, Incubator (DEMETER), Madrigal, CDAWeb, and ICON instrument methods
- The DE2 and ICON Constellation objects
- pysat.utils.setdatadir
Documentation
- Updated docstrings with deprecation notes
Bug Fix
- Closes files after download in NASA CDAWeb methods
- Updated the ICON constellation object to reflect new instrument standards
Added version cap for numpy

- Python
Published by rstoneback almost 5 years ago

pysat - v2.2.2 Release

The v2.2.2 release is primarily focused on bug fixes and support for active missions.

Changelog

New Features
- netCDF4 files produced using to_netcdf4() now have an unlimited time dimension
Documentation
- Updated guidance on numpy version for installation
Bug Fix
- Updated madrigal methods to simplify compound data types and enable creation of netCDF4 files using self.to_netcdf4().

Note

The pysat v2.2.x series is the last intended series of releases to include support for both Python 2 and 3. pysat v3.0 will support Python 3.6+ only.

pysat v3.0 breaks out a number of features and instruments into individual packages to create a pysat ecosystem. Core functionality will remain within the pysat package. To minimize the disruption caused by these changes, all areas with known updates are marked with appropriate DeprecationWarnings along with a note in the docstring. A final v2.2.x release, v.2.2.deprecation may be released with v3.0.

- Python
Published by rstoneback about 5 years ago

pysat - v2.2.1 Release

The pysat v2.2.x series is the last intended public release that includes support for both Python 2 and 3. pysat v3.0 will support Python 3.6+ only. pysat v3.0 further includes some breaking changes to the pysat interface. To minimize the disruption caused by these changes, v2.2.x supports writing forward compatible code with v3.0. All areas due for changes are marked with appropriate DeprecationWarnings along with a note in the docstring.

[2.2.1]

Documentation
- Improved organization of documentation on ReadTheDocs
Bug Fix
- Adopted .readthedocs.yml to restore online documentation on ReadTheDocs
- Modified MANIFEST.in to include pysat_testing instruments
- Rename default branch as main

- Python
Published by rstoneback over 5 years ago

pysat - Summer 2020 Update

[2.2.0] - 2020-07-24

New Features
- Decreased time to load COSMIC GPS data by about 50%
- Added DE2 Langmuir Probe, NACS, RPA, and WATS instruments
- Updated test_files.py to be pytest compatible
- Added check to ensure non-pysat keywords supplied at instantiation are supported by underlying data set methods
- Updates to instrument testing objects for consistency
- Changed madrigal methods to use madrigalWeb as a module rather than calling it externally
- Added warning when FillValue metadata could lead to unexpected results when writing a netCDF4 file
- Use conda to manage Travis CI test environment
- Update ICON instrument file structure
- Added NaN filter for metadata when writing netCDF4 files
- Test instruments now part of compiled package for development elsewhere
- Reviewed and improved documentation
- Custom instrument keywords and defaults are now always found in inst.kwargs
- Added support for ~ and $ variables when setting pysat data dir
- Added custom.attach to make transitions to v3.0 easier
Deprecation Warning
- custom.add will be renamed custom.attach in pysat 3.0.0
- Several functions in coords will be removed in pysat 3.0.0. These functions will move to pysatMadrigal
- geodetictogeocentric
- geodetictogeocentric_horizontal
- sphericaltocartesian
- globaltolocal_cartesian
- localhorizontaltoglobalgeo
- methods.nasacdaweb.listfiles will move to methods.general.list_files in pysat 3.0.0.
Documentation
- Fixed description of tag and sat_id behaviour in testing instruments
- Added discussion of github install, develop branches, and reqs to docs
Bug Fix
- _files._attach_files now checks for an empty file list before appending
- Fixed boolean logic when checking for start and stop dates in _instrument.download
- Fixed loading of COSMIC atmPrf files
- Fixed feedback from COSMIC GPS when data not found on remote server
- Fixed deprecation warning for pysat.utils.coords.scale_units
- Fixed a bug when trying to combine empty f107 lists
- Fixed a bug where remote_file_list would fail for some instruments.
- Made import of methods more robust
- Fixed SettingWithCopyWarning in cnofs_ivm cleaning routine
- Fixed cosmic load method definition to include altitude_bin
- Fixed pysattesting method definition to include manglefile_dates keyword
- Added small time offsets (< 1s) to ensure COSMIC files and data have unique times
- Updates to Travis CI environment
- Removed inplace use in xarray assign function, which is no longer allowed
- Removed old code and incorrect comments from F10.7 support
- Updated use of numpy.linspace to be compatible with numpy 1.18.
- Fixed output of orbit_info during print(inst)
- Fixed a bug when requesting non-existent files from CDAWeb (#426)
- Improved compatibility of parsedelimitedfilenames (#439)
- Fixed bug assigning dates to COSMIC files
- Fixed bug limiting local time orbit breakdowns for instruments much slower than 1 Hz

- Python
Published by rstoneback over 5 years ago

pysat - Fall Update

[2.1.0] - 2019-11-18

New Features
- Added new velocity format options to utils.coords.scale_units
- Improved failure messages for utils.coords.scale_units
- Added some tests for model_utils
- Added option to to_netCDF that names variables in the written file based upon the strings in the Instrument.meta object
- Improved compatibility with NASA ICON's file standards
- Improved file downloading for Kp
- Added keyword ignoreemptyfiles to pysat.Instrument and Files objects to filter out empty files from the stored file list
- Added slice and list ability to meta
- Updated cleaning routines for C/NOFS IVM
- Added S4 scintillation data to the cosmic-gps instrument
- pysat no longer creates a default data directory. User must specify location.
- User set custom attributes are transparently stored within Meta object and are available via both Instrument and Meta.
- Improved robustness of required library specification across multiple platforms
Code Restructure
- Move computational_form to ssnl, old version is deprecated
- Move scale_units to utils._core, old version is deprecated
- Replace season_date_range with create_date_range, old version is deprecated
- Added deprecation warnings to stat functions
- Added deprecation warnings to ssnl and model_utils
- Removed pysat_sgp4 instrument
- Added cleaning steps to the C/NOFS IVM ion fraction data
Bug fix
- Fixed implementation of utils routines in modelutils and jroisr
- Fixed error catching bug in model_utils
- Updated Instrument.concat_data for consistency across pandas and xarray. Includes support for user provided keywords.
- Fixed error introduced by upstream change in NOAA F10.7 file format
- Fixed bugs in DEMETER file reading introduced by changes in codecs
- Fixed issue with data access via Instrument object using time and name slicing and xarray. Added unit test.
- Updated travis.yml to work under pysat organization
- Added missing requirements (matplotlib, netCDF4)
- Fixed a bug when trying to combine empty kp lists
- Updated travis.yml to work with python 2.7.15 and beyond
- Unit tests reload pysattestingxarray for xarray tests
- Updated setup.py to not overwrite default open command from codecs
- Updated Travis CI settings to allow forks to run tests on local travis accounts
- Fixed keep method to be case insensitive
- Fixed a bug with COSMIC GPS downloads
- Fixed selection bugs in the DEMETER IAP, CNOFS IVM, and model_utils routines
- Updated URL link in setup.py
Documentation
- Added info on how to cite the code and package.
- Updated instrument docstring
- Corrected pysat.Instrument examples using COSMIC

- Python
Published by rstoneback over 6 years ago

pysat - Building a Foundation

New Features
- pysatData directory created in user's home directory if no directory specified
- Added preliminary support for xarray to the instrument object
- Support for today, tomorrow, and yesterday as datetime objects
- Added model_utils, featuring preliminary support for data-model comparison
- Added support for 1d median in seasonal averages
- Added routine to convert from kp to Ap
- Added pyglow integration support for python 3.x
- Added option to check that loaded data has a unique and monotonic time index. Will be enforced in a future version.
- Refactored data access through the Instrument object and expanded testing.
- Added .empty attribute to Instrument object, True when no data loaded.
- Added .index access mechanism to Instrument object, providing consistent access to the pandas DatetimeIndex associated with loaded data.
- Added mechanism to return a list of loaded variables, .variables.
- Added Instrument method to concat input data with data already loaded into Instrument object.
- Updated format of printed dates to day month name and year, 01 January 2001.
- Added Instrument property .date, returns date of loaded data.
- Added downloadupdatedfiles, Instrument method that downloads any remote data not currently on the local machine.
- Added remotedaterange, an Instrument method that returns first and last date for remote data.
- Download method now defaults to most recent data (near now).
- Improves input handling for datetime parameters that are more precise than just year, month, and day, where appropriate
- Added merging routines to allow combination of measured and forecasted Kp and F10.7 indexes into a single instrument object
- Files class internally refactored to improve robustness.
- Added feature to handle delimited filenames, in addition to fixed_width names.
- Exposed methods to allow users to more easily benefit from features in Files. Used to support remotefilelists and make data downloads more convenient.
- Expanded testing with Files.
- Updated keyword names to be more complete. 'sec' to 'second', etc.
- Updated Files access mechanisms to remove deprecated calls and improve robustness.
Code restructure
- Moved instrument templates and methods to subdirectories
- Moved utils into multiple subdirectories to aid with organization
Instrument Updates
- NASA CDAWeb download now uses https protocol rather than FTP
- _instrument.py supports xarray
- Support for listing files from remote server
- COSMIC RO data unified into single instrument object
- Added support for DEMETER IAP
- Added support for DMSP IVM Level 2 data. Uses OpenMadrigal.
- Added routines to update DMSP ephemeris and drifts
- Added warnings to instruments without download support
- Added preliminary support for ICON FUV and MIGHTI
- Added support for Jicamarca Radio Observatory ISR
- Added support for F10.7 and more Kp forecast products
- Added instrument templates for Madrigal, CDAWeb, and netcdf_pandas
- Added support for TIMED SABER
- Added support for UCAR TIEGCM
- OMNI HRO instrument now uses CDAWeb methods
- Switched download methods for CDAWeb and COSMIC data to use requests
- Added Madrigal methods
- Removed support for SuperDARN and SuperMAG downloads while server changes are sorted out
Updates to travis configuration
- Tests run for python 2.7 and 3.7
- Added display port to test plots
Updates to community docs
- Added Issue templates
- Added Pull Request Template
- Added note for PR to be made to develop, not master
Style updates throughout
- Consistent documentation for docstrings and instruments
- Cleaned up commented code lines
- PEP8 scrub
Documentation
- Added FAQ section
- Added "powered by pysat" logo
- Updated supported instruments
Unit Test Updates
- Dropped instrument templates from coverage
- Added multiple output options for pysat_testing object to aid with constellation tests. Removed old constellation test objects.
- Added test data for space weather indices to speed up testing
- Cyclic data for test instruments now generated from single test method
- test objects for xarray added
- Added test for parsed delimited files
- Removed ftp downloads from travis tests, still will run locally
Bug fixes
- pandas.ix notation replaced with pandas.loc and pandas.iloc throughout
- Fixed a bug that forced user into interactive mode in ssnl.plot
- Bug fixes and cleanup in demo codes
- Fix for orbit iteration when less than one orbit of data exists. Fix now covers multiple days with less than one orbit.
- Fixed a bug in python 3.7 caused by change in behaviour of StopIteration (#207)
- Update to use of len on xarray to handle new behaviour (#130)
- Updated import of reload statements now that python 3.3 has reached end of life
- Updated deprecated behaviour of get_duplicates, .apply, and .to_csv when using pandas
- Fixed bug in assigning units to metadata (#162)
- Fixed timing bug introduced by reading only the first date/data pair from each line in the 45-day file data blocks

- Python
Published by rstoneback over 6 years ago

pysat - SuperMAG and more

SuperMAG support added
Increased data access robustness when using integer indexing
Added template for supporting netCDF4 based instruments (netcdf4_pandas)
Added support for MSIS within the pysat satellite simulation (based on sgp4)
Added plotting routine to sgp4
Initial support for the upcoming NASA/INPE SPORT Ion Velocity Meter (IVM)
Fixed bug triggered when invoking multifileday option in Instrument object

- Python
Published by rstoneback over 7 years ago

pysat - Constellation Support, Satellite Simulations, and DMSP IVM Support

Initial support for Constellation objects, which allows operations and analysis on mixed groups of Instrument objects. Developed by UT Dallas senior undergraduate computer science students (UTDesign 2018).
- Bug fixes when iterating by file
- Added pysat_sgp4, a Two Line Element based satellite orbit propagator that is coupled with ionosphere, thermosphere, and geomagnetic models. Supports projecting these quantities onto the relevant spacecraft frame to create signals suitable for satellite data simulation and testing. Routine uses pyglow, pysatMagVect, sgp4, and pyEphem.
- Further along the road toward windows compatibility
- Fixed orbit number reporting in orbits.current
- Added support for Defense Meteorological Satellite Program (DMSP) Ion Velocity Meter (IVM) data. Downloads from the Madrigal database (https://openmadrigal.org)
- Added support for both sat_id and tag variations within filenames in the NASA CDAWeb template
- Updated docummentation covering requirements for adding new instruments to pysat

- Python
Published by rstoneback over 7 years ago

pysat - JGR-Space Physics Publication Update

Updated CNOFS IVM file support to the latest on NASA's CDAWeb
Improved robustness and speed of metadata code
Improved compatibility with SPDF ISTP/IACG standard for netCDF scientific files

- Python
Published by rstoneback almost 8 years ago

pysat - JGR-Space Physics Publication Release

PYSAT: Python Satellite Data Analysis Toolkit has been published in the Journal of Geophysical Research - Space Physics (DOI: 10.1029/2018JA025297)

Changelog for v1.0

Improved consistency when handling higher order metadata
Improved translation of metadata within netCDF4 files to pysat standard
Added pysatCDF as package requirement
PEP8 upgrades throughout
Updated load_netCDF4 routine to support ICON EUV files natively
tonetCDF4 function updated to be consistent with loadnetCDF4
Meta object upgraded to handle more attributes by default
Meta object has been upgraded to preserve case of variable and attribute names
Metadata access is case insensitive for ease of use
Changes to unitslabel or namelabel are automatically applied to underlying metadata
Improved handling of custom units and name labels at Instrument level
Additional functions added to Meta object, attrs, keys, keysnD, hasattr, routines that return preserved case
Additional unit tests for Meta added
Reduced resources required for unit tests
Improved windows compatibility
Added more unit tests for seasonal averages
Added more simulated data types to pysat_testing2D
Added initial support for ICON EUV
Added initial support for ICON IVM
Added support for version/revision numbers in filenames within Files class constructor from_os

- Python
Published by rstoneback almost 8 years ago

pysat - ICON Release

Instrument Support - Moved routines to pysatCDF - Added ISS-FPMU - Updated NASA CDAWeb data distribution locations - Improved windows compatibility - Improved terminal listing of Pysat.Instrument properties - Fixed bug reading dates from COSMIC-2013 files - Added functions using OMNI data, clock angle and IMF steadiness - Added support for changing default names for 'units' and 'long_name'

Orbit Support - Improved compatibility with new numpy versions

File Support - Removed support for netCDF3 from tonetCDF4 function - Updated time storage, now using milliseconds - Added option for name of datetime index written to file - Improved metadata handling - Decreased write time by a factor up to 3 - Fixed deprecation warnings for todatetime - Added keyword to enable compression

- Python
Published by rstoneback almost 8 years ago

pysat -

Added summary information about Instrument object, available via print or command line feedback
Added empty attribute to Instrument object, True if there is no loaded data.
Updated pysat loading procedure to transfer extra attributes from the Meta object to Instrument object. Thus extra attributes contained in netCDF or similar files are now attached to the Instrument object when data is loaded.
Expanding case support in the Meta object. There are multiple file standards with different metadata conventions, this change supports better compatibility in the future.
SuperDARN code transitioned to latest Davitpy.
Expanded unit tests.

- Python
Published by rstoneback over 8 years ago

pysat - Unit tests abound

Unit testing was expanded. Issues related to iterating backwards via orbit have been fixed. The interaction of data padding features and multi file day data sets was corrected for edge cases. Some situations resulted in a loss of loaded data. Standards for instrument routines were updated to support testing. Orbit testing has been expanded to ensure reliable orbit determination with gappy data. Gaps range from less than an orbit to multi-day.