nctoolkit - v1.2.3

This is a minor release. It makes an internal change to make nctoolkit compatible with an upcoming version of pip.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson 11 months ago

nctoolkit - v1.2.2

This is a minor release that improves the internal performance of some operators.

Where appropriate, nctoolkit will now use CDO's parallel processing options to speed up some operations on single file datasets. Some methods, such as fill_na will now be faster when you setnc.options(cores=6) etc, with single file datasets.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson about 1 year ago

nctoolkit - v1.2.1

This is a minor release that enhances the bottom method.

You can now specify if you want bottom to extract the bottom level in the dataset, or if you want to extract the bottom-most level with non-missing values.

A couple of additional improvements have been made to nctoolkit internals.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson over 1 year ago

nctoolkit - v1.2.0

This release changes the default behaviour of to_dataframe. Bound type variables will now be ignored, as they are typically redundant once a dataset has been converted to a dataframe. For example a variable like lon_bnds will now be ignored. To retain the old behaviour use to_dataframe(drop_bnds=False).

There are additional improvements to internals that do not modify package usage.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson over 1 year ago

nctoolkit - v1.1.11

This is a minor release that makes improvements to pub_plot.

The pub_plot method will now display arrows bars on the colour bar when limits are supplied to make it clear values can exceed those shown in the colour bar.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson almost 2 years ago

nctoolkit -

Version 1.1.10

Quieter warnings on Linux. Improvements have been made to reduce the number of repetitive warnings on Linux.

The pub_plot method now lets you set the dpi in saved plots.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson about 2 years ago

nctoolkit - v1.1.8

This is a minor release that fixes a bug that caused errors when norm was set using pub_plot. Everything else should work as before.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson about 2 years ago

nctoolkit - v1.1.7

Minor bug fix release

This is a minor release that improves how pub_plot handles some arguments to provide more consistent behaviour.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson about 2 years ago

nctoolkit - v1.1.6

Adding the ability to create vertical level bounds

This release adds a new dataset method generate_level_bounds, which will generate vertical bounds from the vertical levels. This is useful in case this info is not provided in the original data and they need to be estimated for vertical averaging etc.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson over 2 years ago

nctoolkit - v1.1.5

This is a minor release that improves the package import speed.

Faster import speeds

Previously, the package tried to import geoviews at startup. This was a bottleneck, as gevoviews takes a long time to import. Geoviews is now imported when first used.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson over 2 years ago

nctoolkit - v1.1.3

This is a minor release with some improvements to temporal methods, such as tmean.

Enhanced temporal methods

You can now supply a window argument to statistical temporal methods. For example, if you wanted to calculate a weekly mean value, you could do the following: ds.tmean(window=7).

Scientific Software - Peer-reviewed - Python
Published by robertjwilson over 2 years ago

nctoolkit - v1.1.2

This is a minor release.

The subset method has been improved internally. Previously, if you ran, for example, ds.subset(month=1) and one or more of the files in a dataset had no data for January, an error was thrown. This has now been changed, so that a warning is thrown and the files are removed from the dataset.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson over 2 years ago

nctoolkit - v1.1.1

This is a minor release with some improvements to internals. Code written for the previous release will be fully compatible.

The annual_anomaly method is now able to run in parallel will multiple cores are defined using options.

The match_points method now has a quiet argument.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson over 2 years ago

nctoolkit - v1.1.0

This is a major release with some improvements and new methods.

This release requires that CDO version 2.0.5 or above is installed. Code written for the previous version of nctoolkit will be compatible.

The cost of maintaining support for CDO versions 1.9.10 and below was increasingly high compared with the declining user base for these versions, so support has been dropped.

A method for inverting vertical levels and latitude will be introduced: invert.

The pub_plot method will use slightly better colour scales for diverging scales.

Some improvements have been made to package internals.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson over 2 years ago

nctoolkit - v1.0.0

This is a major release with breaking changes.

A critical change is that accessing dataset attributes will force evaluation, i.e. ds.run() will occur if you access an attribute such as ds.times. This makes behaviour more consistent with what new users would expect.

This should not have a major impact on any coding workflows.

The cdo_command method has been changed so that it no longer checks the validity of the command before calling CDO. Checks can now be run using the check argument.

Some improvements have been made to method internals for pub_plot.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson almost 3 years ago

nctoolkit - v0.9.4

This is a minor release with no breaking changes.

The pub_plot method now accepts fig as an argument to make it easier to manipulate figures.

Improvements have been made to some internals.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson almost 3 years ago

nctoolkit - v0.9.3

This is a bug-fix release to ensure nctoolkit runs from scripts in macOS. This changes internal behaviour on macOS, so that multiprocess instead of multiprocessing is used for handling parallel processing.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson almost 3 years ago

nctoolkit - v0.9.2

This release contains a new method for producing high quality static plots. Yuri Artioli of Plymouth Marine Laboratory contributed the core code to this new pub_plot method.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson about 3 years ago

nctoolkit - v0.9.1

This is a quick release to make the package fully compatible with Pandas v2.0.0. Changes in the pandas api were causing some add/subtract etc. methods to fail for daily data. This is now fixed.

A couple of improvements have been made to internals.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson about 3 years ago

nctoolkit - v0.9.0

This is a major(ish) release with some breaking changes related to plotting.

On pypi, cartopy has been switched to an optional dependency because it was causing installation difficulties for some users. You can now do a "complete" installation using pip to get all optional dependencies::

$ pip install nctoolkit[complete]

This does not impact the conda version, which will behave as before.

Support is now available for Python 3.11.

File paths with spaces are now supported.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson about 3 years ago

nctoolkit - v0.8.7

This is a minor release that makes it easier to use nctoolkit on shared resources.

You can now specify who is using the imported version of nctoolkit using options.

So, if you did the following:

import nctoolkit as nc
nc.options(user = "george_costanza")

all temporary files generated will have "george_costanza" in the name. This makes it easier to identify files generated by you and others when you are sharing computational resources.

There are also some internal improvements in this release.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson over 3 years ago

nctoolkit - v0.8.6

This is a minor releases that tidies up some issues and has some method enhancements.

The regrid and to_latlon methods can now be more efficient for multi-file datasets where all files have the same grid. Previously, the methods identified the grids for all methods, which is obviously inefficient if they are all the same. You can now set the one_grid argument to True, which will result in the methods assuming all files have the same grid, and only the first file being checked.

There was an issue with multi-file datasets in parallel in Python 3.8 and 3.9. A confusing TypeError was being thrown due to signalling issues by multiprocessing. This gave the impression there was a problem with processing when there wasn’t one. This problem is now fixed.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson over 3 years ago

nctoolkit - v0.8.5

This is a minor release that deals with clean up issues on Jupyter notebooks. A change in a recent version of ipykernel was causing nctoolkit to not automatically remove temporary files on exit when using notebooks. This should now be fixed.

The annual_anomaly method now lets users temporally align the output, in the same way as other temporal methods such as roll_mean.

Some improvements have been made to internals for better warnings and errors.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson over 3 years ago

nctoolkit - v0.8.4

This is a minor update that improves the ability to handle missing values.

A new method is introduced for changing the fill value of missing values, set_fill.

Another method missing_as is introduced. This will be the opposite of as_missing. It will set missing values to a constant value.

Dataset contents will now show the fill value for variables. Furthermore, open_data will now check if the fill value is zero, which can cause problems for logical comparisons etc.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson over 3 years ago

nctoolkit - v0.8.3

This is a minor release. Improvements have been made to the way warnings from CDO are translated to nctoolkit warnings.

Methods remain the same as before.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson over 3 years ago

nctoolkit - v0.8.2

This release changes plotting so that it does not show coastlines by default.

Plotting with coastlines was causing plotting to crash on some systems due to issues with how nctoolkit's Python dependencies work with non-Python dependencies. These were not a problem with nctoolkit installations from conda, which will install non-Python dependencies, but some non-conda insttallations would no longer plot maps as a Python dependency could be incompatible with the non-Python dependencies on user systems.

If you want to plot the coastline, do the following

ds.plot(coast=True)

Scientific Software - Peer-reviewed - Python
Published by robertjwilson over 3 years ago

nctoolkit - v0.8.1

This is a minor release that modifies version pinning for a couple of dependencies.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson over 3 years ago

nctoolkit - v0.8.0

This is a major release that introduces some breaking changes.

The major improvement in this release will be to vertical methods. All vertical methods should now work with files with vertical axes that are either consistent or vary spatially. Before some methods only worked with z-levels, i.e. files with fixed vertical levels.

This change will result in a requirement that vertical_mean, vertical_interp andvertical_integration need users to specify whether the vertical levels are fixed spatially, using the fixed arg. As a result, calls to these methods without this argument will throw an error.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson over 3 years ago

nctoolkit - v0.7.6

This is a minor release that significantly simplifies basic arithmetic and logical operations.

Simple methods such as +, - etc. can now use standard python syntax.

For example, if you wanted to add 2 to a dataset you can now do the following:

ds.add(2)

as this instead

ds+2

The same goes for logical operators. You can do the following to identify if the values in a datset are below 2:

ds<2

whereas you previously had to do this:

`ds.compare("<2")

Note: because nctoolkit methods only modify datasets and do not return datasets, the following will not work:

ds1+ds2+2

Instead, you would need to do:

ds1+ds2 ds1+2

Scientific Software - Peer-reviewed - Python
Published by robertjwilson over 3 years ago

nctoolkit - v0.7.5

This release smoothes out issues caused by xarray being unable to read files. This caused some issues in methods where xarray is used as a backend. open_data will no longer through an error in when xarray cannot open the file (xarray is used to parse times), and check will provide a meaningful check.

No other changes have been made in this version and it is fully compatible with the previous release.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson over 3 years ago

nctoolkit - v0.7.4

Fast release that pins the matplotlib version. Fixes an issue that meant an nctoolkit dependency did not import, which broke nctoolkit.

Also features a couple of under-the-hood improvements to methods.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson over 3 years ago

nctoolkit - v0.7.3

This is a quick release that fixes an issue (caused by a dependency update) that results in multi-variable datasets not plotting on occasion.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson over 3 years ago

nctoolkit - v0.7.2

This is a quick release. It fixes a bug where global settings could not be set for all possible options.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson over 3 years ago

nctoolkit - v0.7.1

This is a major release with some breaking changes.

The deprecated select method has now been removed. Users should now use the subset method.

A progress bar will now display when processing large datasets. This will only show when nctoolkit thinks something will take a while. If you want to always show a progress bar for multi-file datasets, you can do this: nc.options(progress = 'on').

Scientific Software - Peer-reviewed - Python
Published by robertjwilson over 3 years ago

nctoolkit - v0.6.3

This is a minor release that includes a couple of enhancements to the match_points methods. They are now able to accept pressure in the point observation dataframe. You are also now able to generate new variables in match_points using kwargs, which will be sent to the assign method.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson over 3 years ago

nctoolkit - v0.6.2

This is a minor release with a couple of method upgrades.

The subset method now lets users subset hours. For example ds.subset(hour=1) will select the first hour of the day.

The match point methods have also been modified so that the dataframe sent to them does not have to strictly match names, so that names such as "longitude", "Longitude" will automatically be decoded as longitude.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson almost 4 years ago

nctoolkit - v0.6.1

This is a minor release. It introduces a couple of new methods: rolling_var and rolling_stdev for calculating rolling variances and standard deviations.

All code written for the previous version of nctoolkit will be compatible.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson almost 4 years ago

nctoolkit - v0.6.0

This is a major release that introduces some breaking changes.

All methods that carry out temporal averaging of any sort will now align output times to the right. This applies to methods such as tmean and rolling_mean. The internals when align = "left" option have been modified, as the CDO call was sometimes giving incorrect results.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson almost 4 years ago

nctoolkit - v0.5.4

This is minor release, which features some method enhancements and bug fixes.

It improves the abilities of temporal methods, giving users the ability to select how they want times in output to be aligned. For example, if you are calculating a rolling mean, you might want the output times to be the first, middle or final time in the temporal window. This release will add that ability to nctoolkit's temporal methods. Previously nctoolkit used CDO's default methods, and did not allow users to do anything else. By default, output dates will be aligned to the middle time step.

The match_points methods were throwing an error when there were non-unique vertical values. This is now fixed.

Some improvements have been made to package internals.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson almost 4 years ago

nctoolkit - v0.5.3

A quick bug fix release.

Fixes an issue with imports that caused match_points to fail sometimes. Also, fixes subset so that it will allow * and ? wildcards for variables.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson almost 4 years ago

nctoolkit - v0.5.2

This release improves error handling when nctoolkit fails due to an error in Climate Data Operators. Datasets will now be reset to the starting point when this happens and a clear message will given to users.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson almost 4 years ago

nctoolkit - v0.5.1

This is a minor release. It includes method enhancements.

The subset method now allows negative time slicing.

The set_missing method is deprecated and replaced with a less ambiguously named as_missing method.

The plot method will no longer show a plot title by default to make things cleaner.

The vertical_integration method now works with multi-file datasets and will not calculate vertical integrations for the thickness variable.

Some improvements have been made to improve error messages, and the check method now checks for data type of time.

A new method as_type has been added for changing data type of individual variables and coordinates.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson almost 4 years ago

nctoolkit - v0.5.0

This release enhances a couple of methods.

The subset method now lets users subset vertical levels using levels or depths. This requires a two element list, and the subsetting will give you the levels between the two given. So, ds.subset(levels = [0, 100]) would give you the top 100 metres in an oceanic dataset.

The match_points method now allows extrapolation to vertical depths, with the maximum distance extrapolated given by the max_extrap argument.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson almost 4 years ago

nctoolkit - v0.4.9

This release enhances the subset method. You can now subset levels in datasets.

Some improvements have also been made to error handling when CDO fails.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson almost 4 years ago

nctoolkit - v0.4.8

This release improves temporal merging of large datasets. Previously on some systems this would fail on datasets made up of more than 1,000 files due to system limits. Under the hood, nctoolkit now deals with this.

The merge method also now contains a check argument that can be used to speed up merging of large datasets when you know the files can be merged problem-free. Previously, merge always checked if files being merged had the same variables when doing a temporal merge. This can now be switched off if you are confident this does not need to happen.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson almost 4 years ago

nctoolkit - v0.4.7

This release contains a new method called match_points that can do matchups with a spatiotemporal dataframe.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson almost 4 years ago

nctoolkit - v0.4.6

This release enhances existing methods, with a focus on improving subsetting.

The select method is replaced by subset. This behaves in the way same way as select, but will also allow users to subset data based on longitude and latitude using lon and lat as args. select is now deprecated.

The export methods to_nc, to_xarray and to_dataframe now allow only a subset of the data to be exported. Additional arguments can be sent to the methods, and they will then be sent to the subset method.

The new matchpoint methods for matching netCDF and point data have been smoothed out with additional options.

Minor bug fix: The weights in datasets with recycled regridding weights were not copied properly. This is now fixed.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson almost 4 years ago

nctoolkit - v0.4.5

This is a minor release.

A couple of improvements have been made to the matchpoint methods.

This release is largely being made to deal with the fact that the readthedocs failed to produce the docs for version 0.4.4.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson about 4 years ago

nctoolkit - v0.4.4

A minor release with enhancements.

This version introduces a new class called Matchpoint which will allow automated matchups between netCDF files and point observations in pandas dataframes. This class is created using nc.open_matchpoint. Matchups are generated by using the add_data, add_points, add_depths, and matchup methods.

For datasets, ds now provides a more informative summary of dataset contents.

The split method now automatically sorts the files, so that they are sorted by date when temporal splitting occurs.

The methods surface, merge_time and tvariancehave been removed after periods of deprecation. Usetop,mergeandtvar` instead.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson about 4 years ago

nctoolkit - v0.4.3

This is a release with some new methods, improvements to internals and some bug fixes. Code written for previous 0.4x versions of nctoolkit will be compatible.

This version will be compatible with CDO versions 2.0.5x.

A new function open_geotiff will allow GeoTiff files to be opened. This is a wrapper around rioxarray, which will convert the GeoTiff to NetCDF. It will require rioxarray to be installed.

A new method surface_mask has been added to enable identifying top levels with data in cases when there are missing values in the actual top level.

A new method is_corrupt has been added. This can identify whether NetCDF files are likely to be corrupt. Under-the hood, methods will now suggest running is_corrupt when system errors imply the files are corrupt.

The methods to_xarray and to_dataframe no long accept the cdo_times argument, as this has essentially been redundant for a few nctoolkit versions.

The plot method now lets users send kwargs to hvplot to make customizations, such as log-scales an option. This will require the latest version of ncplot.

The select method now lets user select days of month, using ds.select(day = 1).

The split method now allows splitting by timestep using split("timestep").

Scientific Software - Peer-reviewed - Python
Published by robertjwilson about 4 years ago

nctoolkit - v0.4.2

This is a minor release that includes a couple of method enhancements.

You can now save plots as html file using the out argument. An example of how to do this is as follows:

ds.plot(out = "foo.nc")

The nc_command method now automatically uses multiple cores when these are set by nc.options. Previously it could not run in parallel on multi-file datasets.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson about 4 years ago

nctoolkit - v0.4.1

This is a minor release focusing on improving nctoolkit internals.

New methods and improvements

A new method, called check is introduced that can be used to troubleshoot data problems and to ensure there are no obvious data issues (such as a lack of CF-compliance).

Users can now access dataset calendars using ds.calendar.

The drop method now lets you remove time steps using the times argument.

The dataset attribute variables_detailed is now removed after being replaced by contents in version 0.3.9.

Some coding improvements have enhanced the performance of the add, subtract etc. methods.

Support for older versions of CDO

This version will recommend a CDO version of at least 1.9.7, because ensuring nctoolkit compatibility with earlier versions was becoming difficult and likely of little need to users.

Bug fixes

The methods multiply etc. failed when datasets did not have time as a dimension in version 0.4.0. This is now fixed. Previously, ds.contents always returned None for the number of time steps. Now fixed.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson about 4 years ago

nctoolkit - v 0.4.0

This is a major release that features some breaking changes.

Methods for adding, subtracting, multipling and dividing datasets from each other will be enhanced. Until now these methods used a simplistic approach where values from matching time steps were added to each other, etc. So if you are subtracting a 12 time step file from a dataset, only the first 12 time steps were subtracted from. However, often this is not what you want. For example, you might want to subtract data from a file which contains montly values for each year.

This version of nctoolkit updates these methods so that it can automatically figure out, in most cases, what kind of addition etc. it should carry out. For example, if you have a dataset which has monthly values for each year from 1950 to 1999, and use subtract to subtract the values from a file which contains annual means for each year from 1950, it will subtract the annual mean for 1950 from each month in 1950 and the the annual mean for 1951 from each month in 1951, and so on.

Users are now able to specify the numeric precision of datasets using ds.set_precision. By default, nctoolkit uses the underlying netCDF files's data type. This is normally not a problem. However, when the data type is integer, this can cause problems. nc.open_data has been updated with this issue in mind. It will now warn users when the data type of the netCDF is integer, and it will suggest switching to float 'F64' or 'F32'.

The drop method has been enhanced. It now accepts day, month and year as arguments to enable dropping specific time periods. For example ds.drop(month = 2, day = 29) will remove leap days. Code written to use the old drop method will now fail, as keywords are now required.

The method surface has now been renamed top for consistency with bottom. surface is deprecated and will be removed in a few months.

The split method now allows users to split datasets into multiple files by variable.

ds.times now returns a datetime object, not a str as before.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson over 4 years ago

nctoolkit - v0.3.9

This is a minor release focusing on under-the-hood improvements.

A new method, from_xarray is added for converting xarray datasets to nctoolkit datasets.

Methods for identifying how many missing values appear in datasets have been added: na_count and na_frac. These will identify the number or fraction of values that are missing values in each grid cell. The methods operate the same way as the temporal methods. So ds.na_frac("year") will give you the fraction of values are missing values each year.

Methods for better upscaling of datasets will be added: box_mean, box_sum, box_max. This will allow you to upscale to, for example, each 10 by 10 grid box using the mean of that grid box. This is useful for upscaling things like population data where you want the upscaled grid boxes to represent the entirety of the grid box, not the centre.

Improvements to merge have been made. When variables are not included in all files nctoolkit will now only merge those in each file in a multi-file dataset. Previously it threw an error.

Functions for finding the times and months in netCDF files are now available: nc_years and `nc_months.

The attribute variables_detailed has been changed to contents. It will also now give the number of time steps available for each variable.

cdo_command now allows users to specify whether the CDO command used is an ensemble method. Previously methods applied on a file by file basis.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson over 4 years ago

nctoolkit - v0.3.8

This is a minor release, focusing on under-the-hood improvements and introducing better handling of files with varying vertical layers.

New methods

A method, vertical_integration for calculating vertically integrated totals for netCDF data of the likes of oceanic data, where the vertical levels vary spatially, is introduced. vertical_mean has been improved and can now calculate vertical mean in cases where the cell thickness varies in space.

Deprecations

merge_time is deprecated, and its functionality will be incorporated into merge. So, following this release ensemble merging operations should use merge.

Improvements

open_url is now able to handle multiple urls. Previously it could only handle one.

Some under-the-hood improvements have been made to assign to ensure that truth statements do not occassionally throw an error.

Full Changelog: https://github.com/pmlmodelling/nctoolkit/compare/v0.3.7...v0.3.8

Scientific Software - Peer-reviewed - Python
Published by robertjwilson over 4 years ago

nctoolkit - v0.3.7

This is a minor release with new features and one bug fix.

New methods

New mathematical methods for simple operations on variables will be added: abs, power, square, sqrt, exp, log and log10. These methods match numpy names.

Bug fix

assign previously did not accept log10 in the lambda function. Now fixed.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson almost 5 years ago

nctoolkit - v0.3.6

This is a minor release.

New methods

New methods ensemble_mean and ensemble_stdev are introduced for calculating variance and standard deviation across ensembles.

Method renaming

The method tvariance is deprecated and is now renamed tvar for naming consistency.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson almost 5 years ago

nctoolkit - v0.3.5

This is a minor release focusing on some under-the-hood improvements in performance and a couple of new methods.

It drops support for CDO version 1.9.3, as this is becoming too time-consuming to continue given the increasingly low reward.

A couple of new methods have been added. distribute enables files to be split up spatially into equally sized m by n rectangles. collect is the reverse of distribute. It will collect distributed data into one file.

In prior releases assign calls could not be split over multiple lines. This is now fixed.

There was a bug in previous releases where regrid did not work with multi-file datasets. This was due to the enabling of parallel processing with nctoolkit. The issue is now fixed.

The deprecated methods mutate and assign have now been removed. Variable creation should use assign.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson about 5 years ago

nctoolkit - v0.3.4

This is a minor release. All non-deprecated methods are unchanged from the prior release.

A new method fill_na has been introduced that allows missing values to be filled with the distanced weighted average neighbour.

The previously deprecated methods removevariables and cellareas have been removed and are replaced permanently by drop and cell_area.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson about 5 years ago

nctoolkit - v0.3.2

This is a quick and minor release to fix a bug in to_nc that resulted in files not being saved to the base directory if specified.

Other than code tidying, there are not other changes to the package.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson about 5 years ago

nctoolkit - v0.3.1

This is a minor release that includes new methods, under-the-hood improvements, minor bug fixes and the removal of deprecated methods. Code written for non-deprecated methods in v0.3.0 will be compatible with v0.3.1.

New methods

New methods are introduced for identifying the first time step when specific numerical thresholds are first exceeded or fallen below etc: first_above, first_below, last_above and last_below. The thresholds are either single numbers or can come from a gridded dataset for grid-cell specific thresholds.

Methods to compare a dataset with another dataset or netCDF file have been added: gt and lt, which stand for ‘greater than’ and ‘less than’.

Method improvements

Users will now be able to recycle the weights calculated when interpolating data. This can enable much faster interpolation of multiple files with the same grid.

There have been some under-the-hood improvements to improve automatic deletion of temporary files when running in parallel.

Bug fixes

assign threw an error when an internal CDO function was used more than once. This is now fixed.

Method removals post-deprecation

The temporal methods replaced by tmean etc. have now been removed from the package. So all previous temporal averaging methods of the form daily_, monthly_ and annual_ can no longer be used.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson about 5 years ago

nctoolkit - v0.3.0

This is a major release that includes important significant enhancements to nctoolkit's functionality and breaking changes to prior releases.

Variable creation

A new method has been added for creating new variables: assign. This operates in a similar way to the assign method in pandas, and means users can create new variables using lambda functions. This replaces the mutate and transmute methods, which were always intended to be place-holders for a more effective method.

Parallel processing

nctoolkit processing chains can now be run in parallel after under-the-hood changes. This means you can wrap a processing chain into a function, and use the multiprocessing package to process files in parallel.

Evaluation is now lazy by default

In previous releases, evaluation was eager, not lazy, by default. From now on, evaluation will be lazy by default, but can be changed using nc.options(lazy = False).

Configuration files

The package now accepts the use of a configuration file, which should be called .nctoolkitrc or nctoolkitrc and be placed in your working directory or home directory. This means arguments sent to options can be set permanently or on a project basis. This makes life easier for people who have a small or shared temporary folder that they do not want to write to.

Datasets now have clearer list-like behaviour

Improvements have been made to datasets to make them behave in a more list-like manner. data.current will now return a list, whereas in the past it would return a str representing the file in a single-file dataset. This has enabled improvements to how files in datasets can be handled. You can now create an empty dataset, for cases when you need to progressively build a dataset. Likewise, there is now a remove method for removing files from datasets, which goes along with the existing append method for appending files.

A potential consequence of this is that code that assumes data.current etc. is a str will likely break, as it will now be a list.

Removed methods

select_ methods, such as select_years, have been removed after a few months of deprecation. Use select instead.

write_nc has been removed after a few months of deprecation. Use to_nc instead.

clip has been removed after a few months of deprecation. Use crop instead.

release has been removed after a few months of deprecations. Use run instead.

Renamings and deprecations

cell_areas renamed to cell_area. mutate and transmute have been replaced by assign. These methods have therefore been deprecated and will be deleted in a couple of months.

remove_variables is now replaced by drop. remove_variables is deprecated.

compare_all is now replaced by compare. compare_all is deprecated.

Bug fixes

An error was being thrown when deleting temp files hanging round from NCO failing to create files. This is now fixed.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson over 5 years ago

nctoolkit - v0.2.4

This minor release adds new features. Code written for prior releases are compatible with this release.

Improved temporal statistics

Calculating temporal statistics has been streamlined. Temporal methods are now prefixed with the letter t, and accept grouping options. So, for example, data.tmean(["year", "month"]) will calculate the mean value for each month in each year.

Mild enhancements to methods

The select and shift methods have been tweaked to make life slightly easier for users, and allow partial matches for args. For example data.shift(hours = 1) and data.shift(hour = 1) are both valid.

open_url is now able to keep attempting to download files for a set period until complete.

open_thredds now has a wait option to stop

Bug fixes

Issues related to user specified temp folders not being set are now fixed.

Renamings and deprecations

Existing temporal statistics methods, such as mean, max etc., have now been deprecated and the new methods tmean etc. should be used.

vertical_cum_sum has been renamed to vertical_cumsum. vertical_cum_sum is now deprecated.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson over 5 years ago

nctoolkit - v0.2.3

Plotting improvements

Automatic plotting now relies on the new ncplot package. Plotting is now greatly enhanced and includes automatic coastlines.

Mild enhancements to methods

Users can now specify the temporary folder to be used by nctoolkit with nctoolkit.options(temp_dir = x)

Deprecations and renaming

var is now called variance to be more consistent with python. var is deprecated and will be removed early 2021.

bug fixes

append was not working properly for ensembles. Now fixed

Scientific Software - Peer-reviewed - Python
Published by robertjwilson over 5 years ago

nctoolkit - v0.2.2

This release has some new methods, a minor bug fix and some enhancements. It is fully compatible with code written for the previous release.

New methods

centre. This method let's you work out the latitudinal and longitudinal centre.
strip_variable. This method let's you removed associated variables from variables.
resample_grid. This method let's you resample grids.
format. This method let's you change the format of dataset NetCDF files.

Deprecations

write_nc has been replaced by to_nc. write_nc will be replaced in a few months.

Bug fixes

cell_areas was throwing errors for CDO 1.9.6. This is now fixed.

Enhancements

Improvements have been made to plot to improve handling of non-spatial datasets. Plotting development is now separated into a new package ncplot. This will become a dependency in an upcoming version of nctoolkit. Read about it here.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson over 5 years ago

nctoolkit - v0.2.1

New methods

annual_sum: a method for calculating the annual sums
daily_mean etc.: methods for calculating daily statistical values
select: a wraper around existing select methods such as select_year and select_variables
shift_months and shift_years: methods to shift time by months and years
shift: a wrapper around existing shift methods

Deprectations

select_season is now called select_seasons. select_season has been deprecated and will be removed in a few months.
select_timestep is now called select_timesteps. select_timestep has been deprecated and will be removed in a few months.

Minor fixes

plot was throwing errors when plotting time series graphs for some types of data. This is now fixed.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson over 5 years ago

nctoolkit - v0.2.0

macOS support

This release adds support for macOS.

New methods

Meridonial statistics can be calculated using meridonial_mean etc.

open_url is a version of open_dataset for urls.

open_thredds is a version of open_dataset for thredds.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson over 5 years ago

nctoolkit - v0.1.5

Minor update that is compatible with the prior 0.1.4 release.

Method improvement

cor_space and cor_time methods for comparing datasets now work for multiple variables.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson almost 6 years ago

nctoolkit - v0.1.4

New methods

You can now shift time forwards or backwards using the shifthours and shiftdays methods.

You can now add new files to existing datasets using the append method.

Bug fixes

Cleanup code fixed to ensure temporary grid files are removed. This only appears to have affected to_latlon.

ensemble_min ignored time when cdo version = 1.9.3. Now fixed to work properly with cdo version 1.9.3.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson almost 6 years ago

nctoolkit - v0.1.3

This release is a minor update, which is compatible with the previous releases.

New methods have been added for calculating zonal statistics: zonal_mean, zonal_min, zonal_max and zonal_range.

open_data is now able to accept urls.

Scientific Software - Peer-reviewed - Python
Published by robertjwilson almost 6 years ago

nctoolkit - v0.1.2

Initial release

Scientific Software - Peer-reviewed - Python
Published by robertjwilson about 6 years ago

Recent Releases of nctoolkit

nctoolkit - v1.2.3

nctoolkit - v1.2.2

nctoolkit - v1.2.1

nctoolkit - v1.2.0

nctoolkit - v1.1.11

nctoolkit -

Version 1.1.10

nctoolkit - v1.1.8

nctoolkit - v1.1.7

Minor bug fix release

nctoolkit - v1.1.6

Adding the ability to create vertical level bounds

nctoolkit - v1.1.5

Faster import speeds

nctoolkit - v1.1.3

Enhanced temporal methods

nctoolkit - v1.1.2

nctoolkit - v1.1.1

nctoolkit - v1.1.0

nctoolkit - v1.0.0

nctoolkit - v0.9.4

nctoolkit - v0.9.3

nctoolkit - v0.9.2

nctoolkit - v0.9.1

nctoolkit - v0.9.0

nctoolkit - v0.8.7

nctoolkit - v0.8.6

nctoolkit - v0.8.5

nctoolkit - v0.8.4

nctoolkit - v0.8.3

nctoolkit - v0.8.2

nctoolkit - v0.8.1

nctoolkit - v0.8.0

nctoolkit - v0.7.6

nctoolkit - v0.7.5

nctoolkit - v0.7.4

nctoolkit - v0.7.3

nctoolkit - v0.7.2

nctoolkit - v0.7.1

nctoolkit - v0.6.3

nctoolkit - v0.6.2

nctoolkit - v0.6.1

nctoolkit - v0.6.0

nctoolkit - v0.5.4

nctoolkit - v0.5.3

nctoolkit - v0.5.2

nctoolkit - v0.5.1

nctoolkit - v0.5.0

nctoolkit - v0.4.9

nctoolkit - v0.4.8

nctoolkit - v0.4.7

nctoolkit - v0.4.6

nctoolkit - v0.4.5

nctoolkit - v0.4.4

nctoolkit - v0.4.3

nctoolkit - v0.4.2

nctoolkit - v0.4.1

New methods and improvements

Support for older versions of CDO

Bug fixes

nctoolkit - v 0.4.0

nctoolkit - v0.3.9

nctoolkit - v0.3.8

New methods

Deprecations

Improvements

nctoolkit - v0.3.7

New methods

Bug fix

nctoolkit - v0.3.6

New methods

Method renaming

nctoolkit - v0.3.5

nctoolkit - v0.3.4

nctoolkit - v0.3.2

nctoolkit - v0.3.1

New methods

Method improvements

Bug fixes