Data

Tools

Indexes

Applications

Experiments

Open Source Science

Recent Releases of async-retriever

async-retriever - v0.19.3

Release Notes

Internal Changes

  • Use orjson instead of ujson due to the package not being maintained anymore. The developer of ujson raised conrcerns about security vulnerabilities and recommended using orjson instead.
  • Refactor the package to run in Jupyter notebooks without using nest_asyncio. This is done by creating and initializing a single global event loop thread dedicated to only running the asynchronous parts of this package.
  • Use threading.Event in _AsyncLoopThread for a more robust way to signal the event loop thread to stop running. This should prevent the event loop thread from hanging when the main thread exits before the event loop thread is done running. Overall, this improves thread safety and robustness of the package.
  • More robust handling of starting and stopping threads by lazy generation of a dedicated thread for the library and making _AsyncLoopThread a singleton. This can avoid issues that might arise from using TinyRetriever with other libraries that also use threads such as shapely.

- Python
Published by github-actions[bot] 12 months ago

async-retriever - v0.19.1

Release Notes

Internal Changes

  • Replace max_connection argument in all retrieve* functions with limit_per_host. This uses the aiohttp argument limit_per_host to limit the number of connections per host which prevents the server from being overwhelmed with too many connections. The default value is set to 5.
  • Add aiofiles as a new dependency for writing responses to a file in the stream_write function. This should speed up the writing process and make it more efficient.

Bug Fixes

  • Fix an issue with running the library inside IPython terminal. (50{.interpreted-text role="issue_async"})

Breaking Changes

  • Move stream_write function to its own module called streaming. This is to separate the streaming functionality from the main module and make it easier to maintain.
  • Simplify the stream_write function to only accept a list of URLs and a directory path to save the files. It no longer accepts passing keyword arguments and assumes that the user has added the necessary arguments to the URLs itself and is encoded correctly. Additionally, the chunk size by default is set to 1 MB.

- Python
Published by github-actions[bot] about 1 year ago

async-retriever - v0.19.0

Release Notes

Internal Changes

  • Replace max_connection argument in all retrieve* functions with limit_per_host. This uses the aiohttp argument limit_per_host to limit the number of connections per host which prevents the server from being overwhelmed with too many connections. The default value is set to 5.
  • Add aiofiles as a new dependency for writing responses to a file in the stream_write function. This should speed up the writing process and make it more efficient.

Bug Fixes

  • Fix an issue with running the library inside IPython terminal. (50{.interpreted-text role="issue_async"})

Breaking Changes

  • Move stream_write function to its own module called streaming. This is to separate the streaming functionality from the main module and make it easier to maintain.
  • Simplify the stream_write function to only accept a list of URLs and a directory path to save the files. It no longer accepts passing keyword arguments and assumes that the user has added the necessary arguments to the URLs itself and is encoded correctly. Additionally, the chunk size by default is set to 1 MB.

- Python
Published by github-actions[bot] about 1 year ago

async-retriever - v0.18.0

Release Notes

Bug Fixes

  • Pin the minimum version of aiohttp-client-cache to 0.12.3 to an issue with the latest version of aiohttp. (48{.interpreted-text role="issue_async"})

Breaking Changes

  • Drop support for Python 3.8 since its end-of-life date is October 2024.
  • Remove all exceptions from the main module and raise them from the exceptions module. This is to declutter the main module and make it easier to maintain.

Internal Changes

  • Make nest_asyncio a required dependency since most users will need it to run the package in Jupyter notebooks.

- Python
Published by github-actions[bot] over 1 year ago

async-retriever - v0.17.1

Release Notes

Internal Changes

  • Drop support for Python 3.8 since its end-of-life date is October 2024.

- Python
Published by github-actions[bot] over 1 year ago

async-retriever - v0.17.0

Release Notes

Internal Changes

  • Add the exceptions module to the high-level API to declutter the main module. In the future, all exceptions will be raised from this module and not from the main module. For now, the exceptions are raised from both modules for backward compatibility.
  • Switch to using the src layout instead of the flat layout for the package structure. This is to make the package more maintainable and to avoid any potential conflicts with other packages.
  • Add artifact attestations to the release workflow.

- Python
Published by github-actions[bot] over 1 year ago

async-retriever - v0.16.1

Release Notes

New Features

  • When a request fails with encoding error, it gets retried with the encoding set to latine1. This is to handle cases where the server sends a response with an encoding that is not supported by the client. (120{.interpreted-text role="issue_hydro"})

- Python
Published by github-actions[bot] almost 2 years ago

async-retriever - v0.16.0

Release Notes

New Features

  • Add a new environmental variable called "HYRIVER_SSL_CERT" for setting the path to a SSL certificate file other than the default one. You can do this like so:

``` python import os

os.environ["HYRIVERSSLCERT"] = "path/to/file.pem" ```

- Python
Published by github-actions[bot] about 2 years ago

async-retriever - v0.15.2

Release Notes

Bug Fixes

  • Fix an issue with getting all valid keywords that aiohttp accepts by using aiohttp.ClientSession()._request directly.

- Python
Published by github-actions[bot] over 2 years ago

async-retriever - v0.15.0

Release Notes

From release 0.15 onward, all minor versions of HyRiver packages will be pinned. This ensures that previous minor versions of HyRiver packages cannot be installed with later minor releases. For example, if you have py3dep==0.14.x installed, you cannot install pydaymet==0.15.x. This is to ensure that the API is consistent across all minor versions.

Bug Fixes

  • When raise_status is False, responses for failed requests used to return as None but their requests ID was not returned, so sorting would have failed. Now request IDs are returned for all requests regardless of whether they were successful or not.
  • Give precedence to non-default arguments for caching related arguments instead of directly getting them from env variables. This is to avoid the case where the user sets the env variables but then passes different arguments to the function. In this case, the function should use the passed arguments instead of the env variables.

- Python
Published by github-actions[bot] almost 3 years ago

async-retriever - v0.14.0

Release Notes

New Features

  • Add a new option to all functions called raise_status. If False no exception will be raised and instead None is returned for those requests that led to exceptions. This will allow for returning all responses that were successful and ignoring the ones that failed. This option defaults to True for retaining backward compatibility.
  • Set the cache expiration time to one week from never expire. To ensure all users have a smooth transition, cache files that were created before the release of this version will be deleted, and a new cache will be created.

Internal Changes

  • Sync all minor versions of HyRiver packages to 0.14.0.

- Python
Published by github-actions[bot] almost 3 years ago

async-retriever - v0.3.12

Release Notes

Internal Changes

  • Rewrite the private async_session function as two separate functions called async_session_without_cache and async_session_with_cache. This makes the code more readable and easier to maintain.
  • Fully migrate setup.cfg and setup.py to pyproject.toml.
  • Convert relative imports to absolute with absolufy-imports.
  • Make utils module private.
  • Sync all patch versions of HyRiver packages to x.x.12.

- Python
Published by github-actions[bot] about 3 years ago

async-retriever - v0.3.10

Release Notes

New Features

  • Refactor the show_versions function to improve performance and print the output in a nicer table-like format.

Bug Fixes

  • Fix a bug in reading the HYRIVER_CACHE_EXPIRE environmental variable.
  • Bump the minimum version of aiohttp-client-cache to 0.8.1 to fix a bug in reading cache files that were created with previous versions. (:issue_async:[41]{.title-ref})

Internal Changes

  • Enable fast_save in aiohttp-client-cache to speed up saving responses to the cache file.
  • Use pyright for type checking instead of mypy and fix all type errors.
  • Skip 0.13.8/9 versions so the minor version of all HyRiver packages become the same.

- Python
Published by github-actions[bot] about 3 years ago

async-retriever - v0.3.8

Release Notes

New Features

  • Add support for specifying the chunk size in stream_write. Defaults to None which was the default behavior before, and means iterating over and writing the responses as they are received from the server.

Internal Changes

  • Use pyupgrade package to update the type hinting annotations to Python 3.10 style.
  • Modify the codebase based on Refurb suggestions.

- Python
Published by github-actions[bot] about 3 years ago

async-retriever - v0.3.7

Release Notes

New Features

  • Add support for specifying the chunk size in stream_write. Defaults to None which was the default behavior before, and means iterating over and writing the responses as they are received from the server.

Internal Changes

  • Use pyupgrade package to update the type hinting annotations to Python 3.10 style.
  • Modify the codebase based on Refurb suggestions.

- Python
Published by github-actions[bot] about 3 years ago

async-retriever - v0.3.6

Release Notes

Internal Changes

  • Add the missing PyPi classifiers for the supported Python versions.
  • Release the package as both async_retriever and async-retriever on PyPi and Conda-forge.

- Python
Published by github-actions[bot] over 3 years ago

async-retriever - v0.3.5

Release Notes

Breaking Changes

  • Append \"Error\" to all exception classes for conforming to PEP-8 naming conventions.

Internal Changes

  • Bump minimum version of aiohttp-client-cache to 0.7.3 since the attrs version issue has been addressed.

- Python
Published by github-actions[bot] over 3 years ago

async-retriever - v0.3.4

Release Notes

New Features

  • Add a new function, stream_write, for writing a response to a file as it\'s being retrieved. This could be very useful for downloading large files. This function does not use persistent caching.

- Python
Published by github-actions[bot] over 3 years ago

async-retriever - v0.3.3

Release Notes

Breaking Changes

  • Set the minimum supported version of Python to 3.8 since many of the dependencies such as xarray, pandas, rioxarray have dropped support for Python 3.7.

Internal Changes

  • Use micromamba for running tests and use nox for linting in CI.

- Python
Published by github-actions[bot] over 3 years ago

async-retriever - v0.3.2

Release Notes

New Features

  • Add support for setting caching-related arguments using three environmental variables:

    • HYRIVER_CACHE_NAME: Path to the caching SQLite database.
    • HYRIVER_CACHE_EXPIRE: Expiration time for cached requests in seconds.
    • HYRIVER_CACHE_DISABLE: Disable reading/writing from/to the cache file.

    You can do this like so:

``` python import os

os.environ["HYRIVERCACHENAME"] = "path/to/file.sqlite" os.environ["HYRIVERCACHEEXPIRE"] = "3600" os.environ["HYRIVERCACHEDISABLE"] = "true" ```

Internal Changes

  • Include the URL of a failed request in its exception error message.

- Python
Published by github-actions[bot] almost 4 years ago

async-retriever - v0.3.1

Release Notes

New Features

  • Add three new functions called retrieve_text, retrieve_json, and retrieve_binary. These functions are derived from the retrieve function and are used to retrieve the text, JSON, or binary content of a response. They are meant to help with type hinting since they have only one return type instead of three different return types that the retrieve function has.

Internal Changes

  • Move all private functions to a new module called utils. This makes the code-base more readable and easier to maintain.

- Python
Published by github-actions[bot] about 4 years ago

async-retriever - v0.3.0

Release Notes

Breaking Changes

  • Set the expiration time to never expire by default.

New Features

  • Add two new arguments to retrieve for controlling caching. First, delete_url_cache for deleting caches for specific requests. Second, expire_after for setting a custom expiration time.
  • Expose the ssl argument for disabling the SSL certification verification (:issue_day:[41]{.title-ref}).
  • Add a new option called disable that if True, it temporarily disables caching requests and gets new responses. It defaults to False.

- Python
Published by github-actions[bot] about 4 years ago

async-retriever - v0.2.5

Release Notes

New Features

  • Add two new arguments, timeout and expire_after, to retrieve. These two arguments gives the user more control for dealing with issues related to caching.

Internal Changes

  • Revert to pytest as the testing framework.
  • Use importlib-metadata for getting the version insead of pkg_resources to decrease import time as discussed in this issue.

- Python
Published by github-actions[bot] over 4 years ago

async-retriever - v0.2.4

Release Notes

Internal Changes

  • Use usjon for converting responses to JSON.

Bug Fixes

  • Fix an issue with catching service error messages.

- Python
Published by github-actions[bot] over 4 years ago

async-retriever - v0.2.3

Release Notes

Internal Changes

  • Use ujson for JSON parsing instead of orjson since orjson only serializes to bytes which is not compatible with aiohttp.

- Python
Published by github-actions[bot] over 4 years ago

async-retriever - v0.2.2

Release Notes

New Features

  • Add a new function, clean_cache, for manually removing the expired responses from the cache database.

Internal Changes

  • Handle all cache file related operations in the create_cachefile function.

- Python
Published by github-actions[bot] over 4 years ago

async-retriever - v0.2.1

Release Notes

New Features

  • The responses now are returned in the same order as the input URLs.
  • Add support for passing connection type, i.e., IPv4 only, IPv6, only or both via family argument. Defaults to both.
  • Set trust_env=True so the session can read system\'s netrc files. This can be useful for working with services such as EarthData service that read the user authentication info from a netrc file.

Internal Changes

  • Replace AsyncRequest class with _retrieve function to increase readability and reduce overhead.
  • More robust handling of validating user inputs via a new class called ValidateInputs.
  • Move all if-blocks in async_session to other functions to improve performance.

- Python
Published by github-actions[bot] over 4 years ago

async-retriever - v0.2.0

Release Notes

Breaking Changes

  • Make persistent caching dependencies required.
  • Rename request to request_method in retrieve which now accepts both lower and upper cases of get and post.

Internal Changes

  • Refactor the entire code-base for more efficient handling of different request methods.
  • Check validity of inputs before sending requests.
  • To avoid the issues such as this, pass a loop explicitly to nest_asyncio.
  • Improve documentation.
  • Improve cache handling by removing the expired responses before returning the results.
  • Increase testing coverage to 100%.

- Python
Published by github-actions[bot] over 4 years ago

async-retriever - v0.1.0

Release Notes

Release Notes

- Python
Published by github-actions[bot] almost 5 years ago