Piptools-sync

A pre-commit plugin to align pre-commit repository versions with those derived by pip-tools.

pip-tools and Pre-Commit are two of my favourite development tools. However, they don't always necessarily agree on what versions of packages should be installed. This lack of consolidation can lead to problems with pre-commit. This pre-commit plugin syncs the pre-commit package versions with the versions generated by pip-tools' compile process.

Introduction

Project Rationale

A while ago I came across an issue committing files to my local git repository. The issue occured at the flake8 linting stage using pre-commit. There was however no problem manually running flake8 from the command line. Upon investigation It was found that flake8 had an incompatibility issue with one of its plugins. With further investigation I noticed that pip-tools had pinned flake8 to an earlier version to what pre-commit was caching.

In short - pip-tools does a spectacularly good job pinning your dependencies and the dependencies of these dependencies. It has one job to do and it does it perfectly. The pre-commit autoupdate command just updates the "rev" for the "repo" in the .pre-commit-config.yaml file to the latest version available and ignores the dependencies.

The following example displays the way flake8 and a few plugins are configured by the two tools :

• pip-tools (via requirements.txt)

shell flake8==4.0.1 # via ... flake8-bugbear==22.8.23 # via ... flake8-comprehensions==3.10.0 # via ... flake8-eradicate==1.3.0 # via ... flake8-simplify==0.19.3 # via ...

• pre-commit (via .pre-commit-config.yaml)

shell - repo: https://github.com/pycqa/flake8 rev 5.0.4 hooks: - id: flake8 additional_dependencies: - flake8-bugbear - flake8-comprehensions - flake8-eradicate - flake8-simplify args: ["--ignore=E800,F842,F841,W503"]

If we force pip-tools to use a later version of flake8 (e,g 5.0.4) by manually pinning the version in the ".in" file and recompiling it is clear pip-tools is unhappy... A quick look at the dependencies shows why.

shell Could not find a version that matches flake8!=3.2.0,<5.0.0,<6,==5.0.4,>=3.0.0,>=3.3.0,>=3.5,>=3.7,>=3.9.1,>=4.0.1 (from -r requirements\development.in (line 12)) Tried: 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 1.1, 1.2, 1.3, 1.3.1, 1.4, 1.5, 1.6, 1.6.1, 1.6.2, 1.7.0, 2.0, 2.1.0, 2.2.0, 2.2.0, 2.2.1, 2.2.1, 2.2.2, 2.2.2, 2.2.3, 2.2.3, 2.2.4, 2.2.5, 2.3.0, 2.3.0, 2.4.0, 2.4.0, 2.4.1, 2.4.1, 2.5.0, 2.5.0, 2.5.1, 2.5.1, 2.5.2, 2.5.2, 2.5.3, 2.5.3, 2.5.4, 2.5.4, 2.5.5, 2.5.5, 2.6.0, 2.6.0, 2.6.1, 2.6.1, 2.6.2, 2.6.2, 3.0.0, 3.0.0, 3.0.1, 3.0.1, 3.0.2, 3.0.2, 3.0.3, 3.0.3, 3.0.4, 3.0.4, 3.1.0, 3.1.0, 3.1.1, 3.1.1, 3.2.0, 3.2.0, 3.2.1, 3.2.1, 3.3.0, 3.3.0, 3.4.0, 3.4.0, 3.4.1, 3.4.1, 3.5.0, 3.5.0, 3.6.0, 3.6.0, 3.7.0, 3.7.0, 3.7.1, 3.7.1, 3.7.2, 3.7.2, 3.7.3, 3.7.3, 3.7.4, 3.7.4, 3.7.5, 3.7.5, 3.7.6, 3.7.6, 3.7.7, 3.7.7, 3.7.8, 3.7.8, 3.7.9, 3.7.9, 3.8.0, 3.8.0, 3.8.1, 3.8.1, 3.8.2, 3.8.2, 3.8.3, 3.8.3, 3.8.4, 3.8.4, 3.9.0, 3.9.0, 3.9.1, 3.9.1, 3.9.2, 3.9.2, 4.0.0, 4.0.0, 4.0.1, 4.0.1, 5.0.0, 5.0.0, 5.0.1, 5.0.1, 5.0.2, 5.0.2, 5.0.3, 5.0.3, 5.0.4, 5.0.4 Skipped pre-versions: 3.0.0b1, 3.0.0b1, 3.0.0b2, 3.0.0b2, 3.0.2.dev0, 3.0.2.dev0, 3.0.2.dev1, 3.3.0.dev0, 3.8.0a1, 3.8.0a1, 3.8.0a2, 3.8.0a2 There are incompatible versions in the resolved dependencies: flake8==5.0.4 (from -r requirements\development.in (line 12)) flake8>=3.7 (from flake8-simplify==0.19.3->-r requirements\development.in (line 20)) flake8<5.0.0 (from flake8-bugbear==22.8.23->-r requirements\development.in (line 14)) flake8!=3.2.0,>=3.0 (from flake8-comprehensions==3.10.0->-r requirements\development.in (line 15)) flake8<6,>=3.5 (from flake8-eradicate==1.3.0->-r requirements\development.in (line 17))

The tools store packages in different areas on the file system. Pre-commit uses a cache area in the user folder, whereas pip-tools uses the usual site-packages area.

It should be noted that not all pre-commit hooks are written in Python for Python, so it makes sense that it has its own cache file system independent of pip. Therefore, not all pre-commit repositories have an entry in PyPI. They can simply be defined by a GitHub repository URL.

Lastly, browsing the internet looking for a solution I came across the following issue raised under the pip-tools project (dated 24 Jun 2021). The issue is still open and describes the exact same problem.

https://github.com/jazzband/pip-tools/issues/1437

Installation

1 - Install into pre-commit

Just add to the pre-commit configuration file (.pre-commit-config.yaml). I have configured it to run at every commit at the pre-commit stage of git.

shell - repo: https://github.com/Stephen-RA-King/piptools-sync rev: 0.3.1 hooks: - id: piptools_sync

2 - Install by pip

Installing by pip enables the package to be run from the command line at any time.

sh pip install piptools-sync

Usage

1 - Automatic running by pre-commit

Passing example.

shell piptools_sync............................................................Passed

Failing example.

```shell piptoolssync............................................................Failed - hook id: piptoolssync - exit code: 1 - files were modified by this hook

INFO:init:flake8 - piptools: 4.0.1 != pre-commit: 5.0.4 ```

Note: In this case piptools-sync will automatically update the pre-commit config file with pip-tools version. However, the commit with fail and will need to be re-run.

2 - Running from the command line

Passing example.

shell $ piptools_sync Success! - pre-commit is in sync with piptools

Failing example.

shell $ piptools_sync flake8 - piptools: 4.0.1 != pre-commit: 5.0.4

Note: In this case piptools-sync will automatically update the pre-commit config file with pip-tools version

For more examples and usage, please refer to the Wiki.

Documentation

Read the Docs

• Example Usage
• Credits
• Changelog
• API Reference

Wiki

Future Enhancements

• Move some global variables into a separate settings file (toml).
• Settings configurable from the command line (and therefore configurable from the pre-commit.yaml file).
• Improve web request performance with asyncio / aiohttp libraries.

Meta

Stephen R A King : sking.github@gmail.com

Distributed under the MIT license. See for more information.

Created with Cookiecutter template: version 1.2.1

Digital Object Identifier: