skbuild-conan: A conan extension for scikit-build

PyBind11 and scitkit-build enable us to easily write native C++-modules for Python. However, you get problems if your C++-code has dependencies. This extension tries to make defining C++-dependencies as easy as defining the Python-dependencies. This way you can easily add any C++-library that has a conan recipe to your Python-project.

This project originates from our need to use complex C++-libraries in Python projects and missing any nice option to include C++-dependencies. For a few projects, we wrote individual code to fetch the dependencies or just added instructions on how to install them (which of course can scare pure Python users).

This project is currently just a nice interface to hacks we accumulated. We try to make it as universal and robust as possible as we rely on this for multiple projects, but we are working fast-paced and will quickly abandon this tool once there is something better.

We (TU Braunschweig, Algorithms Group) are not affiliated with scikit-build or conan.

Installation

For skbuild-conan to work, you need to have a full Python and C++ development environment. This means that you need to have a compiler installed and the Python development files. The Python development files are usually called python3-dev or python3-devel and the compiler is usually called gcc or clang. On most Linux distributions, these are installed by default.

You can simply add "skbuild_conan", to requires=[...] in pyproject.toml. E.g. The pyproject.toml could look like this

toml [build-system] requires = [ "conan>=2.0.0", "setuptools", "scikit-build>=0.17.3", "skbuild-conan", "cmake>=3.23", "ninja", ] build-backend = "setuptools.build_meta"

If you want to use for example setup.py build, you need to install skbuild_conan to your environment. You can do so by pip install skbuild_conan.

Usage

The usage is very similar to scitkit-build (and setuptools). We just added a few additional arguments to setup().

See how to use scikit-build first, as this is just a small extension to it.

The added options are

• conanfile: Path to the folder with the conanfile.[py|txt]. By default the root is assumed. The conanfile can be used to define the dependencies. Alternatively, you can also use conan_requirements to define the conan dependencies without a conanfile. This option is exclusive. If you define conan_requirements, this option is ignored.
• conan_recipes: List of paths to further conan recipes. The conan package index is far from perfect, so often you need to build your own recipes. You don't always want to upload those, so this argument gives you the option to integrate local recipes. Just the path to the folder containing the conanfile.py.
• conan_requirements: Instead of providing a conanfile, you can simply state the dependencies here. E.g. ["fmt/[>=10.0.0]"] to add fmt in version >=10.0.0.
• conan_profile_settings: Overwrite conan profile settings. Sometimes necessary because of ABI-problems, etc.
• wrapped_setup: The setup-method that is going to be wrapped. This would allow you to extend already extended setup functions. By default, it is the setup of skbuild, which extends the setup of setuptools.
• conan_output_folder: The folder where conan will write the generated files. No real reason to change it unless the default creates conflicts with some other tool.
• cmake_args: This is actually an argument of skbuild but we will extend it. It hands cmake custom arguments. We use it to tell cmake about the conan modules.
• conan_profile: The name of the conan profile to use. By default, it is skbuild_conan_py. This profile is created automatically and should work for most cases. If you need to change it, you can do so by editing ~/.conan2/profiles/skbuild_conan_py.
• conan_env: Environment variables that are used for the conan calls. By default it will override CC and CXX with empty strings. This is necessary to work around problems with anaconda, but it should not cause any problems with other setups. You could define CONAN_HOME to ./conan/cache to use a local cache and not install anything to the user space.

An example usage could be as follows

```python from skbuildconan import setup from setuptools import findpackages

setup( # https://scikit-build.readthedocs.io/en/latest/usage.html#setup-options name="simpleskbuildconanexample", version="0.1.1", packages=findpackages("src"), # Include all packages in ./src. packagedir={"": "src"}, # The root for our python package is in ./src. pythonrequires=">=3.7", # lowest python version supported. installrequires=[], # Python Dependencies conanrequirements=["fmt/[>=10.0.0]"], # C++ Dependencies cmakeminimumrequired_version="3.23", ) ```

See ./examples/simpleskbuildconan_example for a full example.

Examples

If you do not have any C++-dependencies, you can just use scikit-build which also provides a set of examples.

Simple Example with fmt

The example in ./examples/simpleskbuildconan_example provides a minimal example of how to use fmt in Python using PyBind11 and skbuildconan. fmt is a nice library for formatting strings and is used by many other libraries. Python, of course, comes with extensive inbuilt string formatting, thus, this example is not very useful. However, it is a good starting point to understand how to use skbuildconan with a simple external library.

Complex Example with CGAL: Using CGAL in Python

Sometimes, your dependencies are significantly more complex. For example, you may want to do some geometry processing and use CGAL. CGAL is a very complex library with many dependencies, but also the most powerful library for geometric operations and often the only choice for many problems. CGAL has a conan recipe, but it took a while until it received updates for conan2, such that we wrote our own recipe. In the meantime, the official recipe was updated, but for the sake of the example, we will use our own recipe. In case you are faced with the problem of an outdated conan recipe (or none at all), you can use the same trick.

See ./examples/cgalskbuildconan_example for an example of how to use CGAL via a custom conan recipe in Python using PyBind11 and skbuild_conan.

Note that there is also the cgalpy project by my friends at TAU (which I visited for a few months in 2022/2023), which is a nearly complete and efficient wrapper of CGAL. It may need some more documentation, but Efi put a lot of thought into efficiency and configurability.

Common problems

Feel free to copy these comments. Attribution is appreciated but not necessary.

ABI problems: Undefined symbole ...__cxx1112basic_stringIcSt11char_...

This problem should be automatically fixed. Please open an issue if you still encounter it.

See https://docs.conan.io/1/howtos/managegccabi.html for more details.

glibcxx problems:

If you get an error such as

ImportError: /home/krupke/anaconda3/envs/mo310/bin/../lib/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /home/krupke/anaconda3/envs/mo310/lib/python3.10/site-packages/samplns/cds/_cds_bindings.cpython-310-x86_64-linux-gnu.so)

you are probably using conda (good!) but need to update glibcxx. Install the latest version by

sh conda install -c conda-forge libstdcxx-ng

In some cases, this still is not enough, especially if you are using a very up to date rolling-release distribution, such as Arch Linux, or if you installed libstdcxx-ng some time ago. This could lead to the system having a slightly newer version of glibcxx than conda. First try to upgrade libstdcxx-ng with

sh conda upgrade -c conda-forge --all

If this does not help, you can try to install g++ (caveat: Linux only, Mac OS needs clang) into your conda environment and use it to compile the package.

sh conda install -c conda-forge gxx_linux-64 # This should enforce a modern g++ version. conda install -c conda-forge cxx-compiler # This should make sure that the compiler is used.

Note that just the second command first may install an outdated g++ version (at least I observed that it installed gcc11 instead of gcc13, messing up my whole environment as this is too old). When compiling from source, you probably should delete the _skbuild-folder and do a proper uninstall of the previous installation first.

conan problems

If you encounter problems with conan, you can try to delete the conan profile and let it be recreated.

sh rm ~/.conan2/profiles/skbuild_conan_py

Maybe you can also just take a look at the file and see if conan detected your compiler correctly. A proper profile on Linux should for example look like this (different for other systems):

[settings] arch=x86_64 build_type=Release compiler=gcc compiler.cppstd=gnu17 compiler.libcxx=libstdc++11 compiler.version=13 os=Linux

If the problem persists, you can try to delete the conan cache.

sh rm -rf ~/.conan2

fatal error: Python.h: No such file or directory

This is a rare problem with your Python installation. Make sure that you have the development files installed.

sudo apt-get install python3-dev # Ubuntu sudo yum install python3-devel # CentOS, RHEL sudo dnf install python3-devel # Fedora sudo zypper in python3-devel # OpenSUSE sudo apk add python3-dev # Alpine sudo apk add python3-dev # Cygwin

In case you have multiple Python versions installed, you may need to specify the Python version.

If you still encounter problems, please open an issue.

MetadataPathFinder.invalidate_caches() missing 1 required positional argument: 'cls'

A very odd bug that we encountered with a distribution of Python 3.12 was the following:

``` ... TypeError: MetadataPathFinder.invalidate_caches() missing 1 required positional argument: 'cls'

ERROR: MetadataPathFinder.invalidate_caches() missing 1 required positional argument: 'cls' ```

The reason for this was surprisingly a bug in the Python distribution of importlib itself. Someone forgot to add the @classmethod to the MetadataPathFinder. Updating the Python distribution, in this case in conda via conda update python solved the problem.

Contribution

We are happy about any contribution and also about reported issues. Sometimes it can take some time before we are able to take care of something, as we need to prioritize quite often.

Changelog

• 1.3.0 The Debug/Release will propagate to the conan profile. (thanks to @xandox)
• 1.2.0 Workaround for Windows and MSVC found by Ramin Kosfeld (TU Braunschweig).
• 1.1.1 Fixing problem if the conan default profile has been renamed via environment variable.
• 1.1.0 conan is now called directly. This is kind of hacky, but circumvents problems with conan not being in the path if only installed for build.
• 1.0.0 Custom conan profile and workaround for anaconda problem.
• 0.2.0 Improved logging.
• 0.1.4 Fixing problem with paths that contain spaces. Switching back to manual versioning.
• 0.1.3 Fixing bug if no settings are given.
• 0.1.2 Moved workaround into setup to make it more explicit.
• 0.1.1 First tested and apparently working version.