https://github.com/csiro-hydroinformatics/pyrefcount

Python package for reference counting native pointers

Science Score: 23.0%

This score indicates how likely this project is to be science-related based on various indicators:

○
CITATION.cff file
✓
codemeta.json file
Found codemeta.json file
○
.zenodo.json file
○
DOI references
○
Academic publication links
✓
Committers with academic emails
2 of 4 committers (50.0%) from academic institutions
○
Institutional organization owner
○
JOSS paper metadata
○
Scientific vocabulary similarity
Low similarity (15.1%) to scientific vocabulary

Keywords

cffi handle interop native-libraries python reference-counting

Keywords from Contributors

productivity labels

Last synced: 5 months ago · JSON representation

Repository

Python package for reference counting native pointers

Basic Info

Host: GitHub
Owner: csiro-hydroinformatics
License: mit
Language: Python
Default Branch: main
Homepage:
Size: 1.14 MB

Statistics

Stars: 2
Watchers: 1
Forks: 0
Open Issues: 0
Releases: 4

Topics

cffi handle interop native-libraries python reference-counting

Created about 7 years ago · Last pushed 11 months ago

Metadata Files

Readme Changelog Contributing License

refcount - Python classes for reference counting

Reference counted native handles

This package is primarily for managing resources in native libraries, written for instance in C++, from Python. While it boils down to "simply" maintaining a set of counters, it is deceptively complicated to do so properly and not end up with memory leaks or crashes. This package offers structured options for reliably managing external native resources. Surprisingly I could not locate an existing package doing just what I needed. Other use cases requiring reference counting, aside from native library resources, may benefit from reusing and extending classes in refcount.

refcount ( >=0.7) includes classes using cffi. Other low-level interoperability mechanisms may well be added in the future.

License

MIT (see License.txt)

Documentation

Hosted at refcount via readthedocs.io

Source code

The code repository is on GitHub.

Installation

conda-forge

Using conda or mamba:

sh mamba install -c conda-forge refcount

pypi

sh pip install refcount

From source (development)

sh pip install -r requirements.txt pip install -e .

Sample use

The following example is based on one of the unit tests.

Say we have a C++ library with objects and a C API:

```C++

define TESTDOGPTR testnative::dog*

define TESTOWNERPTR testnative::owner*

define TESTCOUNTEDPTR testnative::reference_counter*

testnative::dog* createdog(); testnative::owner* createowner(testnative::dog* d); void saywalk(testnative::owner* owner); void release(testnative::referencecounter* obj); // etc. ```

From the outside of the library the API is exported with opaque pointers void* (C structs pointers and native C99 types could be handled too).

C++ void* create_dog(); void* create_owner(void* d); void say_walk(void* owner); void release(void* obj); // etc.

Starting with the end in mind, from Python we want an API hiding the low level details close to the C API, in particular avoiding managing native memory via release C API calls, piggybacking the python GC instead.

python dog = Dog() owner = DogOwner(dog) owner.say_walk() print(dog.position) dog = None # the "native dog" is still alive though, as the owner incremented the ref count owner = None

This is doable with refcount and the cffi package. One possible design is:

```python ut_ffi = cffi.FFI()

utffi.cdef('extern void* createdog();') utffi.cdef('extern void* createowner( void* d);') utffi.cdef('extern void saywalk( void* owner);') ut_ffi.cdef('extern void release( void* obj);')

etc.

utdll = utffi.dlopen('c:/path/to/testnativelibrary.dll', utffi.RTLDLAZY) # Lazy loading

class CustomCffiNativeHandle(CffiNativeHandle): def init(self, pointer, priorrefcount = 0): super(CustomCffiNativeHandle, self).init(pointer, typeid='', priorrefcount = priorref_count)

def _release_handle(self) -> bool:
    ut_dll.release(self.get_handle())
    return True

class Dog(CustomCffiNativeHandle): def init(self, pointer = None): if pointer is None: pointer = utdll.createdog() super(Dog, self).init(pointer) # etc.

class DogOwner(CustomCffiNativeHandle):

def __init__(self, dog):
    super(DogOwner, self).__init__(None)
    self._set_handle(ut_dll.create_owner(dog.get_handle()))
    self.dog = dog
    self.dog.add_ref() # Do note this important reference increment

def say_walk(self):
    ut_dll.say_walk(self.get_handle())

def _release_handle(self) -> bool:
    super(DogOwner, self)._release_handle()
    # super(DogOwner, self)._release_handle()
    self.dog.release()
    return True

```

Related work

Ancestry, acknowledgements

This python package refcount actually spawned from prior work for interoperability between C++, R and .NET (R.NET)

refcount features using cffi were also significantly informed by Kevin Plastow's work while he was at the Australian Bureau of Meteorology; this contribution is gratefully acknowledged.

In you have native interop needs you may also want to look at:

the nuget package dynamic-interop-dll for .NET/native interop.
a set of mostly c++ software tools for interop with C/C++
a C# library for generating interop glue code on top of C API glue code.

Other python packages

refcount was created in part because no existing prior (Python) work could quite fit the need. There are however packages that may better address your particular need:

infi.pyutils contains a reference counting class.

Development branch

Owner

Name: CSIRO Hydroinformatics
Login: csiro-hydroinformatics
Kind: organization

Repositories: 11
Profile: https://github.com/csiro-hydroinformatics

CSIRO - hydroinformatics repositories

GitHub Events

Total

Release event: 1
Delete event: 5
Issue comment event: 10
Push event: 16
Pull request event: 10
Create event: 12

Last Year

Release event: 1
Delete event: 5
Issue comment event: 10
Push event: 16
Pull request event: 10
Create event: 12

Committers

Last synced: almost 3 years ago

All Time

Total Commits: 98
Total Committers: 4
Avg Commits per committer: 24.5
Development Distribution Score (DDS): 0.092

Past Year

Commits: 52
Committers: 4
Avg Commits per committer: 13.0
Development Distribution Score (DDS): 0.115

Top Committers

Name	Email	Commits
J-M	j**d@c**u	89
Jean-Michel Perraud	p**2@c**u	5
dependabot[bot]	4**]@u**m	2
semantic-release	s****e	2

Committer Domains (Top 20 + Academic)

csiro.au: 2

Issues and Pull Requests

Last synced: 6 months ago

All Time

Total issues: 5
Total pull requests: 21
Average time to close issues: about 2 years
Average time to close pull requests: 2 months
Total issue authors: 1
Total pull request authors: 1
Average comments per issue: 1.0
Average comments per pull request: 1.29
Merged pull requests: 4
Bot issues: 0
Bot pull requests: 21

Past Year

Issues: 0
Pull requests: 10
Average time to close issues: N/A
Average time to close pull requests: 3 months
Issue authors: 0
Pull request authors: 1
Average comments per issue: 0
Average comments per pull request: 1.4
Merged pull requests: 0
Bot issues: 0
Bot pull requests: 10

View more stats

Top Authors

Issue Authors

jmp75 (5)

Pull Request Authors

dependabot[bot] (22)

Top Labels

Issue Labels

enhancement (2)

Pull Request Labels

dependencies (22)

Packages

Total packages: 2
Total downloads:
- pypi 109 last-month

Total dependent packages: 0
(may contain duplicates)
Total dependent repositories: 2
(may contain duplicates)
Total versions: 20
Total maintainers: 1

pypi.org: refcount

Python classes for reference counting

Homepage: https://csiro-hydroinformatics.github.io/pyrefcount
Documentation: https://csiro-hydroinformatics.github.io/pyrefcount
License: MIT
Latest release: 1.2.7
published about 1 year ago

Versions: 17
Dependent Packages: 0
Dependent Repositories: 1
Downloads: 109 Last month

Rankings

Dependent packages count: 10.0%

Dependent repos count: 21.7%

Downloads: 21.8%

Average: 23.0%

Forks count: 29.8%

Stargazers count: 31.9%

Maintainers (1)

jmp_oz

Last synced: 6 months ago

conda-forge.org: refcount

This package helps you achieve reliable management of memory allocated in native libraries, written for instance in C++. While it boils down to "simply" maintaining a set of counters, it is deceptively complicated to do so properly and not end up with memory leaks or crashes. <https://pyrefcount.readthedocs.io/en/latest/>.

Homepage: https://github.com/csiro-hydroinformatics/pyrefcount
License: BSD-3-Clause
Latest release: 1.1.1
published over 3 years ago

Versions: 3
Dependent Packages: 0
Dependent Repositories: 1

Rankings

Dependent repos count: 24.3%

Average: 51.4%

Dependent packages count: 51.6%

Stargazers count: 63.6%

Forks count: 66.1%

Last synced: 6 months ago

Dependencies

docs/requirements.txt pypi

cffi *
mkdocs *
mkdocs-material *
mkdocs-material-extensions *
mkdocstrings *
mkdocstrings-python *
mock *
pillow *
recommonmark *
six *

poetry.lock pypi

atomicwrites 1.4.1 develop
attrs 22.1.0 develop
bleach 5.0.1 develop
cached-property 1.5.2 develop
certifi 2022.6.15 develop
charset-normalizer 2.1.0 develop
click 8.1.3 develop
click-log 0.4.0 develop
colorama 0.4.5 develop
coverage 6.4.3 develop
cryptography 37.0.4 develop
docutils 0.19 develop
dotty-dict 1.3.1 develop
ghp-import 2.1.0 develop
gitdb 4.0.9 develop
gitpython 3.1.27 develop
griffe 0.22.0 develop
idna 3.3 develop
importlib-metadata 4.12.0 develop
iniconfig 1.1.1 develop
invoke 1.7.1 develop
jeepney 0.8.0 develop
jinja2 3.1.2 develop
keyring 23.8.2 develop
markdown 3.3.7 develop
markupsafe 2.1.1 develop
mergedeep 1.3.4 develop
mkdocs 1.3.1 develop
mkdocs-autorefs 0.4.1 develop
mkdocstrings 0.19.0 develop
mkdocstrings-python 0.7.1 develop
mypy 0.971 develop
mypy-extensions 0.4.3 develop
packaging 21.3 develop
pkginfo 1.8.3 develop
pluggy 1.0.0 develop
py 1.11.0 develop
pydantic 1.9.2 develop
pygments 2.12.0 develop
pymdown-extensions 9.5 develop
pyparsing 3.0.9 develop
pytest 7.1.2 develop
pytest-cov 3.0.0 develop
python-dateutil 2.8.2 develop
python-gitlab 3.8.1 develop
python-semantic-release 7.31.2 develop
pywin32-ctypes 0.2.0 develop
pyyaml 6.0 develop
pyyaml-env-tag 0.1 develop
readme-renderer 36.0 develop
requests 2.28.1 develop
requests-toolbelt 0.9.1 develop
rfc3986 2.0.0 develop
secretstorage 3.3.2 develop
semver 2.13.0 develop
six 1.16.0 develop
smmap 5.0.0 develop
tomli 2.0.1 develop
tomlkit 0.10.2 develop
tqdm 4.64.0 develop
twine 3.8.0 develop
typed-ast 1.5.4 develop
typing-extensions 4.3.0 develop
urllib3 1.26.11 develop
watchdog 2.1.9 develop
webencodings 0.5.1 develop
zipp 3.8.1 develop
cffi 1.15.1
pycparser 2.21

pyproject.toml pypi

mkdocs ^1.3.0 develop
mkdocs-autorefs ^0.4.1 develop
mkdocstrings ^0.19.0 develop
mkdocstrings-python ^0.7.1 develop
mypy ^0.971 develop
pydantic ^1.9.1 develop
pytest ^7.1.2 develop
pytest-cov ^3.0.0 develop
python-semantic-release ^7.29.7 develop
cffi ^1.11.5
python ^3.7

requirements.txt pypi

cffi >=1.11.5

.github/workflows/build-matrix.yml actions

actions/checkout v3 composite
actions/setup-python v3 composite
codecov/codecov-action v3 composite

setup.py pypi

docs/environment.yml conda

cffi
mkdocs
mkdocstrings
mkdocstrings-python

https://github.com/csiro-hydroinformatics/pyrefcount

Science Score: 23.0%

Keywords

Keywords from Contributors

Repository

Basic Info

Statistics

Topics

Metadata Files

README.md

refcount - Python classes for reference counting

License

Documentation

Source code

Installation

conda-forge

pypi

From source (development)

Sample use

define TESTDOGPTR testnative::dog*

define TESTOWNERPTR testnative::owner*

define TESTCOUNTEDPTR testnative::reference_counter*

etc.

Related work

Ancestry, acknowledgements

Other python packages

Development branch

Owner

GitHub Events

Total

Last Year

Committers

All Time

Past Year

Top Committers

Committer Domains (Top 20 + Academic)

Issues and Pull Requests

All Time

Past Year

Top Authors

Issue Authors

Pull Request Authors

Top Labels

Issue Labels

Pull Request Labels

Packages

pypi.org: refcount

Rankings

Maintainers (1)

conda-forge.org: refcount

Rankings

Dependencies