https://github.com/donkomura/fsspec-chfs
fsspec implementations for CHFS
Science Score: 13.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
○.zenodo.json file
-
○DOI references
-
○Academic publication links
-
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (7.2%) to scientific vocabulary
Keywords
Repository
fsspec implementations for CHFS
Basic Info
Statistics
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
- Releases: 0
Topics
Metadata Files
README.md
fsspec-chfs
fsspec-chfs is a file system interface to CHFS. CHFS is a parallel consistent hashing file system created instantly using node-local storages such as persistent memory and NVMe SSD for high performance computing. This repository includes the integration for Dask.
Requirements
Getting Started
Installation
$ pip install fsspec-chfs
Create file system
You can create CHFS by chfsctl and set CHFS_SERVER environmental variable.
$ eval `chfsctl start`
$ chlist # show started servers
How to use fsspec-chfs
```python import fsspec
fs = fsspec.filesystem('chfs')
with fs.open('/hello') as f: f.write(b'world') ```
Use in Dask
fsspec-chfs provides CHFSClientDaemon plugin for Dask worker, and it optimizes CHFS initialization/termination in Dask.
python
client = Client(LocalCluster())
plugin = CHFSClientDaemon()
client.register_worker_plugin(plugin)
def func(path, data):
fs = fsspec.filesystem("chfs_stub")
fs.pipe(path, data)
return 0
future = client.submit(func, "/tmp/foo", b'abcde')
counts = future.result()
Developing
VSCode devcontainer
You can use VSCode devcontainer to develop fsspec-chfs. The setup steps are follows:
- Install Docker and Remote-Container extension.
- Press Ctrl+Shift+P in VSCode.
- select
Remote-Container: Open the folder in the Container
Testing
$ eval `chfsctl start` # start the server and set CHFS_SERVER
$ tox
Owner
- Name: donko
- Login: donkomura
- Kind: user
- Location: Japan, Tsukuba
- Repositories: 2
- Profile: https://github.com/donkomura
GitHub Events
Total
Last Year
Packages
- Total packages: 1
-
Total downloads:
- pypi 7 last-month
- Total dependent packages: 0
- Total dependent repositories: 0
- Total versions: 1
- Total maintainers: 1
pypi.org: fsspec-chfs
Pythonic file system for CHFS
- Homepage: https://github.com/donkomura/fsspec-chfs
- Documentation: https://fsspec-chfs.readthedocs.io/
- License: mit
-
Latest release: 0.0.1
published over 3 years ago
Rankings
Maintainers (1)
Dependencies
- koyamaso/chfs master build
- pychfs >=0.0.1