New features

Support __len__ for SPDL dataloader by @vbourgin in https://github.com/facebookresearch/spdl/pull/839
Add SizedIterable[WithShuffle] protocol by @mthrok in https://github.com/facebookresearch/spdl/pull/840
Move DistributedSampler to SPDL core by @vbourgin in https://github.com/facebookresearch/spdl/pull/841
Move load_npy to C++ by @mthrok in https://github.com/facebookresearch/spdl/pull/849
Use libdeflate for loading compressed NumPy Z file. by @mthrok in https://github.com/facebookresearch/spdl/pull/855
Add transfer_tensor function by @mthrok in https://github.com/facebookresearch/spdl/pull/853

What's Changed

Release the GIL when flushing decoder by @mthrok in https://github.com/facebookresearch/spdl/pull/828
Fix embed_shuffle by @moto-meta in https://github.com/facebookresearch/spdl/pull/857

Documentation updates

Add intro to async doc by @mthrok in https://github.com/facebookresearch/spdl/pull/834

BC-breaking changes

Remove deprecated feature by @mthrok in https://github.com/facebookresearch/spdl/pull/832

Full Changelog: https://github.com/facebookresearch/spdl/compare/v0.1.1...v0.1.2

- Python
Published by mthrok 6 months ago

BC-breaking changes

Replace run_pipeline_in_subprocess impl by @moto-meta in https://github.com/facebookresearch/spdl/pull/780

Previously run_pipeline_in_subprocess returned an iterator and was not reusable. Now it returns an iterable, which can be iterated multiple times. Following this change, the object returned from run_pipeline_in_subprocess does not provide iterator-specific method such as next

Example

```python # Construct a builder builder = ( spdl.pipeline.PipelineBuilder() .addsource(...) .pipe(...) ... .addsink(...) )

# Move it to the subprocess, build the Pipeline iterable = runpipelinein_subprocess(builder, ...)

# Iterate - epoch 0 for item in iterable: ...

# Iterate - epoch 1 for item in iterable: ... ```

What's Changed

Support multiple initializers in iterate_in_subprocess and run_pipeline_insubprocess by @mthrok in https://github.com/facebookresearch/spdl/pull/783 Now you can pass multiple initializer functions to iterate_in_subprocess and run_pipeline_insubprocess which are called in the subprocess before the first iteration starts.
Allow Zero Weights in MergeIterator by @vbourgin in https://github.com/facebookresearch/spdl/pull/784 Previously, the MergeIterator did not allow 0 value as weights. Now this is relaxed and you can pass 0. The iterators with 0 weight are not iterated.
Shuffle Before Iterating by Default for embed_shuffle by @vbourgin in https://github.com/facebookresearch/spdl/pull/785 Previously, embed_shuffle defaulted to shuffle after each iteration, but this was counter-intuitive. Now the default behavior is to shuffle before each iteration.
Adhere to PEP 561 (Fixes a mypy bug) by @alxmrs in https://github.com/facebookresearch/spdl/pull/790 Now SPDL package contain py.typed file so that type-checkers can analyze the code.
Ensure core bindings is loaded when loading CUDA binding by @mthrok in https://github.com/facebookresearch/spdl/pull/792 When loading CUDA extension in spdl.io module, it ensures that the CPU extension is loaded.
Support batch loading images with NVJPEG by @mthrok in https://github.com/facebookresearch/spdl/pull/794 The spdl.io.decode_image_nvjpeg function supports loading multiple images. (Note that the function is still experimental.)
Enable compilation warnings by @mthrok in https://github.com/facebookresearch/spdl/pull/806, https://github.com/facebookresearch/spdl/pull/821 The C++ code of spdl.io is hardened by turning few selected compiler warnings into error.
Fix nanobind option for archive module by @mthrok in https://github.com/facebookresearch/spdl/pull/814 The extension module for archive (zip) parsing was not compiled with free-threading support.
Mypy type-checking feedback for all "pyre safe" sources. by @alxmrs in https://github.com/facebookresearch/spdl/pull/801 Now CI mypy type checking. The mypy compatibility is not enforced at the moment as pyre has been (and still is) used. We plan to gradually make the codebase compatible with mypy.

New Contributors

@alxmrs made their first contribution in https://github.com/facebookresearch/spdl/pull/790

Full Changelog: https://github.com/facebookresearch/spdl/compare/v0.1.0...v0.1.1

- Python
Published by mthrok 8 months ago

spdl - v0.1.0

What's Changed

[BC-breaking] Replace load_npz implementation by @mthrok in https://github.com/facebookresearch/spdl/pull/739
[BC-breaking] Remove fallback for API transition by @mthrok in https://github.com/facebookresearch/spdl/pull/770
[BC-breaking] Replace iterateinsubprocess implementation by https://github.com/facebookresearch/spdl/pull/776 @mthrok in https://github.com/facebookresearch/spdl/pull/779
Add support for DEFLATE algorithm by @mthrok in https://github.com/facebookresearch/spdl/pull/744
Name threads by @mthrok in https://github.com/facebookresearch/spdl/pull/728
Update load_npy to accept memoryview by @mthrok in https://github.com/facebookresearch/spdl/pull/738
Remove libzip dependency by @mthrok in https://github.com/facebookresearch/spdl/pull/746
Default to the default context by @mthrok in https://github.com/facebookresearch/spdl/pull/762
Revise the way periodic task is terminated by @mthrok in https://github.com/facebookresearch/spdl/pull/768
Add the funtionality to suppress repeated errors in spdl by @gregorpm in https://github.com/facebookresearch/spdl/pull/773
Add function to convert IterableWithShuffle to Iterable by @mthrok in https://github.com/facebookresearch/spdl/pull/775, https://github.com/facebookresearch/spdl/pull/777

Examples

Add example to benchmark IO functions by @mthrok in https://github.com/facebookresearch/spdl/pull/730

Docs

Add section on data format by @mthrok in https://github.com/facebookresearch/spdl/pull/729
Introduce Case Studies by @mthrok in https://github.com/facebookresearch/spdl/pull/731, https://github.com/facebookresearch/spdl/pull/741, https://github.com/facebookresearch/spdl/pull/754, https://github.com/facebookresearch/spdl/pull/755
Add caveats and update links by @mthrok in https://github.com/facebookresearch/spdl/pull/752, https://github.com/facebookresearch/spdl/pull/751

Full Changelog: https://github.com/facebookresearch/spdl/compare/v0.0.14...v0.1.0

- Python
Published by mthrok 9 months ago

spdl - v0.0.14

Documentation Update

Improved the documentation. The SPDL was started as a experiment for multi-threading, but now it's not just multi-threading but also multi-processing. It's more about production readiness, iterative optimization enabled by observability and flexibility. The Overview and Getting Started have been updated to reflect this
Optimization Guide https://facebookresearch.github.io/spdl/0.0.14/performance_analysis/index.html In the new Optimization Guide section, we share the methodologies we develop while optimizing various production pipelines. It illustrates how we estimate the data loading inefficiency, identify the bottleneck, and how we approach solving the bottleneck.

BC-breaking Changes

Propagate severe Pipeline errors to front-end by @mthrok in https://github.com/facebookresearch/spdl/pull/699 When Pipeline encounters an unexpected error, in addition to shutting down the pipeline, the error is now propagated to the foreground.

The unexpected error includes an error occurred in source iterator. The errors happen in functions provided by users to Pipeline.pipe are expected, so they are not part of it.

Bugfixes

Fix BC function name by @mthrok in https://github.com/facebookresearch/spdl/pull/701

Other Changes

Increase the pipe output buffer size to 2 by @mthrok in https://github.com/facebookresearch/spdl/pull/700
Do not auto-populate stop_after by @mthrok in https://github.com/facebookresearch/spdl/pull/702
Make disaggregate to async by @mthrok in https://github.com/facebookresearch/spdl/pull/703

Full Changelog: https://github.com/facebookresearch/spdl/compare/v0.0.13...v0.0.14

- Python
Published by mthrok 10 months ago

spdl - v0.0.13 - Observability Improvment

Better observability towards bottleneck detection

In this release, we re-organized the pipeline hooks, and expose the periodic performance measurement mechanism as public interface. This allows to tap into the performance statistics of data processing pipeline, which makes it easy to analyze the pipeline performance, and optimize the pipeline.

Please checkout the renewed Performance Analysis documentation, and example to learn how to log the runtime performance statistics.

BC-breaking Changes

Merge QueueClass and hook creation interface by @moto-meta in https://github.com/facebookresearch/spdl/pull/691

New Features

Expose stats logging as public API by @mthrok in https://github.com/facebookresearch/spdl/pull/686
Add performance analysis example by @mthrok in https://github.com/facebookresearch/spdl/pull/694
Add performance analysis docs by @mthrok in https://github.com/facebookresearch/spdl/pull/693

Full Changelog: https://github.com/facebookresearch/spdl/compare/v0.0.12...v0.0.13

- Python
Published by mthrok 10 months ago

spdl - v0.0.12

New features

Improved streaming media processing capability, including multi-stream demuxing, remuxing, encoding and complex filter graph capability.
- You can demux multiple streams from the source. (such as audio and video packets.)
- With the new Muxer class, you can encode media or remux packets.
- Please check out the streaming_video_processing.py for audio remuxing, streaming video decoding, and encoding.
The new FilterGraph class can perform complex filtering, multiple-input, multiple-output and multi-media processing. One popular example is overlay to add watermark on videos processed by AI. Please checkout the documentation.

What's Changed

Rename BSF implementation by @mthrok in https://github.com/facebookresearch/spdl/pull/631
Add tests for codec by @mthrok in https://github.com/facebookresearch/spdl/pull/632
Add BSF class by @mthrok in https://github.com/facebookresearch/spdl/pull/634
Replace the implementation of BSF for nvdec by @mthrok in https://github.com/facebookresearch/spdl/pull/635
Add method to query stream index by @mthrok in https://github.com/facebookresearch/spdl/pull/636
Support multi-stream demuxing by @mthrok in https://github.com/facebookresearch/spdl/pull/637
Add duration option to multi-stream demuxing by @mthrok in https://github.com/facebookresearch/spdl/pull/638
Do not run packaging jobs on main branch after merge by @mthrok in https://github.com/facebookresearch/spdl/pull/644
Add channellayout/sampleaspect_ratio attributes by @mthrok in https://github.com/facebookresearch/spdl/pull/640
Tweak test util for consistency by @mthrok in https://github.com/facebookresearch/spdl/pull/641
Add default stream by @mthrok in https://github.com/facebookresearch/spdl/pull/642
Returns None if there is no packet/frames by @mthrok in https://github.com/facebookresearch/spdl/pull/643
Add filter graph public class by @mthrok in https://github.com/facebookresearch/spdl/pull/639
Fix issues on encoding by @mthrok in https://github.com/facebookresearch/spdl/pull/646
Improve error message by @mthrok in https://github.com/facebookresearch/spdl/pull/647
Add audio encode config by @mthrok in https://github.com/facebookresearch/spdl/pull/648
Add video encode config by @mthrok in https://github.com/facebookresearch/spdl/pull/649
Add muxer by @mthrok in https://github.com/facebookresearch/spdl/pull/645
Add audio encoding by @mthrok in https://github.com/facebookresearch/spdl/pull/650
Add video encoding by @mthrok in https://github.com/facebookresearch/spdl/pull/651
Deprecate streamingdemuxvideo by @mthrok in https://github.com/facebookresearch/spdl/pull/653
Remove obsolete function by @mthrok in https://github.com/facebookresearch/spdl/pull/652
Return packets directly when stream is specified with single int by @mthrok in https://github.com/facebookresearch/spdl/pull/654
Add streaming video processing example by @mthrok in https://github.com/facebookresearch/spdl/pull/655
Update mermaid js version by @mthrok in https://github.com/facebookresearch/spdl/pull/658
Do not run lint job on master branch by @mthrok in https://github.com/facebookresearch/spdl/pull/659
Set cancellation by @mthrok in https://github.com/facebookresearch/spdl/pull/660
Fix doc by @mthrok in https://github.com/facebookresearch/spdl/pull/661
Bump version by @mthrok in https://github.com/facebookresearch/spdl/pull/662
Update doc by @mthrok in https://github.com/facebookresearch/spdl/pull/663
[BC-Breaking] Replace encodeimage with saveimage by @mthrok in https://github.com/facebookresearch/spdl/pull/656
Merge the internal implementation of FilterGraph by @mthrok in https://github.com/facebookresearch/spdl/pull/657
Get rid of custom source and use existing url attr by @mthrok in https://github.com/facebookresearch/spdl/pull/664
Remove wheel from build dep by @mthrok in https://github.com/facebookresearch/spdl/pull/665
Retrieve proper time base from BSF by @mthrok in https://github.com/facebookresearch/spdl/pull/666
Ensure no new Python download by @mthrok in https://github.com/facebookresearch/spdl/pull/667
Merge frame_rate impl by @mthrok in https://github.com/facebookresearch/spdl/pull/668
Improve the error message when using subprocess by @mthrok in https://github.com/facebookresearch/spdl/pull/669
Match the signature default value to usecase by @mthrok in https://github.com/facebookresearch/spdl/pull/670
Add pipeline ID to stage names by @mthrok in https://github.com/facebookresearch/spdl/pull/671
Add queue occupation rate log by @moto-meta in https://github.com/facebookresearch/spdl/pull/672
Update docs by @mthrok in https://github.com/facebookresearch/spdl/pull/673
Default to build nvdec by @mthrok in https://github.com/facebookresearch/spdl/pull/677
Set CMAKEBUILDTYPE=Release by @mthrok in https://github.com/facebookresearch/spdl/pull/676

Full Changelog: https://github.com/facebookresearch/spdl/compare/v0.0.11...v0.0.12

- Python
Published by mthrok 10 months ago

spdl - v0.0.11

New features

Add streaming NVDEC decoding example https://github.com/facebookresearch/spdl/pull/571
Add streamingloadvideo_nvdec func https://github.com/facebookresearch/spdl/pull/538
Use None for skipping item in pipeline https://github.com/facebookresearch/spdl/pull/545
Add codec metadata https://github.com/facebookresearch/spdl/pull/569
Add stream sync https://github.com/facebookresearch/spdl/pull/537
Add streaming decoder https://github.com/facebookresearch/spdl/pull/553, https://github.com/facebookresearch/spdl/pull/566
Add timestamp to frames __repr__ https://github.com/facebookresearch/spdl/pull/586
Add function to convert RGB video tensor to Frames https://github.com/facebookresearch/spdl/pull/587

The following new features will be available in the next release.

Add video encoder https://github.com/facebookresearch/spdl/pull/592
Add Muxer implementation https://github.com/facebookresearch/spdl/pull/594
Add audio encoding https://github.com/facebookresearch/spdl/pull/597
Add colorspace options https://github.com/facebookresearch/spdl/pull/603
Add time_base attributes to Audio/Video codec https://github.com/facebookresearch/spdl/pull/604
Add end time in Frames timestamp representation https://github.com/facebookresearch/spdl/pull/605
Add FilterGraph public C++ interface https://github.com/facebookresearch/spdl/pull/606
Add stream_index to Packets https://github.com/facebookresearch/spdl/pull/626

BC-breaking changes

[BC-breaking] Remove streamingdecodepackets https://github.com/facebookresearch/spdl/pull/562
[BC-Breaking] Do not copy attach codec when stream demuxing https://github.com/facebookresearch/spdl/pull/621

Fixes

Fix nvJPEG build https://github.com/facebookresearch/spdl/pull/546

Test/CI

Use ffmpeg 7.0.2 for build https://github.com/facebookresearch/spdl/pull/568
Add encoder parity test https://github.com/facebookresearch/spdl/pull/584

Full Changelog: https://github.com/facebookresearch/spdl/compare/v0.0.10...v0.0.11

- Python
Published by mthrok 11 months ago

spdl - v0.0.10

Summary

You can now install SPDL with pip install spdl, and use CUDA and NVDEC.
- The extension module has been split into core and CUDA. Now it's possible to use CUDA package in environments without CUDA.
- The PyPI distributions for Linux x86_64 includes CUDA extension.
NVEDC support has been added to CUDA extension. See the doc for the detail.

New Features

Added spdl.io.apply_bsf to apply bit-stream filtering to packets (https://github.com/facebookresearch/spdl/pull/522, https://github.com/facebookresearch/spdl/pull/503)
Added function to fetch FFmpeg versions (https://github.com/facebookresearch/spdl/pull/507)
Added NV12 to RGB color conversion functions (https://github.com/facebookresearch/spdl/pull/528, https://github.com/facebookresearch/spdl/pull/525, https://github.com/facebookresearch/spdl/pull/526)
Added a new NvDecDecoder, which supports streaming decoding. (https://github.com/facebookresearch/spdl/pull/517, https://github.com/facebookresearch/spdl/pull/520, https://github.com/facebookresearch/spdl/pull/531)
Added support to fetch codec from Packets and Demuxer. (https://github.com/facebookresearch/spdl/pull/521)

BC-Breaking changes

Removed alpha channel from NVDEC decoding (https://github.com/facebookresearch/spdl/pull/527)
The HOLD_GIL build option has been removed (https://github.com/facebookresearch/spdl/pull/509, https://github.com/facebookresearch/spdl/pull/511)
Renamed the build config query functions is_available to built_with (https://github.com/facebookresearch/spdl/pull/514, https://github.com/facebookresearch/spdl/pull/505)

Bugfix

Fixed the allocator function signature by @mthrok in https://github.com/facebookresearch/spdl/pull/504
Fixed segfault when not built with CUDA by @mthrok in https://github.com/facebookresearch/spdl/pull/513

Split code/binding/packages

Split binding by @mthrok in https://github.com/facebookresearch/spdl/pull/496
Do not link CUDA to SPDL core binding by @mthrok in https://github.com/facebookresearch/spdl/pull/508

Refactoring CUDA-related code

Use spdl::cuda namespace for CUDA-related features by @mthrok in https://github.com/facebookresearch/spdl/pull/495
Reduce FFmpeg redundancy from CUDA extension by @mthrok in https://github.com/facebookresearch/spdl/pull/498
Do not pass AVCodecID directly by @mthrok in https://github.com/facebookresearch/spdl/pull/499
Do not link ffmpeg when building CUDA ext by @mthrok in https://github.com/facebookresearch/spdl/pull/502
Use iterator by @mthrok in https://github.com/facebookresearch/spdl/pull/506
Fix CUDA availability by @mthrok in https://github.com/facebookresearch/spdl/pull/512
Define CUDAStoragePtr by @mthrok in https://github.com/facebookresearch/spdl/pull/519
Move CUDA utils to detail by @mthrok in https://github.com/facebookresearch/spdl/pull/530
Add missing glog header inclusion to logging by @mthrok in https://github.com/facebookresearch/spdl/pull/524
Add constant accessor to CUDABuffer by @mthrok in https://github.com/facebookresearch/spdl/pull/523
Remove stray defs by @mthrok in https://github.com/facebookresearch/spdl/pull/515
Remove AVCodec from CUDA extension by @mthrok in https://github.com/facebookresearch/spdl/pull/501
Define CUDAAPIPERTHREADDEFAULT_STREAM by @mthrok in https://github.com/facebookresearch/spdl/pull/518
Clean up CUDA initialization logic by @mthrok in https://github.com/facebookresearch/spdl/pull/529

CI Improvement

Run NVDEC tests by @mthrok in https://github.com/facebookresearch/spdl/pull/500
Merge packages by @mthrok in https://github.com/facebookresearch/spdl/pull/510

Full Changelog: https://github.com/facebookresearch/spdl/compare/v2025.03.17.00...v0.0.10

- Python
Published by mthrok 11 months ago

spdl - v0.0.9

What's Changed

The FFmpeg7 integration has been fixed.

Add version attribute by @mthrok in https://github.com/facebookresearch/spdl/pull/412
Append queue name with index by @mthrok in https://github.com/facebookresearch/spdl/pull/418
Prefix the queue/stage order by @mthrok in https://github.com/facebookresearch/spdl/pull/421
Fix FFmpeg7 Integration by @mthrok in https://github.com/facebookresearch/spdl/pull/189

Full Changelog: https://github.com/facebookresearch/spdl/compare/v0.0.8...v0.0.9

Please checkout the Assets bellow to download CUDA binaries.

- Python
Published by mthrok 12 months ago

spdl - Release Assets

- Python
Published by mthrok 12 months ago

spdl - v0.0.8 - The first (yet experimental) release

SPDL is now available on PyPI for the following platform and Python versions. You can install it with pip install spdl.

Linux/macOS
Python 3.10, 3.11, 3.12 and 3.13

There is no binary distribution for 3.13t yet.

The IO module with CUDA support is not available on PyPI. Instead you can download wheels compiled with CUDA 12.6 from the following Assets section. You need to install nvidia-cuda-runtime-cu12 package. (It's available in https://pypi.org/project/nvidia-cuda-runtime-cu12/ or
if you are installing PyTorch following from the official channel, it will be automatically installed.)

The spdl distribution is an empty meta package. The actual functionalities are split into spdl_io and spdl_core packages. They are independent each other and can be installed separately.

For the detail of package usage, please refer to the documentation. https://facebookresearch.github.io/spdl/0.0.8/index.html

- Python
Published by mthrok 12 months ago

Recent Releases of spdl

spdl - v0.1.2

New features

What's Changed

Documentation updates

BC-breaking changes

spdl - v0.1.1

BC-breaking changes

What's Changed

New Contributors

spdl - v0.1.0

What's Changed

Examples

Docs

spdl - v0.0.14

Documentation Update

BC-breaking Changes

Bugfixes

Other Changes

spdl - v0.0.13 - Observability Improvment

Better observability towards bottleneck detection

BC-breaking Changes

New Features

spdl - v0.0.12

New features

What's Changed

spdl - v0.0.11

New features

BC-breaking changes

Fixes

Test/CI

spdl - v0.0.10

Summary

New Features

BC-Breaking changes

Bugfix

Split code/binding/packages

Refactoring CUDA-related code

CI Improvement

spdl - v0.0.9

What's Changed

spdl - Release Assets

spdl - v0.0.8 - The first (yet experimental) release