Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/awslabs/soci-snapshotter

A containerd snapshotter plugin which enables standard OCI images to be lazily loaded without requiring a build-time conversion step.

https://github.com/awslabs/soci-snapshotter

Science Score: 26.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ○
    Committers with academic emails
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (11.7%) to scientific vocabulary

Keywords from Contributors

generation embeddings chart vulnerabilities excel archival observability interactive projection sequences
Last synced: 10 months ago · JSON representation

Repository

A containerd snapshotter plugin which enables standard OCI images to be lazily loaded without requiring a build-time conversion step.

Basic Info
  • Host: GitHub
  • Owner: awslabs
  • License: apache-2.0
  • Language: Go
  • Default Branch: main
  • Homepage:
  • Size: 8.44 MB
Statistics
  • Stars: 637
  • Watchers: 22
  • Forks: 66
  • Open Issues: 136
  • Releases: 15
Created almost 4 years ago · Last pushed 11 months ago
Metadata Files
Readme Contributing License Code of conduct Codeowners

README.md

SOCI Snapshotter

PkgGoDev Go Report Card Build Static Badge

SOCI Snapshotter is a containerd snapshotter plugin. It enables standard OCI images to be lazily loaded without requiring a build-time conversion step. "SOCI" is short for "Seekable OCI", and is pronounced "so-CHEE".

The standard method for launching containers starts with a setup phase during which the container image data is completely downloaded from a remote registry and a filesystem is assembled. The application is not launched until this process is complete. Using a representative suite of images, Harter et al FAST '16 found that image download accounts for 76% of container startup time, but on average only 6.4% of the fetched data is actually needed for the container to start doing useful work.

One approach for addressing this is to eliminate the need to download the entire image before launching the container, and to instead lazily load data on demand, and also prefetch data in the background.

Design considerations

No image conversion

NOTE This section describes SOCI Index Manifest v1. While the lack of an image conversion step is appealing, for production scenarios it also creates the potential for performance changes across multiple dimensions if an index is added to or removed from an existing image that is widely deployed. To address these downsides, we introduced SOCI Index Manifest v2 which does use a build-time conversion step. For more information see the SOCI Index Manifest v2 documentation

Existing lazy loading snapshotters rely on a build-time conversion step, to produce a new image artifact. This is problematic for container developers who won't or can't modify their CI/CD pipeline, or don't want to manage the cost and complexity of keeping copies of images in two formats. It also creates problems for image signing, since the conversion step invalidates any signatures that were created against the original OCI image.

SOCI addresses these issues by loading from the original, unmodified OCI image. Instead of converting the image, it builds a separate index artifact (the "SOCI index"), which lives in the remote registry, right next to the image itself. At container launch time, SOCI Snapshotter queries the registry for the presence of the SOCI index using the mechanism developed by the OCI Reference Types working group.

Workload-specific load order optimization

Another big consideration that we haven't implmented/integrated into SOCI is to image load order based on your specific workload. See design README for more details.

Documentation

Integration-specific documentation

Project Origin

There a few different lazy loading projects in the containerd snapshotter community. This project began as a fork of the popular Stargz-snapshotter project from commit 743e5e70a7fdec9cd4ab218e1d4782fbbd253803 with the intention of an upstream patch. During development the changes were fundamental enough that the decision was made to create soci-snapshotter as a standalone project. Soci-snapshotter builds on stargz's success and innovative ideas. Long term, this project intends and hopes to join containerd as a non-core project and intends to follow CNCF best practices.

Owner

  • Name: Amazon Web Services - Labs
  • Login: awslabs
  • Kind: organization
  • Location: Seattle, WA

AWS Labs

Committers

Last synced: about 2 years ago

All Time
  • Total Commits: 549
  • Total Committers: 34
  • Avg Commits per committer: 16.147
  • Development Distribution Score (DDS): 0.867
Past Year
  • Commits: 241
  • Committers: 20
  • Avg Commits per committer: 12.05
  • Development Distribution Score (DDS): 0.776
Top Committers
Name Email Commits
Kern Walster w****r@a****m 73
Jin Dong j****n@a****m 61
David Son d****n@a****m 54
dependabot[bot] 4****] 48
Yasin Turan t****n@a****m 42
Clarence "Sparr" Risher c****c@a****m 36
Rishabh Singhvi r****n@a****m 36
GitHub n****y@g****m 32
Austin Vazquez m****v@a****m 32
Viktor Kuznietsov v****t@a****m 30
Arjun Yogidas a****y@a****m 22
Hanyue Liang h****g@a****m 15
Kazuyoshi Kato k****u@a****m 13
Scott Buckfelder b****t@a****m 12
djdongjin d****n 9
Wayne Mesard w****m@a****m 6
Tuan Anh Tran me@t****g 5
Dave Nguyen d****n@a****m 3
Davanum Srinivas d****m@g****m 2
Henry Wang h****g@a****m 2
Iain Macdonald x****x@g****m 2
Scott Haddlesey h****t@a****m 2
Akihiro Suda a****z@h****p 1
Rishabh Singhvi 8****n 1
Eng Zer Jun e****n@g****m 1
Amazon GitHub Automation 5****o 1
Carl Hiltbrunner c****t@a****m 1
Kohei Tokunaga k****l@g****m 1
Manuj Grover m****1@g****m 1
Phil Estes e****p@g****m 1
and 4 more...
Committer Domains (Top 20 + Academic)
amazon.com: 18 vpbank.com.vn: 1 dominodatalab.com: 1 gopaddy.ch: 1 hco.ntt.co.jp: 1 tuananh.org: 1 github.com: 1

Issues and Pull Requests

Last synced: 10 months ago

All Time
  • Total issues: 118
  • Total pull requests: 727
  • Average time to close issues: 6 months
  • Average time to close pull requests: 9 days
  • Total issue authors: 29
  • Total pull request authors: 28
  • Average comments per issue: 1.08
  • Average comments per pull request: 0.86
  • Merged pull requests: 287
  • Bot issues: 1
  • Bot pull requests: 414
Past Year
  • Issues: 21
  • Pull requests: 282
  • Average time to close issues: 13 days
  • Average time to close pull requests: 8 days
  • Issue authors: 13
  • Pull request authors: 13
  • Average comments per issue: 0.33
  • Average comments per pull request: 1.02
  • Merged pull requests: 95
  • Bot issues: 1
  • Bot pull requests: 168
View more stats
Top Authors
Issue Authors
  • Kern-- (36)
  • sparr (18)
  • sondavidb (12)
  • djdongjin (11)
  • austinvazquez (9)
  • debajyoti-truefoundry (3)
  • ollypom (3)
  • turan18 (3)
  • arnaldo2792 (2)
  • coderbirju (2)
  • AkihiroSuda (1)
  • tmokmss (1)
  • rdpsin (1)
  • HusainZafar (1)
  • amalthundiyil (1)
Pull Request Authors
  • dependabot[bot] (334)
  • sondavidb (103)
  • github-actions[bot] (79)
  • austinvazquez (72)
  • Kern-- (55)
  • coderbirju (26)
  • sparr (10)
  • turan18 (8)
  • pendo324 (5)
  • djdongjin (5)
  • Shubhranshu153 (5)
  • haddscot (3)
  • sbuckfelder (3)
  • henry118 (3)
  • swagatbora90 (2)
Top Labels
Issue Labels
feature (68) bug (36) blocked (5) performance (3) kubernetes (3) testing (2) documentation (2) dependencies (2) docker (1) stability (1) tracking issue (1) Kubernetes (1)
Pull Request Labels
dependencies (407) go (243) github_actions (44) testing (40) documentation (14) easy-to-review (10) automated-pr (7) docker (7) benchmarking (3) API Change (3) bug (1) kubernetes (1) feature (1) 📦 deps (1) 🤖 bot (1)

Packages

  • Total packages: 2
  • Total downloads: unknown
  • Total docker downloads: 14,099
  • Total dependent packages: 5
    (may contain duplicates)
  • Total dependent repositories: 6
    (may contain duplicates)
  • Total versions: 152
proxy.golang.org: github.com/awslabs/soci-snapshotter
  • Versions: 70
  • Dependent Packages: 5
  • Dependent Repositories: 6
  • Docker Downloads: 14,099
Rankings
Dependent repos count: 2.1%
Stargazers count: 3.3%
Average: 3.8%
Forks count: 4.5%
Dependent packages count: 5.2%
Last synced: 11 months ago
proxy.golang.org: github.com/awslabs/soci-snapshotter/cmd
  • Versions: 82
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Stargazers count: 2.2%
Forks count: 3.3%
Average: 5.5%
Dependent packages count: 7.0%
Dependent repos count: 9.3%
Last synced: 10 months ago

Dependencies

cmd/go.mod go
  • github.com/Microsoft/go-winio v0.5.1
  • github.com/Microsoft/hcsshim v0.9.4
  • github.com/awslabs/soci-snapshotter v0.0.0-local
  • github.com/awslabs/soci-snapshotterv0.0.0-local=> ../
  • github.com/awslabs/soci-snapshotterv0.0.0=> ../
  • github.com/beorn7/perks v1.0.1
  • github.com/cespare/xxhash/v2 v2.1.2
  • github.com/containerd/cgroups v1.0.3
  • github.com/containerd/console v1.0.3
  • github.com/containerd/containerd v1.6.8
  • github.com/containerd/continuity v0.2.2
  • github.com/containerd/fifo v1.0.0
  • github.com/containerd/go-cni v1.1.6
  • github.com/containerd/ttrpc v1.1.0
  • github.com/containerd/typeurl v1.0.2
  • github.com/containernetworking/cni v1.1.1
  • github.com/containernetworking/plugins v1.1.1
  • github.com/coreos/go-systemd/v22 v22.3.2
  • github.com/cpuguy83/go-md2man/v2 v2.0.0
  • github.com/davecgh/go-spew v1.1.1
  • github.com/docker/cli v20.10.11+incompatible
  • github.com/docker/docker v20.10.7+incompatible
  • github.com/docker/docker-credential-helpers v0.6.4
  • github.com/docker/go-events v0.0.0-20190806004212-e31b211e4f1c
  • github.com/docker/go-metrics v0.0.1
  • github.com/docker/go-units v0.4.0
  • github.com/go-logr/logr v1.2.2
  • github.com/gogo/googleapis v1.4.0
  • github.com/gogo/protobuf v1.3.2
  • github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da
  • github.com/golang/protobuf v1.5.2
  • github.com/google/go-cmp v0.5.6
  • github.com/google/gofuzz v1.2.0
  • github.com/google/uuid v1.2.0
  • github.com/googleapis/gnostic v0.5.5
  • github.com/hanwen/go-fuse/v2 v2.1.1-0.20210825171523-3ab5d95a30ae
  • github.com/hashicorp/errwrap v1.1.0
  • github.com/hashicorp/go-cleanhttp v0.5.1
  • github.com/hashicorp/go-multierror v1.1.1
  • github.com/hashicorp/go-retryablehttp v0.7.0
  • github.com/hashicorp/golang-lru=>github.com/ktock/golang-lru v0.5.5-0.20211029085301-ec551be6f75c
  • github.com/imdario/mergo v0.3.12
  • github.com/json-iterator/go v1.1.12
  • github.com/klauspost/compress v1.13.6
  • github.com/matttproud/golang_protobuf_extensions v1.0.2-0.20181231171920-c182affec369
  • github.com/moby/locker v1.0.1
  • github.com/moby/sys/mountinfo v0.5.0
  • github.com/moby/sys/signal v0.6.0
  • github.com/moby/sys/symlink v0.2.0
  • github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
  • github.com/modern-go/reflect2 v1.0.2
  • github.com/opencontainers/go-digest v1.0.0
  • github.com/opencontainers/image-spec v1.0.3-0.20211202183452-c5a74bcca799
  • github.com/opencontainers/runc v1.1.2
  • github.com/opencontainers/runtime-spec v1.0.3-0.20210326190908-1c3f411f0417
  • github.com/opencontainers/selinux v1.10.1
  • github.com/oras-project/artifacts-spec v1.0.0-draft.1.1
  • github.com/pelletier/go-toml v1.9.4
  • github.com/pkg/errors v0.9.1
  • github.com/prometheus/client_golang v1.11.1
  • github.com/prometheus/client_model v0.2.0
  • github.com/prometheus/common v0.30.0
  • github.com/prometheus/procfs v0.7.3
  • github.com/rs/xid v1.3.0
  • github.com/russross/blackfriday/v2 v2.0.1
  • github.com/shurcooL/sanitized_anchor_name v1.0.0
  • github.com/sirupsen/logrus v1.8.1
  • github.com/spf13/pflag v1.0.5
  • github.com/urfave/cli v1.22.5
  • go.etcd.io/bbolt v1.3.6
  • go.opencensus.io v0.23.0
  • golang.org/x/net v0.0.0-20211216030914-fe4d6282115f
  • golang.org/x/oauth2 v0.0.0-20210819190943-2bc19b11175f
  • golang.org/x/sync v0.0.0-20210220032951-036812b2e83c
  • golang.org/x/sys v0.0.0-20220412211240-33da011f77ad
  • golang.org/x/term v0.0.0-20210615171337-6886f2dfbf5b
  • golang.org/x/text v0.3.7
  • golang.org/x/time v0.0.0-20210723032227-1f47c861a9ac
  • google.golang.org/appengine v1.6.7
  • google.golang.org/genproto v0.0.0-20211208223120-3a66f561d7aa
  • google.golang.org/grpc v1.43.0
  • google.golang.org/protobuf v1.27.1
  • gopkg.in/inf.v0 v0.9.1
  • gopkg.in/yaml.v2 v2.4.0
  • gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b
  • k8s.io/api v0.23.0
  • k8s.io/apimachinery v0.23.0
  • k8s.io/client-go v0.23.0
  • k8s.io/cri-api v0.24.0-alpha.0
  • k8s.io/klog/v2 v2.30.0
  • k8s.io/kube-openapi v0.0.0-20211115234752-e816edb12b65
  • k8s.io/utils v0.0.0-20210930125809-cb0fa318a74b
  • oras.land/oras-go/v2 v2.0.0-20220623152852-2210572939e5
  • sigs.k8s.io/json v0.0.0-20211020170558-c049b76a60c6
  • sigs.k8s.io/structured-merge-diff/v4 v4.1.2
  • sigs.k8s.io/yaml v1.2.0
cmd/go.sum go
  • 1411 dependencies
go.mod go
  • github.com/Microsoft/go-winio v0.5.1
  • github.com/Microsoft/hcsshim v0.9.4
  • github.com/beorn7/perks v1.0.1
  • github.com/cespare/xxhash/v2 v2.1.2
  • github.com/containerd/cgroups v1.0.3
  • github.com/containerd/containerd v1.6.8
  • github.com/containerd/continuity v0.2.2
  • github.com/containerd/ttrpc v1.1.0
  • github.com/containerd/typeurl v1.0.2
  • github.com/davecgh/go-spew v1.1.1
  • github.com/docker/cli v20.10.11+incompatible
  • github.com/docker/docker v20.10.7+incompatible
  • github.com/docker/docker-credential-helpers v0.6.4
  • github.com/docker/go-events v0.0.0-20190806004212-e31b211e4f1c
  • github.com/docker/go-metrics v0.0.1
  • github.com/go-logr/logr v1.2.2
  • github.com/gogo/protobuf v1.3.2
  • github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da
  • github.com/golang/protobuf v1.5.2
  • github.com/google/go-cmp v0.5.6
  • github.com/google/gofuzz v1.2.0
  • github.com/googleapis/gnostic v0.5.5
  • github.com/hanwen/go-fuse/v2 v2.1.1-0.20210825171523-3ab5d95a30ae
  • github.com/hashicorp/errwrap v1.1.0
  • github.com/hashicorp/go-cleanhttp v0.5.1
  • github.com/hashicorp/go-multierror v1.1.1
  • github.com/hashicorp/go-retryablehttp v0.7.0
  • github.com/imdario/mergo v0.3.12
  • github.com/json-iterator/go v1.1.12
  • github.com/klauspost/compress v1.13.6
  • github.com/matttproud/golang_protobuf_extensions v1.0.2-0.20181231171920-c182affec369
  • github.com/moby/locker v1.0.1
  • github.com/moby/sys/mountinfo v0.5.0
  • github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
  • github.com/modern-go/reflect2 v1.0.2
  • github.com/onsi/ginkgo v1.16.4
  • github.com/onsi/gomega v1.15.0
  • github.com/opencontainers/go-digest v1.0.0
  • github.com/opencontainers/image-spec v1.0.3-0.20211202183452-c5a74bcca799
  • github.com/oras-project/artifacts-spec v1.0.0-draft.1.1
  • github.com/pelletier/go-toml v1.9.4
  • github.com/pkg/errors v0.9.1
  • github.com/prometheus/client_golang v1.11.1
  • github.com/prometheus/client_model v0.2.0
  • github.com/prometheus/common v0.30.0
  • github.com/prometheus/procfs v0.7.3
  • github.com/rs/xid v1.3.0
  • github.com/sirupsen/logrus v1.8.1
  • github.com/spf13/pflag v1.0.5
  • go.etcd.io/bbolt v1.3.6
  • go.opencensus.io v0.23.0
  • golang.org/x/crypto v0.0.0-20210817164053-32db794688a5
  • golang.org/x/net v0.0.0-20211216030914-fe4d6282115f
  • golang.org/x/oauth2 v0.0.0-20210819190943-2bc19b11175f
  • golang.org/x/sync v0.0.0-20210220032951-036812b2e83c
  • golang.org/x/sys v0.0.0-20220412211240-33da011f77ad
  • golang.org/x/term v0.0.0-20210615171337-6886f2dfbf5b
  • golang.org/x/text v0.3.7
  • golang.org/x/time v0.0.0-20210723032227-1f47c861a9ac
  • google.golang.org/appengine v1.6.7
  • google.golang.org/genproto v0.0.0-20211208223120-3a66f561d7aa
  • google.golang.org/grpc v1.43.0
  • google.golang.org/protobuf v1.27.1
  • gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c
  • gopkg.in/inf.v0 v0.9.1
  • gopkg.in/yaml.v2 v2.4.0
  • gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b
  • gotest.tools/v3 v3.0.3
  • k8s.io/api v0.23.0
  • k8s.io/apimachinery v0.23.0
  • k8s.io/client-go v0.23.0
  • k8s.io/cri-api v0.24.0-alpha.0
  • k8s.io/klog/v2 v2.30.0
  • k8s.io/kube-openapi v0.0.0-20211115234752-e816edb12b65
  • k8s.io/utils v0.0.0-20210930125809-cb0fa318a74b
  • oras.land/oras-go/v2 v2.0.0-20220622093312-9fc00616c490
  • sigs.k8s.io/json v0.0.0-20211020170558-c049b76a60c6
  • sigs.k8s.io/structured-merge-diff/v4 v4.1.2
  • sigs.k8s.io/yaml v1.2.0
go.sum go
  • 1221 dependencies
.github/workflows/benchmark_visualization.yml actions
  • actions/checkout v4 composite
  • actions/download-artifact v3 composite
  • actions/setup-go v4 composite
  • actions/upload-artifact v3 composite
  • benchmark-action/github-action-benchmark v1 composite
.github/workflows/build.yml actions
  • actions/checkout v4 composite
  • actions/setup-go v4 composite
.github/workflows/bump-deps.yml actions
  • actions/checkout v4 composite
  • actions/setup-go v4 composite
  • peter-evans/create-pull-request v5 composite
.github/workflows/comparision-test.yml actions
  • actions/checkout v4 composite
  • actions/setup-go v4 composite
.github/workflows/prebuild.yml actions
  • actions/checkout v4 composite
  • actions/setup-go v4 composite
Dockerfile docker
  • golang 1.20.6-bookworm build
  • golang-base latest build
  • registry 2 build
.github/workflows/releases.yml actions
  • actions/checkout v4 composite
  • actions/download-artifact v3 composite
  • actions/setup-go v5 composite
  • actions/upload-artifact v3 composite
  • softprops/action-gh-release v1 composite