loompy - Release recent changes

- Python
Published by slinnarsson over 4 years ago

loompy - Remove dependency on h5py 2.10.0

We had acquired a dependency on h5py>=2.10.0 when we introduced variable-length UTF8 string attributes (specifically, we used h5py.string_dtype). This was sad, because scanpy has a dependency h5py!=2.10.0 which makes loompy and scanpy strictly incompatible. However, it turns out there's a way to describe the same thing that works in older versions of h5py (specifically, using special_dtype(vlen=str)). This release uses the old way, and removes the dependency on h5py 2.10.0.

- Python
Published by slinnarsson almost 7 years ago

loompy - Backwards compatibility of validator

Validator now automatically validates against the spec version that the file itself declares (rather than a spec version requested by the caller). This can be overridden by specifying a specific version for validation.

- Python
Published by slinnarsson almost 7 years ago

loompy - Upgraded the validator

The validator was upgraded to conform to the 3.0.0 spec.

- Python
Published by slinnarsson almost 7 years ago

loompy - Bugfixes and spec update

Fixes a couple of minor bugs

• Missing dependency added: numpy_groupies
• Accidental Python 3.7 dependency removed

Updates the Loom v3.0.0 file format specification

- Python
Published by slinnarsson almost 7 years ago

loompy - Loompy 3.0.1

Minor release mainly with updated docs and tutorials for loompy fromfq command-line tool.

- Python
Published by slinnarsson almost 7 years ago

loompy - Loompy 3.0

What's new

• Improved performance
• Full sparse matrix support
• Unlimited size of global attributes
• Variable-length Unicode string attributes
• New command-line tool to create .loom directly from fastq using kallisto

- Python
Published by slinnarsson almost 7 years ago

loompy - Bugfixes and cellranger V3 compatibility

Various bugfixes and improvements

• Support importing cellranger 3.0.0 folders (#84)
• Fixes bugs in sparse array support (#82)
• Makes scan() faster (#90)

- Python
Published by slinnarsson over 7 years ago

loompy - Handle non-standard files with UTF8 strings

Fixes #78 by not crashing when opening a file that has variable-length UTF8-encoded strings (in violation of the Loom specification).

- Python
Published by slinnarsson over 7 years ago

loompy - Minor bugfixes

Bugfixes

• Bug in slicing views using tuples #77
• Fix: when attempting to connect to non-existing file, an empty file would be created

- Python
Published by slinnarsson almost 8 years ago

loompy - Bugfixes, docs

Bugfixes and better docs

Fixes many minor and not so minor bugs.

Greatly improved documentation at loompy.org

- Python
Published by slinnarsson almost 8 years ago

loompy - Updated documentation

Minor release only affecting the docs.

- Python
Published by slinnarsson almost 8 years ago

loompy - More sparsity support, empty layers & experimental plotting features

Any layer can be created from sparse matrix

This now works:

python G = 1000 C = 100 S = sparse.eye(G, C) with loompy.connect("test.loom") as ds: ds["layer"] = S

Fixes #66.

Create empty file

loompy.new() creates an empty loom file, and returns it as a context manager. The file can then be populated with data. This is especially useful when you're building a dataset incrementally, e.g. by pooling subsets of other datasets:

python with loompy.new("outfile.loom") as dsout: for sample in samples: with loompy.connect(sample) as dsin: logging.info(f"Appending {sample}.") dsout.add_columns(ds.layers, col_attrs=dsin.col_attrs, row_attrs=dsin.row_attrs)

As a consequence, create_append() is now deprecated.

Fixes #42.

Experimental plotting features

ds.pandas() to return a Pandas DataFrame for the whole matrix, or selected parts. The interface is intended to simplify plotting, since many plotting libraries take Pandas as input. The interface is experimental, and e.g. lacks support for layers.

ds.embedding() and ds.embeddings() to find attributes that are >1-dimensional. Again intended to support plotting, by making it easy to find the X/Y coordinates without knowing if they are stored as TSNE, PCA or something else. The interface is liable to change (in particular, I'd like to find a shorter name than "embedding").

Two useful colormaps: loompy.zviridis is a zero-inflated version of viridis, good for plotting zero-inflated data. loompy.cat_colors() is a function that generates N distinct colors, in pleasing and distinguishable hues, for large N.

Contributes to #62.

- Python
Published by slinnarsson almost 8 years ago

loompy - Compatibility with Seurat & create empty layers

Compatibility with Seurat & loomR

In some cases, Seurat would create loom files with attributes being variable-length ascii. This technically violates the loom specification, but what's worse is that loompy would read them as byte arrays. We now handle such strings gracefully and they are returned as arrays of string objects (supporting unicode).

Create empty layers

Previously, there was no way to create an empty layer without supplying a dense matrix. This would cause problems when you wanted to add a larger-than-RAM layer to an existing file. We now support an elegant syntax for creating an empty layer, directly on disk, by assigning a data type to the layer name. For example:

python with loompy.connect("filename.loom") as ds: ds.layers["intronic"] = "int16"

Or, using the shorthand syntax directly on the connection object:

python with loompy.connect("filename.loom") as ds: ds["intronic"] = "int16"

Once the layer has been created, you can assign values to (parts of) the layer, building it up incrementally.

- Python
Published by slinnarsson about 8 years ago

loompy - Bugfixes

Note: If you're using sparse matrices to create loom files, this update fixes a nasty performance bug (see #48)

Bugfixes

• Exception when assigning a layer directly on the connection object:

python with loompy.connect("filename.loom") as ds: ds["layername"] = m

• Severe performance bug when creating from large sparse matrix (issue #48)
• Error in example of valid file format (issue #44)
• Improved handling of connections opened in readonly mode (see commit d92bac9f04880bffcf00c1655d98d8fdc84a67d1)

- Python
Published by slinnarsson about 8 years ago

loompy - Bugfixes and minor new features

Bugfixes

• Bug in sparse() caused it to load only the first 1000 cells

New features

• scan() method now accepts boolean mask arrays to select items (rows/cols)
• Opening a file that uses old-style row_edges and col_edges automatically adds row_graphs and col_graphs

- Python
Published by slinnarsson over 8 years ago

loompy - Bugfixes and __version__ attribute

Bugfixes and minor new features:

• loompy.__version___ attribute (gives version of loompy package)
• Better handling of global attributes when combining files
• Make it possible to delete global attributes

Experimental new file feature: file spec version stamp

LOOM_SPEC_VERSION HDF5 attribute on the root group. Use it like this:

python with loompy.connect(filename) as lc: if "LOOM_SPEC_VERSION" in lc.attrs: print("file version: " + lc.attrs.LOOM_SPEC_VERSION) else: print("file version: less than 2.0.1")

Note: this is experimental, and is a proposed new feature of the loom file format specification. As such, the attribute is not guaranteed to exist, and you must always check for it before trying to read it. If it doesn't exist, then the loom file spec version is assumed to be less than "2.0.1" by default.

Note: the LoomSpecVersion is not the same thing as the loompy package version.

- Python
Published by slinnarsson over 8 years ago

loompy - Bugfix release

Finally fixed sparse matrix support, so that this code is verified to work:

python import numpy as np import loompy import scipy.sparse as sparse filename = "test.loom" matrix = sparse.coo_matrix((100, 100)) row_attrs = { "SomeRowAttr": np.arange(100) } col_attrs = { "SomeColAttr": np.arange(100) } loompy.create(filename, matrix, row_attrs, col_attrs)

(CSR, CSC and COO matrices are all supported and tested).

Also updated the docs with the example above.

- Python
Published by slinnarsson over 8 years ago

loompy - Bugfix release

What's new

• Fixed a bug where create() would not recognise sparse matrices

- Python
Published by slinnarsson over 8 years ago

loompy - Fix a deployment issue

Getting PyPi to accept the deployment via Travis CI

The version number is the only change.

- Python
Published by slinnarsson over 8 years ago

loompy - Bug fixes and timestamps

Timestamps

Implements timestamps to keep track of file modifications as described in #26

Bug fixes

Fixes bugs introduced by the timestamp code, which resulted in KeyError mentioned in #30

- Python
Published by slinnarsson over 8 years ago

loompy - v2.0

Announcement

I'm happy to announce Loom 2.0, an almost complete rewrite of loompy. v2.0 implements many recently requested features such as Unicode support, multidimensional attributes, and multidimensional global attributes.

It also supports a powerful new concept of in-memory views (essentially a slice through the file, including all layers, attributes, and graphs) which works great with the new scan() method for out-of-memory algorithms.

It is much more Pythonic, with a uniform API for reading/writing/deleting attributes, graphs and layers.

It is more generous (allowing almost any kind of list, tuple or array to be assigned to attributes, or any kind of sparse or dense adjacency matrix for graphs). At the same time, it normalizes everything to conform to the file spec.

Finally, I have updated the file format specification to be more specific about the allowed datatypes and shapes.

All this, and it remains fully backwards compatible with your old loom files. I have used 2.0 for a couple of months already and it's a significant step forward. Your code will be simpler, more expressive and more capable.

Note: your current code should mostly work without problems. You'll get log messages for deprecated methods and attributes, but they can be safely ignored. There are two breaking changes, both explained below. After fixing instances of these two issues, our current analysis pipeline runs without error and uses nearly every feature of loompy 2.0, so I would consider this a reasonably stable release. Out-of-memory manifold learning and clustering of a 500k cells by 27k genes dataset with all the bells and whistles runs just fine on my MacBook with 16 GB RAM (admittedly, it takes a few hours)!

I've updated the docs, which are now (for technical reasons) hosted at http://linnarssonlab.org/loompy.

What's new

Note: in the short examples below, ds is a LoomConnection object.

1. More expressive and pythonic attribute management

• Attributes can be accessed via ds.ca(column attributes) and ds.ra properties.
• Both named access and dictionary-style access: ds.ca.CellID and ds.ca["CellID"]
• Set attribute values by assignment: ds.ra.Chromosome = <values>
• Delete attributes directly: del ds.ra.Gene
• Old-style attributes col_attrs and row_attrs remain, but are considered deprecated
• Breaking change: Old-style direct named attributes such as ds.Gene are no longer supported
• Attribute values are lazy-loaded. Nothing is loaded when you connect to a file, so connecting to files of any size is instantaneous and consumes almost no resources.

1. More expressive and pythonic graph management

• Graphs can be accessed via ds.col_graphs and ds.row_graphs
• Both by name and as dictionaries, e.g. ds.col_graphs["KNN"] or ds.col_graphs.KNN
• Graphs are stored and accessed as scipy.sparse graphs
• Store graphs directly from sparse matrices: ds.col_graphs.MySparseGraph = sparse.coo_matrix(...)
• Delete graphs: del ds.col_graphs.KNN
• Graphs are lazy loaded
• Old-style graph methods remain, but are considered deprecated

1. New in-memory view object

• Call ds.view[..., ...] to slice along rows and columns
• Views automatically slice through all layers, attributes and graphs
• New scan() method returns view objects representing partial views of the underlying file

1. Improved support for sparse matrices

• Fixed bugs in loompy.create() when supplying a sparse matrix
• Attribute values can now be sparse
• Store graphs from sparse matrices: ds.col_graphs.MySparseGraph = sparse.coo_matrix(...)

1. More flexible assignment of attributes

• Automatic conversion from tuples, lists, np.ndarray, np.matrix or scipy.sparse
• Automatic conversion of elements that are (unicode) strings, byte arrays, string objects and all kinds of numbers
• More accurate validation of inputs
• Whatever the values are, normalize them to HDF5 numbers or fixed-length ascii, per spec

1. New attribute data types

• Support for numeric types int8, int16, int32, int64, uint8, uint16, uint32, uint64, float16, float32 and float64.
• Support for multi-dimensional attributes (i.e. each value is an N-dimensional array)

1. New layer data types

• Support for numeric types int8, int16, int32, int64, uint8, uint16, uint32, uint64, float16, float32 and float64.

1. Better handling of file lifecycle

• Support for context manager (with loompy.connect(...) as ds:)
• Better handling of errors on create() and close()
• Breaking change: create() and create_from_cellranger() do not return a value (they used to return an open LoomConnection, but this usually would leak a file handle).

1. Unicode support

• Roundtrip support for Unicode strings on the Python side
• ASCII encoding and decoding using XML entitites, to ensure maximum compatibility

1. Improved scanning functionality

• New scan method replaces batch_scan and batch_scan_layers
• Scan columns while ordering rows based on key attribute (and vice versa)
• Iterator returns a view object, which is a slice through all layers, attributes and graphs
• Together with the new create_append function, makes it trivial to scan and combine files

1. Sample datasets

• We provide published datasets for browsing and download at http://loom.linnarssonlab.org (Note: broken in Safari; please use Chrome instead for now).

1. Changes to the loom file format specification

• More precise definition of allowed data types
• Multidimensional attributes
• Unicode support

- Python
Published by slinnarsson over 8 years ago