Recent Releases of flowkit
flowkit - 1.2.0
FlowKit 1.2.0 is out! Major updates include support for Boolean gates in FlowJo workspaces, renaming Sample channels and making the API more user-friendly. Thanks to everyone who contributed to this release, keep the bug reports and suggestions coming!
New features:
* Workspace class now supports Boolean gates when parsing a FlowJo workspace (see #179).
* Added get_keywords method to Workspace for getting sample keywords stored in a FlowJo workspace (see #210).
* Workspace method get_gate_events now supports source kwarg to control pre-processing of returned events (see #196).
* Session & GatingStrategy classes now support renaming gates via the rename_gate method (see @219).
* Session & GatingStrategy classes now support removing custom sample gates via the sample_id kwarg in the remove_gate method (see #221).
* Sample class now supports renaming channels via a new rename_channel method (see #198).
* Sample class methods get_events, get_channel_events and as_dataframe support event_mask kwarg for filtering events using a Boolean array (see #218).
* Sample class method as_dataframe supports new col_multi_index option to control column index type.
* Matrix class now supports comparison operator for determining if 2 Matrix instances are the same (see #200).
* New utils module function generate_transforms to generate a set of transforms for a Sample (see #162).
API changes:
* Transform class: removed id attribute. This allows more convenient use of independently using transforms outside of a GatingStrategy (see #191).
* GatingStrategy class: added transformid arg to `addtransformmethod.
* Session class: added transform_id arg toaddtransformmethod.
* Workspace class:gettransformsnow returns a dictionary LUT of transform IDs and transform instances.
* Matrix class: removedidattribute. This allows more convenient use of independently using matrices outside of a GatingStrategy (see #199).
* Session class:getcompmatrices` now returns a dictionary LUT of matrix IDs and Matrix instances.
* GatingResults class: the 'level' column of the report is now indexed at 0 (for root level).
Bug fixes: * Ensure exporting of QuadrantGate raises NotImplementedError (see #209). * Fixed plot_scatter error with 0 or 1 points (and 2 points with extent) (see #197). * Better error for invalid gate names '.' & '..' (see #220).
Documentation: * Added more informative docstrings for transforms * Testing code reorganized to factor constants to new test config file (Thanks @ksebby for PR #213).
Dependencies: * Drops Python 3.8 support (EOL as of Oct 2024). * Updated dependency versions for compatibility w/ Python 3.9 - 3.12 (see requirements.txt for details).
- Python
Published by whitews over 1 year ago
flowkit - 1.1.0
Hey FlowKit community! Lots of changes in this release, including support for Python 3.11 & 3.12. The major updates: we've moved to a src based structure using pyproject.toml and finally converted all plotting functions to Bokeh (dropping matplotlib and Seaborn as requirements). Read the notes below for all the changes.
Special thanks to @hbhargava7 and @laserson for their contributions to this release!
New features:
- New utility function
read_multi_dataset_fcsfor reading FCS files with multiple data sets (returns a list of Sample instances). Workspaceclass now has an option to load FCS files from the URI parameter embedded in the WSP file (Many thanks to @hbhargava7 for PR #168).Sample.plot_scatter_matrixnow displays histograms along the diagonal (Thanks @laserson for the suggestion in #169).- Exporting FlowJo 10 workspaces now supports the
WSPBiexTransform. Sample.plot_scatteraddsevent_maskoption to filter the plotted events.Sample.plot_contours&Sample.plot_channelmethods now use Bokeh.- Custom heatmap color palette was improved (it's subtle but has better visual contrast between density levels).
API changes:
Sample.plot_histogrammethod addsdata_min,data_max, andx_rangekwargs for filtering data ranges and controlling the view range.- Removed argument
ignore_transformsinparse_wsputility function. - Transforms from a GatingML document are no longer imported as undocumented "GML" sub-classes, instead using the documented
Transformsub-classes. - Added kwarg
gate_pathtoplot_scattermethod in both Session & Workspace. sample_idargument now required forGatingStrategy.is_custom_gatemethod.- Rename
plot_scatterargumenthighlight_indicestohighlight_mask(since it takes a Boolean array and not a list of indices). plot_scatterargumentdim_idssplit tox_label&y_labelfor specifying axis labels.plot_gateargumentdim_idssplit tox_label&y_labelfor specifying axis labels.
Bug fixes:
- Fixed KeyError in
Workspace.summarymethod when a sample has no gates (see #155). - Fixed WSP export of "range"
RectangleGateinstances (see #157). - Fix
ValueErrorthrown when compensating a Sample specifying null channels vianull_channel_list(see #175). - Fix
Session.plot_scattermethod where mixed compensations are specified for x & y channels (rare).
Documentation:
- Switched to ReadTheDocs theme for documentation.
- Clarify docstrings for null channel usage (see Sample & Matrix constructor docs).
- All notebooks validated and updated for the new release.
Dependencies:
- Dropped support for Python 3.7, added support for Python 3.11 & 3.12.
- FlowIO updated to >=1.3.0,<1.4
- FlowUtils updated to >=1.1.0,<1.2
- Matplotlib & Seaborn removed as dependencies. All plotting now uses the Bokeh library (>=3.1).
- Pandas updated to >=2.0
- Added new dependency contourpy (the library matplotlib uses).
- Other dependency minimum versions updated for Python 3.8 - 3.12 support (See requirements.txt for details).
- Python
Published by whitews almost 2 years ago
flowkit - 1.0.1
BUGFIX: GatingStrategy instances using gate dimensions referencing built-in FCS compensation would return uncompensated events. This scenario would occur for GatingML 2.0 documents where the reserved 'FCS' value was used to reference the built-in FCS compensation matrix defined in the metadata keywords $SPILLOVER or $SPILL. Special thanks to @strategist333 for finding this issue and submitting PR #153.
- Python
Published by whitews almost 3 years ago
flowkit - 1.0.0
Version 1.0.0 is a major milestone for the FlowKit project. This release adds support for custom sample gates in FlowJo 10 workspaces. As a major release, this version has significant API changes. Please read the release notes below and review the documentation for all the API changes.
New features:
* New dedicated Workspace class for importing FlowJo 10 workspaces. This class is essentially a read-only class for easily importing & extracting gating information from .wsp files.
* Importing .wsp files now supports 'eventsInside' flag for FlowJo gates (see #135)
* Session & GatingStrategy classes now support custom gates by specifying a sample_id when adding a gate to the gate tree.
* Session & GatingStrategy classes now support removing gates from the gate tree via a new remove_gate method. The method includes an option to retain downstream (descendant) gates.
* Sample class adds method get_index_sorted_locations for FACs Diva index sorted files (see #112)
* Sample class adds id attribute, allowing identification of samples created from NumPy arrays or pandas DataFrames.
* Sample class supports sorting (samples will be sorted by id attribute)
* Sample class constructor supports new arguments for controlling behavior in cases where there is a discrepancy in the data offset location (see #136)
* Sample method plot_scatter adds highlight_indices option for highlighting events using a Boolean array.
* Transform classes now support comparison for determining if 2 transforms are equivalent (i.e. have the same parameters).
Removed features:
- Removed
calculate_compensation_from_beads, the function was moved to the flowkit_extras package (see #123) - Session class no longer imports FlowJo 10 workspaces, this functionality now lives in the dedicated Workspace class
- Session class no longer uses sample groups, simplifying the API. Sample groups were needed for FlowJo workspaces.
API changes:
- New
exceptionsmodule containing custom FlowKit exceptions and warnings. This module also imports FlowIO exceptions so users can import all of them from one source. - Session method
get_group_sampleswas removed.group_nameoption removed from all Session methods. - Session and GatingStrategy method
get_child_gateswas removed. A new methodget_child_gate_idswas added. Gate instances are only available viaget_gate. - Session and GatingStrategy method
get_parent_gatewas removed. A new methodget_parent_gate_idwas added. Gate instances are only available viaget_gate. - GatingStrategy now has a
is_custom_gatemethod for determining whether a Gate ID, Sample ID combination has a custom gate or the template gate. - Session & Sample
plot_scattermethods now take an optionalbin_widthargument to control the smoothness of the color density - Gate classes no longer take the
parent_gate_nameargument. Gate relationships are handled completely by the GatingStrategy class. - Gate classes have a new
get_dimension_idsfor retrieving Dimension IDs. - Vertex class was removed. PolygonGate vertices are now just tuples of floats.
Bug fixes:
* Fix sample export of orig data for FCS files with data type int
* Fixed case where adding gates directly under QuadrantGate was possible. Adding a child gate to a QuadrantGate directly is not allowed, they must be added a one of the Quadrant instances.
* Fix parsing of FlowJo workspaces where a sample has no gate data.
* Fixed color density in scatter plots to correct missing 1st bin & poor coloring for data sets with outliers (see #139)
Documentation:
* Docstring for Sample method subsample_events corrected to reflect that the method does not return anything (see #129)
* Top-level functions & exceptions were added to the documentation
* All tutorial & advanced notebooks were updated for compatibility with the new API.
Dependencies: * FlowIO requirement updated to 1.1.1 * Removed statsmodels dependency (see #123) * Other dependency minimum versions updated for Python 3.7 - 3.10 support (See requirements.txt for details)
- Python
Published by whitews almost 3 years ago
flowkit - 0.9.1
Dependencies:
- FlowIO requirement updated to v1.0.1 (fixes issue exporting some FCS files)
API changes:
- new GatingStrategy method findmatchinggate_paths to find gate paths given a gate name
- new Session method findmatchinggatepaths & getchildgateids
Bug fixes:
- Corrected error types (several instances of ValueError changed to TypeError)
- BUGFIX: Fixed issue where adding new gates to a Session where a WSP was imported would cause an error (issue #110)
Documentation:
- Several docstrings updated and fixed for ReadTheDocs
- Example notebooks moved to docs folder & added to ReadTheDocs
- Python
Published by whitews over 3 years ago
flowkit - 0.9.0
New features:
* Support for Python 3.10 (drops support for Python 3.6)
* All C extensions moved to FlowUtils (and FlowUtils now provides pre-built wheels - should avoid the need for a compiler for most users)
* New function extract_wsp_sample_data to parse FlowJo 10 wsp & extract Sample metadata (keywords) to a Python dictionary
* New GatingStrategy method get_max_depth to get max depth of gate tree
* New Session method summary for a quick summary of what the session contains
* Sample class now supports exporting new FCS files with metadata (#89)
API changes:
* Sample attribute channel_lin_log removed, lin / log metadata now available as "pne" column in Sample.channels attribute
* Sample class: export method has new kwarg include_metadata for including all key/value pairs in self.metadata in the output FCS file
Bug fixes: * Fixed typo in FCSSTANDARDKEYWORDS for 'date' keyword * Fixed parsing of FlowJo workspaces where sample groups have "gate-less" samples (#91) * Replace special quote characters that caused a UnicodeDecodeError on some Windows installs (#97)
Dependencies:
* FlowIO requirement updated to 1.0.0
* FlowUtils requirement updated to 1.0.0 (with pre-built binaries for most platforms)
* NumPy minimum requirement of 1.20 when using pre-built FlowUtils binaries (required for NumPy C API compatibility)
* Fix deprecation warnings from Pandas 1.4 (squeeze kwarg removed from read_csv)
Acknowledgements: * Special thanks to @sbenz for help testing this release on arm64 Macs (Apple Silicon)
- Python
Published by whitews almost 4 years ago
flowkit - 0.8.2
- Sample Class
- Error is raised when retrieving compensated or transformed events prior to applying a compensation matrix or a transform (used to issue just a warning)
- plotscattermatrix argument order changed to be more consistent with other plot methods
- Session Class
- Analyzing multiple samples using multiprocessing now uses a more conservative estimate of memory to avoid running out of system memory
- BUGFIX: Fixed failure to properly parse FlowJo workspaces containing an ellipse gate.
- Documentation was updated to include a table of contents with links to methods for the complex classes: Sample, GatingStrategy, GatingResults, & Session
- Python
Published by whitews over 4 years ago
flowkit - 0.8.1
- GatingResults get methods now accept a
gate_pathargument to disambiguate re-used gates - BUGFIX: fixed issue parsing FlowJo 10 wsp files with reused gates that contain a child gate (issue #84)
- Suppressed output for analyzing Session samples when verbose=False
- Suppress FlowKit generated warnings when running tests
- Python
Published by whitews over 4 years ago
flowkit - 0.8.0
New features:
* Support for Python 3.9
* FlowKit version now available via top-level attribute __version__
* GatingStrategy:
* optimized processing order of gates to avoid re-processing
* optional caching of pre-processed events to speed up processing of gates
* Matrix class now has an inverse method
* WSPBiexTransform now has an inverse method
* Significant speed increase for parsing FlowJo 10 WSP files with lots of bi-ex transforms
* Sample method plot_contour now has optional fill argument to created filled contours
* Sample now re-applies a transform when Sample is given a new compensation
* Analyzing samples in a Session using multiprocessing will now try to dynamically allocate CPUs based on available memory to avoid out of memory errors
Removed features: * Anomalous event detection removed from FlowKit, now lives in separate FlowQC project
API changes:
* gates module:
* All gate types now take a pandas DataFrame as input (possible to use gates independently now)
* Sample class:
* subsample argument added back to Sample constructor (better to have subsample ready for plot methods)
* Sample plot methods (except histogram and plot channel) now default to subsample=True
* The various methods for getting NumPy arrays of events are now encapsulated into the single get_events methods that takes a source argument (consistent w/ other Sample methods)
* Method get_channel_data renamed to get_channel_events
* Added option to cache original events, they are no longer stored by default (saves memory)
* Attribute anomalous_indices renamed to flagged_indices, and now set by user for any purpose using set_flagged_events method
* Attribute channels changed from a dictionary to a more readable pandas DataFrame
* New plotting method plot_channel for plotting channel data with "events" as x-axis (similar to plotting vs Time)
* GatingStrategy class:
* gate_path is now always a tuple (not a list)
* Uses of gate_id replaced with gate_name: a gate ID is a tuple of (gatename, gatepath)
* Method get_gate_indices renamed to get_gate_membership (since it returns a boolean array)
* Session class:
* Plotting methods no longer analyze samples, analyze_samples method must be called prior
* Method get_gate_indices renamed to get_gate_membership (since it returns a boolean array)
* Dimension class
* Attribute label renamed to id
* Method get_gate now returns the containing QuadrantGate when a Quadrant ID is given
* Function calculate_extent removed from public API (was never intended for external use)
Bug fixes: * Improved support for Windows * multiprocessing now working * fixed issue where parent gate results were not being properly retrieved, causing lots of re-processing * fixed memory leak in pointsinpolygon C extension (used for processing PolygonGate instances) * fixed issue in C extension code that would sometimes cause mis-allocation of NumPy array sizes
Documentation: * New & comprehensive tutorial notebook series for new FlowKit users * New notebook demonstrating how to create a custom Sample gate in a Session * More docstrings for classes and methods
Dependencies: * FlowIO requirement updated to 0.9.12 (significantly faster reading of FCS files) * FlowUtils requirement updated to 0.9.5 * new requirement networkx >= 2.3 * new requirement psutil ~= 5.8.0
- Python
Published by whitews over 4 years ago
flowkit - 0.7.0
New features:
* It's now easier to assign multiple samples to a Session group
* Session.assignsamples can take either a Sample ID or list of Sample IDs
* Session.addsamples now takes an optional group argument for assigning to a group when loading Samples
* New Session method get_wsp_gated_events for conveniently getting a list of DFs of compensated & transformed gated events
* More informative error message when reloading the same samples (include file name in error message)
* Transforms now support handling of 1-D data arrays
API changes:
* Session: renamed method assign_sample to assign_samples now that it supports a list of sample IDs
* Session methods get_sample_ids() & get_group_sample_ids() now have a loaded_only option to only return sample IDs of loaded samples
* Session: method get_gate now has a sample_id option allowing retrieval of custom sample gates in a sample group's gating strategy
Bug fixes: * Eliminate Pandas PerformanceWarning when creating GatingResults instances (occurred when getting gate indices) * Fixed issue where Session.exportwsp fails to export rectangle gates that include transforms * Fixed issue where Session.getgroupsampleids() incorrectly reports samples as members when that are not part of that group * Test data and code are no longer included in packaged
Documentation: * Showcase exporting to wsp in replicate flowjo example notebook * More Session methods documented with docstrings * Cleaned up examples folder so it is easier for users to browse * New example notebooks for: * Demonstrating a comparison of clustering methods (Leiden vs Louvain) * Demonstrating dimension reduction on processed event data
Dependencies: * FlowUtils requirement updated to 0.9.4 * NumPy requirement updated to >=1.19 * Seaborn requirement updated to >=0.11 * Pandas requirement updated to >=1.1
- Python
Published by whitews over 4 years ago
flowkit - Round and Round
New features:
- Experimental support for exporting FlowJo 10 workspace files (limited to single sample groups)
- Added support for FlowJo 10 biex transform
- Added inverse transform methods for most transformations
- implement
ignore_transformsoption for parsing FJ10 workspaces - Sample.apply_transform supports customized transforms per channel via dictionary
- Create Sample instances from Pandas DataFrame w/MultiIndex columns
- Sample: store original events unprocessed
API changes:
- Sample: added attributes
scatter_indices&time_index - Sample:
apply_transformmethod now has option to include scatter channels - GatingStrategy:
get_gate_idsnow returns a list of tuples containing the gate ID & list of ancestors (gate path) - GatingStrategy: new method
get_root_gatesfor ... - GatingStrategy: simplify GatingStrategy methods for retrieving gates,
- GatingStrategy: added new convenience method getchildgates
- Session: new method
get_group_comp_matrices - Session: all method arguments referencing a sample group name are now consistent ('group_name')
- Session: new method to get all gate IDs & paths for a sample
- Session: new method
get_group_transformsfor retrieving group transformations - Session: new method
export_wspfor exporting a sample group's gating strategy as a FlowJo 10 workspace - Session: new methods for retrieving Sample specific comp matrices & transforms for a group
- Session: new method for retrieving gated events for a Sample
- Compensation utility functions moved to FlowUtils
- make sample utils functions public
- expose calculate_extent function for flowkit-extras
Bug fixes:
- Clearer error message when requesting sub-sampled events from
get_channel_datawhen Sample has not been sub-sampled - Fixed issue where gain was incorrectly applied to time channel data
- Allow re-use of gates on different branches at any level
- Applying re-used gates requires the Gate instance to know the full path
- Fixed plot_scatter for cases where the event arrays are empty
- Session.getgroupsamples() now only returns loaded samples
Documentation:
- Added more docstrings for Sample, Session, Transform methods
- Updated tutorial notebooks
- Added notebook for calculating mean fluorescence intensity (MFI) per channel for gated populations
Dependencies:
- Support for Python 3.8
- Removed MulticoreTSNE dependency
- Removed sklearn dependency
- Most transform implementations moved to FlowUtils
- Python
Published by whitews almost 5 years ago
flowkit - Branching Out
New features:
- GatingStrategy & Session classes now support re-using gate IDs in different gate hierarchy branches (i.e. having a child gate "IFNg+" under both a "CD4+" and "CD8+" gate)
- Added support for FlowJo 10 ellipsoid gates
- Plotting of gates in Session class now plots only events from the parent gate
- Added plotting of Quadrant gates
- Added bokeh hover tool to scatter plots
- Re-enabled multiprocessing by default for faster processing of gates in a Session
- Updated plot colormap for better contrast with gate boundaries
API changes:
- Major cleanup of public/private modules for cleaner API
- New Quadrant class for cleaner API when creating QuadrantGate instances
- New
wsp_transformsmodule containing FlowJo specific Transform sub-classes - GatingStrategy, Session - added
get_transformandget_comp_matrixmethods - GatingStrategy, Session - added
gate_pathkwarg for ambiguous gates inget_gate&get_gate_indices - Session: add
sample_idoption toanalyze_samplesto process a single Sample - Session: added
ignore_missing_filesoption toimport_flowjo_workspacemethod for suppressing warnings about missing FCS files when importing a FlowJo WSP file - Gate class - added
get_dimensionmethod - Sample: rename Sample method to
as_dataframe(consistent with Matrix class), & added options to re-order and rename columns in output DataFrame - Sample: 'fcs' is no longer the default ID when adding a compensation matrix (it is reserved for the embedded comp matrix from the
spillmetadata value) - Matrix:
fluorochromesargument is now optional - Matrix: new method
as_dataframeto get comp matrix as a Pandas DataFrame - Dimension: 'uncompensated' is now the default value when creating Dimension instances
Bug fixes:
- Properly support time channel data when
timestepkeyword is present in FCS metadata - Fixed display of plotted ellipses (corrected angle)
- Fixed parsing of compensated dimension references in FlowJo WSP files
- Fixed import of CSV compensation files with commented header
- Fixed creation of Sample instances from Pandas DataFrame
- Fixed parsing of FlowJo gate IDs for some WSP variants
Documentation:
- Documentation now on ReadTheDocs
- Many more docstrings for classes and methods
- Added contribution guidelines
- New tutorial notebook to demonstrate loading of FlowJo workspaces & programmatically replicating the gating strategy
- New example notebook showing compensation and Matrix class usage
- Python
Published by whitews over 5 years ago
flowkit - Community Service
New features:
- Plotting of gates in Session class: limited to polygon, rectangle, range, and ellipse gates.
- Session now supports importing FlowJo workspace files, though support is limited to WSP files with linear, log, logicle transforms, and only polygon & rectangle gates.
- Session class now supports multiple gating strategies via the implementation of "sample groups", similar to how FlowJo works.
- Sample class now allows retrieving events as Pandas DataFrames, with multi-index columns as the PnN and PnS labels
API changes:
- Session: add get methods for sample IDs, sample groups, and gate IDs
- More consistent handling of Matrix class as input to compensation methods
- Expose parsegatingxml function
- Expose plot_channel function
Bug fixes:
- Handle case where a Sample has no keyword for original filename
- Fix re-subsampling after applying Sample filters
- Ensure correct fluoro order when parsing a comp matrix from a CSV file
- Python
Published by whitews almost 6 years ago
flowkit - Build Gates - 0.3.2
API changes:
- New Session method for adding samples after a Session instance has been created
Bug Fixes:
- Sample export failed to create file
- Python
Published by whitews about 6 years ago
flowkit - Build Gates
New features:
- Programmatically add gates, transforms, and compensation matrices to a GatingStrategy
- Export a GatingML 2.0 compliant document from a GatingStrategy
API improvements:
- Gate sub-classes no longer require a GatingStrategy in their constructor, allowing re-use of instantiated gates
- New convenience methods in the Session class for getting parent gates and creating gates, transforms, and comp matrices
- Better x & y axis labels in scatter plot (includes marker names, if available)
Bug fixes:
- Fixed bug in GatingStrategy method getgateby_reference where an existing QuadrantGate could not be found
- Fixed bug in parsing spillover matrix from a $SPILL keyword value of an FCS file
- Python
Published by whitews over 6 years ago
flowkit - In Session
Multiple API changes, significant performance improvements, and bug fixes for Windows.
- New GatingResults class for reporting gating statistics
- Significant performance improvements in calculating gated events
- Support for multi-processing for evaluating gates
- New C-extension for calculating events in PolygonGates
- Cleaner API for user interaction
- New Session class for combining multiple Samples with a GatingStrategy
- tSNE support for multiple Samples in a Session
- Automatic generation of a compensation matrix from a set of bead files
- Simplified API for exporting Sample events
- Multiple bug fixes for Windows
- Python
Published by whitews over 6 years ago
flowkit - Gatin' 2: Electric Boogaloo
Now fully compliant with the GatingML 2.0 specification.
- Full support for GatingML Quadrant gate
- Now supports Ellipsoid gates of any dimension (not just simple 2-D ellipses)
- Sample class now takes any Transform sub-class, consistent with GatingStrategy transforms
- A new fancy logo!
- BUGFIX: fixed issue where the HyperlogTransform would fail on 2-D arrays
- Python
Published by whitews about 7 years ago
flowkit - Hello, World!
1st release of FlowKit for flow cytometry analysis and visualization, featuring GatingML 2.0 support and support for reading and exporting FCS data
- Python
Published by whitews about 7 years ago