Data

Tools

Indexes

Applications

Experiments

Open Source Science

Recent Releases of https://github.com/bacpop/pp-sketchlib

https://github.com/bacpop/pp-sketchlib - Sketchlib v2.1.3

  • Replace robinhood hashmap (no longer maintained) with unordereddense
  • Add local mode for testing/customising compilation
  • Fix efficiency of query sparse modes

What's Changed

  • Improve efficiency of sparse queries by @johnlees in https://github.com/bacpop/pp-sketchlib/pull/94

Full Changelog: https://github.com/bacpop/pp-sketchlib/compare/v2.1.2...v2.1.3

- C++
Published by johnlees over 2 years ago

https://github.com/bacpop/pp-sketchlib - Sketchlib v2.1.2

  • Fix a previously undetected segfault when multithreaded
  • Hopefully(!) fix for new clang compilers

What's Changed

  • Fix for adding random sequence correction for strand-specific k-mers by @nickjcroucher in https://github.com/bacpop/pp-sketchlib/pull/90
  • Attempt to fix for clang 15 by @johnlees in https://github.com/bacpop/pp-sketchlib/pull/93

Full Changelog: https://github.com/bacpop/pp-sketchlib/compare/v2.1.1...v2.1.2

- C++
Published by johnlees over 2 years ago

https://github.com/bacpop/pp-sketchlib - Sketchlib v2.1.1

Small improvement to poor regressions, to fix a CI failure in PopPUNK

What's Changed

  • Fix buildkite by @johnlees in https://github.com/bacpop/pp-sketchlib/pull/87
  • More robust regression with zero distances by @johnlees in https://github.com/bacpop/pp-sketchlib/pull/88

Full Changelog: https://github.com/bacpop/pp-sketchlib/compare/v2.1.0...v2.1.1

- C++
Published by johnlees over 3 years ago

https://github.com/bacpop/pp-sketchlib - Sketchlib v2.1.0

  • Remove armadillo from the code entirely in favour of Eigen
  • Actually link BLAS (used in the regression), LAPACK not actually needed
  • Use a new regression solver, see 'Normal equations' under https://eigen.tuxfamily.org/dox/group__LeastSquares.html
  • Solve issues with bool in highfive code, now uint8_t explicitly used
  • Use KMeansRex, as the kmeans in armadillo is now removed

nvcc now sees the eigen code and throws up a few warnings, but this doesn't seem to be causing as many problems as it used to

What's Changed

  • BLAS linking; CI fix; better message for deprecated function by @johnlees in https://github.com/bacpop/pp-sketchlib/pull/84

Full Changelog: https://github.com/bacpop/pp-sketchlib/compare/v2.0.1...v2.1.0

- C++
Published by johnlees over 3 years ago

https://github.com/bacpop/pp-sketchlib - Sketchlib v2.0.1

Minor update which should not change/break the API, adds a few small new features. #82 simplifies and parallelises sparsification by a threshold (and removes kNN).

What's Changed

  • Change web sketch output from uint64 to hex by @muppi1993 in https://github.com/bacpop/pp-sketchlib/pull/74
  • Update README.md by @danrlu in https://github.com/bacpop/pp-sketchlib/pull/75
  • Add reads attribute to databases by @johnlees in https://github.com/bacpop/pp-sketchlib/pull/77
  • Lineage model fitting - sketchlib changes by @nickjcroucher in https://github.com/bacpop/pp-sketchlib/pull/82

New Contributors

  • @muppi1993 made their first contribution in https://github.com/bacpop/pp-sketchlib/pull/74
  • @danrlu made their first contribution in https://github.com/bacpop/pp-sketchlib/pull/75

Full Changelog: https://github.com/bacpop/pp-sketchlib/compare/v2.0.0...v2.0.1

- C++
Published by johnlees almost 4 years ago

https://github.com/bacpop/pp-sketchlib - Sketchlib v2.0.0

This is a new major version due to overhaul of the CLI.

New features: - CLI is now written with docopt, and should be more straightforward to use (#72). - New self sparse query, which works with CUDA too (#67). Used in mandrake.

Bug fixes: - Sparsification functions fixed and added to tests.

Other: - Enable pip install and pushing to pypi (#70) - Adding docker build and test on buildkite (#70)

- C++
Published by johnlees about 4 years ago

https://github.com/bacpop/pp-sketchlib - Sketchlib 1.7.4

New features: - Sketching kernel overhaul: should be faster, and supports read datasets larger than the device memory size (#63 #64) - Increase the size of the countmin table on GPUs to 2^30 columns, takes up about 30% of a 3090's memory (#64) - Better progress meter for read sketching (#64) - Save whether the DB was calculated using the reverse complement strand (#65)

Bug fixes: - Fix indices for reference vs query distance calculations, hit when reaching ~100k queries (#64) - If sketching craps out early, save the successful sketches up until that point to the HDF5 DB without corrupting it, so it can be joined later. (#64)

Other: - Refactor the CUDA code a bit, putting classes into their own files. (#64)

- C++
Published by johnlees about 5 years ago

https://github.com/bacpop/pp-sketchlib - Sketchlib 1.7.3

New features: - Better parallel efficiency for distance calculation on a CPU (#62)

Bug fixes: - Progress meter caused crash with small numbers of distances (#62) - Smoother progress meter for distances (#62) - Remove device reset which could cause problems with both GPU dists and a GPU graph in poppunk (#62)

- C++
Published by johnlees about 5 years ago

https://github.com/bacpop/pp-sketchlib - Sketchlib 1.7.2

New features: - Progress bars for main CPU functions #61 - Handle Ctrl-C in both CPU and GPU functions #38

Bug fixes: - Deal with errors during sketching gracefully #61

Other: - Change CMakeLists to work with gcc on OS X (to allow use of openmp)

- C++
Published by johnlees about 5 years ago

https://github.com/bacpop/pp-sketchlib - Sketchlib v1.7.1

Bug fix: - Fix stall on GPUs when regression terminates early due to no matches #60

- C++
Published by johnlees about 5 years ago

https://github.com/bacpop/pp-sketchlib - Sketchlib 1.7.0

This updates __shared__ memory copies to use memcpy_async, a CUDA 11 feature (#59). This also fixes a bug sometimes seen with incorrect distances being calculated on a GPU.

- C++
Published by johnlees over 5 years ago

https://github.com/bacpop/pp-sketchlib - Sketchlib v1.6.5

Bug fixes: - When using random matches, now checks all references are in the object, and adds them if not (previously gave a key error) #56 - The python join interface warns to regenerate random matches if present. The above error would be triggered by a join where random was copied over. - Error in strides of the longToSquareMulti function with query-ref matrices #57

- C++
Published by johnlees over 5 years ago

https://github.com/bacpop/pp-sketchlib - Sketchlib v1.6.4

Bug fix:

  • Open HDF5 files as read only where possible (when querying). This prevents creating a lock, and allows multiple processes to access the same HDF5 file (#52)

- C++
Published by johnlees over 5 years ago

https://github.com/bacpop/pp-sketchlib - Sketchlib v1.6.3

New features:

  • Version and sketch version are saved as module attributes, makes testing module version more reliable #50
  • Boundary code has been removed, now in PopPUNK package extension #50, https://github.com/johnlees/PopPUNK/pull/146

- C++
Published by johnlees over 5 years ago

https://github.com/bacpop/pp-sketchlib - Sketchlib v1.6.2

New features: - Adds functions to grow network in 1D or 2D, by iterating boundary forward #48

Bug fixes: - Compile without openMP

- C++
Published by johnlees over 5 years ago

https://github.com/bacpop/pp-sketchlib - Sketchlib v1.6.1

New features: - More optimised compile options for emcc, including removing exception handling 60a803ff30245e19e8fe0c15b73f9905a8f6cccc

Bug fixes: - Segfault when GPU distances were calculated using multiple blocks #44

Cosmetic code changes: - Combine matrix index functions for CPU and GPU code #44 - Reflow all of the code, fix inconsistent tab/space indentation #45

- C++
Published by johnlees over 5 years ago

https://github.com/bacpop/pp-sketchlib - Sketchlib v1.6.0

New features: - Add WebAssembly version (for single queries) #40. - Allow random on/off in jaccard dist bindings #41.

Bug fixes: - Fixed a stall with GPU distances sometimes triggered due to a misplaced __syncwarp() call. - Check for sketches which exceed GPU shared memory size, and use global memory rather than failing if it won't fit #43. - Check all CUDA API calls, including kernel launches. - Remove -march=native from the CMakeLists.txt, which may have been causing Illegal Instruction errors

- C++
Published by johnlees over 5 years ago

https://github.com/bacpop/pp-sketchlib - Sketchlib v1.5.3

  • Fixes bug with ref v query distances (variable overwritten)
  • Adds a test for ref v query distances

- C++
Published by johnlees over 5 years ago

https://github.com/bacpop/pp-sketchlib - Sketchlib v1.5.2

  • Modifies the operation of the sparsification of distances to correctly deal with zero distances
  • Removes parallelisation of distance sparsification (causing issues on some platforms)
  • SIMD openmp commands removed from matrix operations

- C++
Published by johnlees over 5 years ago

https://github.com/bacpop/pp-sketchlib - Sketchlib 1.5.1

Important bugfix (see https://github.com/johnlees/PopPUNK/issues/95): - Inputs to the CPU query API were being sorted by name, returning distances not in the expected order. This will have worked with poppunk_sketch but not through other software such as PopPUNK, unless the input was alphabetical.

- C++
Published by johnlees almost 6 years ago

https://github.com/bacpop/pp-sketchlib - Sketchlib v1.5.0

New features: - Allow turning off random match computation (#28), which is useful for querying. - Replace threads with OpenMP (#31). Much more efficient CPU parallelisation of distances. - Add GPU/CPU hybrid sketching algorithm for read data (#32). - Add codon phased seeds --codon-phased (#35). This replaces randomly spaced seeds.

Bug fixes: - Update to work with HighFive v2.2.2 - Check for zero Jaccard distances (#36)

- C++
Published by johnlees almost 6 years ago

https://github.com/bacpop/pp-sketchlib - Random match correction

New features: - Add parallelised sparse and dense matrix operations (#21, #23) - Used spaced seeds in hash function; store sketching version in HDF5 file (#22) - Chunk up distance calculation on the GPU, so any size is supported (#24) - Calculate random match chances via Monte Carlo simulation (#27)

Added CI testing: - Add new tests for matrix functions - Add test for distances compared to reference values

Test to be manually run: - Compare GPU and CPU distances

- C++
Published by johnlees about 6 years ago

https://github.com/bacpop/pp-sketchlib - K-mer length enhancements

Adds the following features: - Extract Jaccard distances - Calculate and save genome length using minimum hash - Calculate and save base composition - Adjust comparisons for base composition, and expected random matches - Spaced seeds for short k-mer lengths

Bug fixes: - Error when calculating distances on a GPU (#18)

- C++
Published by johnlees over 6 years ago

https://github.com/bacpop/pp-sketchlib - GPU accelerated distances

New features: - Distance calculations on CUDA compatible GPUs. - Use armadillo linear solver rather than dlib for regressions (on CPU).

Added some benchmarks to the readme

- C++
Published by johnlees over 6 years ago

https://github.com/bacpop/pp-sketchlib - Improved speed and memory

New features: * Implement countmin correctly and pick sensible default parameters for sketching read datasets (#11) * More reliable and faster code for optimiser used in linear regression * Add testing and CI

Bug fixes: * Made CMakeLists.txt and Makefile more general (removed specific paths)

- C++
Published by johnlees over 6 years ago

https://github.com/bacpop/pp-sketchlib - Patched release

Small fixes:

  • #14: Use the mkl_rt library to link, which works better with conda (see https://groups.google.com/forum/#!msg/kaldi-help/m3nyQke0HS0/4fj8gkSWAgAJ, https://software.intel.com/en-us/forums/intel-math-kernel-library/topic/748309, https://github.com/ContinuumIO/anaconda-issues/issues/720)
  • #14: As the sequential library cannot be specified, export this as an environment variable in the api.cpp functions which handle their own threading.
  • Correct version of gcc used to compile

- C++
Published by johnlees over 6 years ago

https://github.com/bacpop/pp-sketchlib - Initial release

Features tested and working on OS X and Linux. Install looks ok, locally

- C++
Published by johnlees over 6 years ago