Recent Releases of https://github.com/cadet/cadet-match
https://github.com/cadet/cadet-match - PyMoo 0.6 release
This release works with PyMoo 0.6 now.
- HTML
Published by Immudzen over 3 years ago
https://github.com/cadet/cadet-match - Arviz dependency
This release just adds a dependency on Arviz and cleans up a few minor things
- HTML
Published by Immudzen over 4 years ago
https://github.com/cadet/cadet-match - PyMOO update
This updates CADETMatch to use the current version of pymoo
- HTML
Published by Immudzen over 4 years ago
https://github.com/cadet/cadet-match - Dependencies updates
This release just changes the dependencies on pymoo and attrs so that the right versions should get installed
- HTML
Published by Immudzen over 4 years ago
https://github.com/cadet/cadet-match - KDE prior and MLE calculation
MLE is now calculated by looking at the highest probability on the chain Prior calculation is now moved into a separate process from MLE
- HTML
Published by Immudzen over 4 years ago
https://github.com/cadet/cadet-match - auto_keq fix and MCMC fixes
auto_keq now works with index
There are also many MCMC fixes. numpy.percentile has been removed and instead arviz.hdi is used. This handles asymmetric distributions much better. Generation of the KDE error model doesn't remove outliers now. Due to how the sampling is done the outliers are not actually outliers.
- HTML
Published by Immudzen over 4 years ago
https://github.com/cadet/cadet-match - Smoothing
Based on work in my thesis the smoothing code has been updated. The normalized root mean squared error is used to set a minimum value the smoothing can't go under. This is set to 1e-4 by default and was found to be a good value after testing. Effectively this means values smaller than 1e-4 * the peak max will get smoothed out of the system. This prevents a lot of noise that was sometimes left in then the L-point indicated that values as low as 1e-7 * peak max could be kept
This release also has various other small fixes in it
- HTML
Published by Immudzen almost 5 years ago
https://github.com/cadet/cadet-match - PyMOO
DEAP has been replaced with PyMOO and a lot of code has been removed as a result. PyMOO implements a more refined version of NSGA3 that works much better. This version should converge faster and closer to the optimum without needing a gradient step. So far in testing this has sped up overall performance.
Documentation has been updated.
- HTML
Published by Immudzen almost 5 years ago
https://github.com/cadet/cadet-match - minor fix in print_version
print_version was looking at the wrong version of the CADET-Python library
- HTML
Published by Immudzen about 5 years ago
https://github.com/cadet/cadet-match - bessel filtering and minor fixes
Smoothing has changed from using butter filters to bessel filters based on testing and a problem of ringing with sharp pulses and butter filters. The resampling process for very densely sampled datasets or datasets with inconsistent time steps is also cleaner.
- HTML
Published by Immudzen about 5 years ago
https://github.com/cadet/cadet-match - MCMC changes and maintenance
MCMC removes the burn-in step entirely and instead runs for 2 more integrated autocorrelation times and discards the first two autocorrelation times. This ensures the chain is independent of the starting state.
numba is now required and some functions in CADETMatch now use numba for acceleration
Removed spearman graphs Support addict 2.4.0
Lots of other small fixes to the code and cleanups
- HTML
Published by Immudzen about 5 years ago
https://github.com/cadet/cadet-match - Locking and examples
CADETMatch now uses file based locking and blocks on the locks for any file that could be read and written at the same time. As a result all the graphing modules now run completely in parallel and performance is improved with some weird races fixed.
CADETMatch can now generate and run examples that cover nearly every feature of CADETMatch.
If you are testing a new version of CADET or changes made to CADET or CADETMatch you can use this as a test suite to make sure your changes have not caused unintended consequences.
Generate examples
python -m CADETMatch --generateexamples directory --cadetexamples path to cadet-binary
Run examples
python -m CADETMatch --run_examples directory
Clean all results
python -m CADETMatch --clean_examples directory
- HTML
Published by Immudzen over 5 years ago
https://github.com/cadet/cadet-match - MCMC MLE changes
MCMC MLE has been almost completely redone. The KDE is sampled to find initial points to optimize and all points in the entire data set are evaluated once to find the best point and use as a starting point.
The MCMC system uses a 5-phase optimization now where the highest probabilities are for the bounds checking and burn in phases with a 5 step restart process (thanks emcee developers for the help dealing with stuck walkers)
- HTML
Published by Immudzen over 5 years ago
https://github.com/cadet/cadet-match - New PyPi release
Just some version requirement changes and a release to PyPi
- HTML
Published by Immudzen over 5 years ago
https://github.com/cadet/cadet-match - Conda prep work
Prep work for a conda package release
- HTML
Published by Immudzen over 5 years ago
https://github.com/cadet/cadet-match - Deterministic smoothing and AltScore added
Smoothing is deterministic now. With this release just about everything other than the GA and MCMC MLE selection should be deterministic or close to deterministic.
Also added a new search algorithm called AltScore which simplifies taking the final result of a run and evaluation it under different scoring systems. ScoreTest was used for this before. It can automatically read the meta front from the run it is based on so that no strings are involved and the code is simpler to rerun.
- HTML
Published by Immudzen over 5 years ago
https://github.com/cadet/cadet-match - More determinism (and close determinism)
Most scores are now deterministic or close to deterministic. Based on tests and depending on the problem this can be up to 6x faster convergence. Basically most of the places that used minimization have either been removed or have extremely narrow search ranges to work on after sampling.
All fractionation has been converted to use spline integrals. Splines where already constructed and using trapz is slower and less accurate.
This improves MCMC and Parameter Estimation.
All of this came out of tracking down a few odd bugs that showed up when doing some recent error models.
- HTML
Published by Immudzen almost 6 years ago
https://github.com/cadet/cadet-match - Determinism in offsets
The biggest change in this version is that time offsets for shape functions are deterministic now. They are also faster than before under testing.
MCMC tube plotting uses the same kernel density estimator created during finding the MLE.
Also lots of small fixes that would cause simulations to fail when peaks could not be found.
- HTML
Published by Immudzen almost 6 years ago
https://github.com/cadet/cadet-match - MCMC changes
Added new auto scores auto, autokeq and autoinverse which automatically switched between a linear 0 to 1 transform and a log transform 0 to 1 scale based on the upper and lower bounds. This helps MCMC and normal usage.
autoinverse is built for filmdiffusion and particle_diffusion estimations and error modeling to help deal with the coupling between them
Creating the error model is no longer done with random sampling. A sobol sequence is used now and the error model is now much more predictably created and the distributions used much more closely match the real distributions.
- HTML
Published by Immudzen almost 6 years ago
https://github.com/cadet/cadet-match - More MCMC fixes
Fixed a few other problems related to MCMC reading priors that occur much further on in the simulation.
- HTML
Published by Immudzen almost 6 years ago
https://github.com/cadet/cadet-match - MCMC fixes
MCMC was not loading prior probability distributions and this should be fixed now.
- HTML
Published by Immudzen almost 6 years ago
https://github.com/cadet/cadet-match - Fractionation graph changes
Fractionation graphs changed to a stair step instead of midpoint.
- HTML
Published by Immudzen almost 6 years ago
https://github.com/cadet/cadet-match - Minimization
Outputs have been changed to minimization and graphs have been updated accordingly. This means that all methods now show minimization for progress and scores.
New noise feature for MCMC. Bases on analysis a new noise feature was added called uvnoise norm that takes the a multiplier and a standard deviation for it "uvnoise_norm": [1.0, 0.001] This is based on noise in UV signal being closely related to the currently measured value and in experimental analysis it looks like this is nearly constant.
- HTML
Published by Immudzen almost 6 years ago
https://github.com/cadet/cadet-match - Smoothing and minor fixes
Smoothing now resamples the data if the data is not uniformly spaced. This improves how well the smoothing works. There are also a number of small fixes related to using splines more efficiently and cleaning up some code.
- HTML
Published by Immudzen almost 6 years ago
https://github.com/cadet/cadet-match - Packaging fix
This is a new version that is put out just to fix a packaging issue. It turns out some of the problems where from the build directory for the wheel not getting cleaned so old files where being includes which was causing problems. This should be fixed now.
- HTML
Published by Immudzen almost 6 years ago
https://github.com/cadet/cadet-match - boundOffset fix
This is just a tiny bug fix for an error related to boundOffset that turned up when running a simulation.
- HTML
Published by Immudzen almost 6 years ago
https://github.com/cadet/cadet-match - Fixed obsolete scores trying to import scoop
This is just a minor release that deals with a problem related to code trying to import scoop which is no longer used
- HTML
Published by Immudzen almost 6 years ago
https://github.com/cadet/cadet-match - Handle blanks in fractionation and new dependency changes
Blank entries are now handled in fractionation data and skipped
Dependencies have been updated and when CADETMatch is run it writes out the version of all libraries along with the version of those libraries this version was tested against.
- HTML
Published by Immudzen almost 6 years ago
https://github.com/cadet/cadet-match - Better stopping
Measuring progress for these algorithms can be difficult. The system should better recognize when significant progress is being made for single and multiple-objectives and should stop when significant progress is no longer being made. When only a single objective was used it was regarding all progress as significant no matter how small which caused the algorithm not to halt.
- HTML
Published by Immudzen about 6 years ago
https://github.com/cadet/cadet-match - Shape, ShapeFront, ShapeBack decay works now
Fixed a minor problem with Shape, ShapeFront and ShapeBack so that decay:0 and decay:1 works correctly.
- HTML
Published by Immudzen about 6 years ago
https://github.com/cadet/cadet-match - removed cadet gradient support
Removed analytical cadet gradient support. The code was not working correctly and interfered with other code.
Updated to a newer version of sklearn and fixed a problem related to using index in transform.
- HTML
Published by Immudzen about 6 years ago
https://github.com/cadet/cadet-match - Stable release
This release has many small bug fixes and improvements to the code and to make it faster so that less time is spent in python code and more time in cadet.
- HTML
Published by Immudzen about 6 years ago
https://github.com/cadet/cadet-match - Faster scatter plots
As the number of simulations rises so does the number of points in the scatter plots. On systems with millions of simulations this involves millions of points in the scatter plot which is really slow with matplotlib. The system now merges close points together and scales up the size so that fewer points are drawn. This results in a massive speed increase for large numbers of points. In one case it sped up the system from an hour to a few seconds.
- HTML
Published by Immudzen about 6 years ago
https://github.com/cadet/cadet-match - DextranShape infinite loop
Fixed an error that occurs in DextranShape under very rare circumstances that can cause an infinite loop.
- HTML
Published by Immudzen about 6 years ago
https://github.com/cadet/cadet-match - New transform system
The transform system has been rewritten to use an abstract base class. There are also two new transforms normmult and normadd which allow you to add or multiply another variable. For example you can estimate nu for one component and then for a higher molecular weight species say nu is 1.5 to 2.5 times the main species.
Blocked a buggy version of openpyxl.
- HTML
Published by Immudzen over 6 years ago
https://github.com/cadet/cadet-match - Generation updates
Autocorrelation graphs are generate now (which helps with debugging) and they are put in the space/mcmc dir.
progresselapsedtime has been added as a new config option which defaults to 300 seconds (5 minutes)
Every progresselapsedtime interval the system will write out the progress within the generation (percent complete, number of sims complete vs number of sims total). This is really helpful for slow generations with large populations to see how the system is progressing.
- HTML
Published by Immudzen over 6 years ago
https://github.com/cadet/cadet-match - MCMC data changes
More data is stored from MCMC now and the process is simpler making it easier to store other information in the future. One of the biggest changes is during the bounds modification step the previous bounds, new bounds and 5% and 95% values are written into mcmc.h5
The system also automatically uses these bounds if a prior is used in a simulation. Since the MCMC interface is pretty stable now it makes sense to make more parts of it automatic.
Another thing that has changed which gives a decent speedup is the sampler is no longer reset at the end of burn in so the random number generator and population are kept intact since this represents the burned in state.
Other minor things include a new version.py file that populates a lot of version, author, email, license etc information in the code. This also means that CADETMatch.version works now.
- HTML
Published by Immudzen over 6 years ago
https://github.com/cadet/cadet-match - Fractionation Graphs
Updated fractionation graphs. The graphs are larger and the time offsets are shown so that the graphs are easier to read.
- HTML
Published by Immudzen over 6 years ago
https://github.com/cadet/cadet-match - Tutorial working again and powell optimization
One of the earlier changes with how directories are created broke the tutorials. They should work again.
Added optional Powell optimization (2nd order local gradient free optimizer) when gradient optimization is used to the system. Just set
"localRefine": "powell",
in the json file.
- HTML
Published by Immudzen over 6 years ago
https://github.com/cadet/cadet-match - Score changes
Important update to the score system
DextranShape, Shape, ShapeBack, ShapeFront
have been updated so that they can run in decay or normal mode and use the derivative or not use the derivative, Their default is to do what they did before but an additional option can be added to configure each of them. This means that so many more scores don't need to be created so Shape can do everything ShapeDecay could do and also cover other cases
To configure the code in the features section for each score set
"derivative": 1 or 0 "decay": 1 or 0
Many of the existing scores are just minor variations on other scores and this causes 4 version to exist for each type of score. Part of the reason for creating this is that sometimes experimental data has glitches in it and while the main curve is smooth enough the derivative (even with smoothing) is quite bad and it makes sense to just turn off usage of the derivative but not create a new score just to do that.
- HTML
Published by Immudzen over 6 years ago
https://github.com/cadet/cadet-match - Dextran fixes
Found and fixed the offset issue in DextranSSE and DextranShape
- HTML
Published by Immudzen over 6 years ago
https://github.com/cadet/cadet-match - More minor fixes
DextranShape and DextranSSE have improvements to calculating the SSE and applying smoothing to the system. Fixed a few other minor issues. There is a small tradeoff between smoothing and accuracy for purely synthetic data but necessary for experimental data and sometimes necessary for synthetic data.
- HTML
Published by Immudzen over 6 years ago
https://github.com/cadet/cadet-match - Minor fixes
Some of the plotting for MCMC was not working correctly. This fixes those issues and makes a few adjustments to building the Kernel Density Estimators used in the error model and maximum likelihood calculations.
- HTML
Published by Immudzen over 6 years ago
https://github.com/cadet/cadet-match - Continuous and deterministic is good
MCMC is working again. A bug was found related to async processing in MCMC.
AbsoluteTime and DextranShape are now continuous in time for finding the offset. This is important for MCMC and also improves the Dextran fits. The system now looks at 90% of peak max instead of peak max so that a spline can be fit and the exact offset found.
KDE bandwidth, time offsets and a few other things use powell's method as a local minimizer to give deterministic and faster results while also improves quality of fit and MCMC performance.
- HTML
Published by Immudzen over 6 years ago
https://github.com/cadet/cadet-match - NSGA3 reference points and chromatogram offset
This version picks the reference points for NSGA3 more dynamically so that it reduces the points per dimension as the dimensions increase. A problem was found when using 54 goals that the memory usage grew so long the system would run out of memory. This should be fixed now. A scaling method for reference points was used from the original NSGA3 paper.
Chromatogram offsets are also found now using resampling to 100 points/second and the cross-correlation to find the offset. This method is deterministic and about 300x faster than the previous search method while also being more robust.
- HTML
Published by Immudzen over 6 years ago
https://github.com/cadet/cadet-match - Updated smoothing and more stable time offset
This release mostly has some small updates to the smoothing code and some other things. The smoothing should work for more types of data now. The code for finding the time offset between two signals has also been upgraded to make it more robust.
- HTML
Published by Immudzen over 6 years ago
https://github.com/cadet/cadet-match - emcee 3 and removal of scoop
Scoop has been removed from CADETMatch and the system now uses the python multiprocessing model. This removes the ability to run on multiple nodes (which was not used) but adds VASTLY better debuggability. The code is much easier to debug now including debugging while running in parallel.
Logging has been upgraded as part of this.
Emcee 3.x upgrade is complete. The code uses a combination of differential evolution moves by default and converges faster than before. The system runs a 3 phase MCMC process. The first phase just samples for a while until the 5 and 95 confidence regions stabilize and then resets the upper and lower bounds to center on this region. The second phase does an automatic burn in and adjusts some step size values to get a good acceptance rate and finally the actual run is performed.
- HTML
Published by Immudzen over 6 years ago
https://github.com/cadet/cadet-match - MCMC and smoothing
MCMC has been upgraded to emcee 3.x and the moves have been changed from stretch to a combination of DE moves from https://emcee.readthedocs.io/en/stable/user/moves/ 0.1 DE-snooker 0.81 DE 0.09 DE(gamma=1.0)
The smoothing system has been completely redone. A butter lowpass filter is first applied to remove most noise and a smoothing spline is applied to get a clean derivative.
- HTML
Published by Immudzen over 6 years ago
https://github.com/cadet/cadet-match - Fixed smoothing spline
The new smoothing spline code failed when used without data normalization and the values where small (<1e-4 or so) which broke some of the searching. This has been fixed.
- HTML
Published by Immudzen over 6 years ago
https://github.com/cadet/cadet-match - Better splines and multistart
This version has a new data smoothing system for dealing with experimental data. It also has an improved Multistart which checks the top points instead of all of them.
- HTML
Published by Immudzen over 6 years ago
https://github.com/cadet/cadet-match - Jupyter Interrupt
Normal search can now be interrupted from inside the jupyter notebook (when inside the gradient search you still can't interrupt the system)
- HTML
Published by Immudzen over 6 years ago
https://github.com/cadet/cadet-match - Workshop try again
Put a limit on emcee version and fixed a few other small bugs
- HTML
Published by Immudzen over 6 years ago
https://github.com/cadet/cadet-match - Hopefully final workshop version
This version has improved graphs and a dedicated gradient score that needs a starting point (for use in the workshop)
- HTML
Published by Immudzen over 6 years ago
https://github.com/cadet/cadet-match - MCMC working again
MCMC is working again after some bugs were found and fixed with the version readied for PyPi More settings have sane default values and can just be left at their defaults.
- HTML
Published by Immudzen over 6 years ago
https://github.com/cadet/cadet-match - Fixed SSE Fractionation
Fractionation SSE has been fixed. This version has been pushed to PyPi also to make it easier for the workshop.
- HTML
Published by Immudzen over 6 years ago
https://github.com/cadet/cadet-match - CADETMatch more automatic v2
Last release had a bug that broke some scores. This version fixes it.
- HTML
Published by Immudzen over 6 years ago
https://github.com/cadet/cadet-match - CADETMatch more automatic
Absolute tolerance, relative tolerance and timeout are all automatic now. Timeout is especially important since the right timeout for your laptop is not the same as the right timeout for a server.
Use the minimum score to determine progress
- HTML
Published by Immudzen over 6 years ago
https://github.com/cadet/cadet-match - CADET workshop testing
Hopefully this version is ready for the workshop but I am sure some more bugs will be found and fixed
With this version I made the new NSGA3 version the default. I moved a lot of older scores and search algorithms to the obsolete folder. I have also upgraded the sliding fractionation system to use splines which has made the score smoother and improved convergence.
- HTML
Published by Immudzen over 6 years ago
https://github.com/cadet/cadet-match - CADETMatch command line interface
CADETMatch now has a command line interface. Information can be found at https://nbviewer.jupyter.org/github/modsim/CADET-Tutorial/blob/master/CADET-Tutorial/CADETMatch%20command%20line.ipynb while a general tutorial for installing and basic setup for Windows can be found at https://nbviewer.jupyter.org/github/modsim/CADET-Tutorial/blob/master/CADET-Tutorial/CADET%20Install%20windows.ipynb .
This version also has better defaults and other upgrades based on writing the Windows installation instructions.
- HTML
Published by Immudzen over 6 years ago
https://github.com/cadet/cadet-match - Depend on CADET-Python
This version now depends on CADET-python and no longer includes the python interface in it.
- HTML
Published by Immudzen over 6 years ago
https://github.com/cadet/cadet-match - PyPi release
This is the first version being released to PyPi in advance of the CADET workshop.
- HTML
Published by Immudzen over 6 years ago