eventdisplay_v4 - Eventdisplay - version v490.7.2

Eventdisplay is a reconstruction and analysis pipeline for data of the VERITAS Imaging Atmospheric Cherenkov Telescopes (IACT).

This is a minor release with two small changes:

for MC processing, reduce the number of error and warning messages in case of corrupt VBF files. This avoids generating TB-sized log files
add new option for evndisp: allow to set minimum required number of pedestal events with --minpedestalevents <int>. Default value is 50.

- C++
Published by GernotMaier 8 months ago

eventdisplay_v4 - Eventdisplay - version v490.7.1

Eventdisplay is a reconstruction and analysis pipeline for data of the VERITAS Imaging Atmospheric Cherenkov Telescopes (IACT).

This is a minor release, allowing to train BDT regression trees per telescopes (note that single telescope analysis is now switched off).

What's Changed

Remove single telescope analysis in mva angular resolution training by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/295

Full Changelog: https://github.com/VERITAS-Observatory/EventDisplay_v4/compare/v490.7...v490.7.1

- C++
Published by GernotMaier about 1 year ago

eventdisplay_v4 - Eventdisplay - release v491.0

Eventdisplay is a reconstruction and analysis pipeline for data of the VERITAS Imaging Atmospheric Cherenkov Telescopes (IACT).

Version v491 is a major release with significant changes in the reconstruction algorithm and important bug fixes.

Detailed release note and tests can be found in the dedicated v491 release repository, see here.

Main Improvements

significant improved high-energy reconstruction by
- allow to fill lookup tables with loss factors larger than those used in the evndisp stage.
- allow to re-analyse events with loss factors larger than those used in the evndisp stage.
- improved dispBDT reconstruction with much better hyper parameter settings (derived from optimization on CTAO simulations)
- added dispBDT energy reconstruction to allow to reconstruct events with large loss factors (migrated from CTAO analysis)
- improved BDT gamma/hadron separation; addition of DispAbsSumWeigth variable for gamma/hadron separation
telescope-dependent pedestal variations are now taken correctly into account in lookup table code
- before: tables where accessed using the average pedvars
- this change requires many and deep changes in both lookup table writing and reading codes.

What's Changed

v490.9 dev1 - improve selection of images on mscw stage; bug fix in event selection for dispAnalyzer by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/254
v490.9-dev2 - removal of size2 by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/265
fix reading of size by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/266
v490.9-dev4 - minor fixes by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/267
Bump docker/build-push-action from 5 to 6 by @dependabot in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/268
increase precision of MJD printout to be useful. by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/269
improved help for logFile tool by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/270
BDT disp training for different loss cuts by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/271
Update to astyle 3.6 by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/273
v491 dev1 - plotting tool for mscw variables. by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/272
v491 dev2 - Adding BDTdispEnergy by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/274
v491-dev 3: MVA interpolation fix by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/275
v491-dev4 by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/276
v491 dev5 - bug fixes in lookup table reading by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/277
v491-dev6 - minor fixes for release candidates by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/278
v491 dev7 by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/279
v491 dev8 (minor plottting updates) by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/280
v491 - final release candidate by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/282

Full Changelog: https://github.com/VERITAS-Observatory/EventDisplay_v4/compare/v490.8...v491.0

- C++
Published by GernotMaier about 1 year ago

eventdisplay_v4 - Eventdisplay - beta release v491-rc2

Eventdisplay is a reconstruction and analysis pipeline for data of the VERITAS Imaging Atmospheric Cherenkov Telescopes (IACT).

v491-rc2 is the second release candidate for v491. It is possible to use this for tests and careful science analysis

v491-rc2 includes a bug fix in TMVAEvaluator.

Version v491 is a major release with significant changes in the reconstruction algorithm and important bug fixes.

Improvements

significant improved high-energy reconstruction by
- allow to fill lookup tables with loss factors larger than those used in the evndisp stage.
- allow to re-analyse events with loss factors larger than those used in the evndisp stage.
- improved dispBDT reconstruction with much better hyper parameter (derived from optimization on CTAO simulations)
- added dispBDT energy reconstruction to allow to reconstruct events with large loss factors
- improved BDT gamma/hadron separation; addition of DispAbsSumWeigth variable for gamma/hadron separation
telescope-dependent pedestal variations are now taken correctly into account in lookup table code
- before: tables where accessed using the average pedvars
- this change requires many and deep changes in both lookup table writing and reading codes.

What's Changed

v491-dev6 - minor fixes for release candidates by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/278

Full Changelog: https://github.com/VERITAS-Observatory/EventDisplay_v4/compare/v491-beta.3...v491-rc2

- C++
Published by GernotMaier over 1 year ago

eventdisplay_v4 - Eventdisplay - beta release v491-beta.4

Eventdisplay is a reconstruction and analysis pipeline for data of the VERITAS Imaging Atmospheric Cherenkov Telescopes (IACT).

v491-rc1 is the first release candidate for v491. It is possible to use this for tests and careful science analysis

Version v491 is a major release with significant changes in the reconstruction algorithm and important bug fixes.

Improvements

significant improved high-energy reconstruction by
- allow to fill lookup tables with loss factors larger than those used in the evndisp stage.
- allow to re-analyse events with loss factors larger than those used in the evndisp stage.
- improved dispBDT reconstruction with much better hyper parameter (derived from optimization on CTAO simulations)
- added dispBDT energy reconstruction to allow to reconstruct events with large loss factors
- improved BDT gamma/hadron separation; addition of DispAbsSumWeigth variable for gamma/hadron separation
telescope-dependent pedestal variations are now taken correctly into account in lookup table code
- before: tables where accessed using the average pedvars
- this change requires many and deep changes in both lookup table writing and reading codes.

What's Changed

v491-dev6 - minor fixes for release candidates by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/278

Full Changelog: https://github.com/VERITAS-Observatory/EventDisplay_v4/compare/v491-beta.3...v491-rc1

- C++
Published by GernotMaier over 1 year ago

eventdisplay_v4 - Eventdisplay - beta release v491-beta.3

Eventdisplay is a reconstruction and analysis pipeline for data of the VERITAS Imaging Atmospheric Cherenkov Telescopes (IACT).

v491-beta.3 is a pre-release intended for testing. Do not use it for science analysis.

This is a major release with significant changes in the reconstruction algorithm and important bug fixes.

Improvements

significant improved high-energy reconstruction by
- allow to fill lookup tables with loss factors larger than those used in the evndisp stage.
- allow to re-analyse events with loss factors larger than those used in the evndisp stage.
- improved dispBDT reconstruction with much better hyper parameter (derived from optimization on CTAO simulations)
- added dispBDT energy reconstruction to allow to reconstruct events with large loss factors
- improved BDT gamma/hadron separation; addition of DispAbsSumWeigth variable for gamma/hadron separation
telescope-dependent pedestal variations are now taken correctly into account in lookup table code
- before: tables where accessed using the average pedvars
- this change requires many and deep changes in both lookup table writing and reading codes.

What's Changed

v491 dev5 - bug fixes in lookup table reading by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/277

Full Changelog: https://github.com/VERITAS-Observatory/EventDisplay_v4/compare/v491-beta.2...v491-beta.3

- C++
Published by GernotMaier over 1 year ago

eventdisplay_v4 - Eventdisplay - beta release v491-beta.2

Eventdisplay is a reconstruction and analysis pipeline for data of the VERITAS Imaging Atmospheric Cherenkov Telescopes (IACT).

v491-beta2 is a pre-release intended for testing. Do not use it for science analysis.

This is a major release with significant changes in the reconstruction algorithm and important bug fixes.

Improvements

significant improved high-energy reconstruction by
- allow to fill lookup tables with loss factors larger than those used in the evndisp stage.
- allow to re-analyse events with loss factors larger than those used in the evndisp stage.
- improved dispBDT reconstruction with much better hyper parameter (derived from optimization on CTAO simulations)
- added dispBDT energy reconstruction to allow to reconstruct events with large loss factors
- improved BDT gamma/hadron separation; addition of DispAbsSumWeigth variable for gamma/hadron separation
telescope-dependent pedestal variations are now taken correctly into account in lookup table code
- before: tables where accessed using the average pedvars
- this change requires many and deep changes in both lookup table writing and reading codes.

What's Changed

v491-dev4 by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/276

Full Changelog: https://github.com/VERITAS-Observatory/EventDisplay_v4/compare/v491-beta.1...v491-beta.2

- C++
Published by GernotMaier over 1 year ago

eventdisplay_v4 - Eventdisplay - beta release v491-beta.1

Eventdisplay is a reconstruction and analysis pipeline for data of the VERITAS Imaging Atmospheric Cherenkov Telescopes (IACT).

v491-beta1 is a pre-release intended for testing. Do not use it for science analysis.

This is a major release with significant changes in the reconstruction algorithm and important bug fixes.

Improvements:

significant improved high-energy reconstruction by
- allow to fill lookup tables with loss factors larger than those used in the evndisp stage.
- allow to re-analyse events with loss factors larger than those used in the evndisp stage.
- improved dispBDT reconstruction with much better hyper parameter (derived from optimization on CTAO simulations)
- added dispBDT energy reconstruction to allow to reconstruct events with large loss factors
- improved BDT gamma/hadron separation; addition of DispAbsSumWeigth variable for gamma/hadron separation

Bug fixes:

replaced 'closes-bin' algorithm for BDT gamma/hadron tree selection by interpolation in zenith. This fixes the issue reported in ELOG 2597, with clearly visible effects from the binning of the BDTs in zenith angle.

What's Changed

v490.9 dev1 - improve selection of images on mscw stage; bug fix in event selection for dispAnalyzer by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/254
v490.9-dev2 - removal of size2 by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/265
fix reading of size by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/266
v490.9-dev4 - minor fixes by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/267
Bump docker/build-push-action from 5 to 6 by @dependabot in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/268
increase precision of MJD printout to be useful. by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/269
improved help for logFile tool by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/270
BDT disp training for different loss cuts by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/271
Update to astyle 3.6 by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/273
v491 dev1 - plotting tool for mscw variables. by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/272
v491 dev2 - Adding BDTdispEnergy by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/274
v491-dev 3: MVA interpolation fix by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/275

Full Changelog: https://github.com/VERITAS-Observatory/EventDisplay_v4/compare/v490.8...v491-beta.1

- C++
Published by GernotMaier over 1 year ago

eventdisplay_v4 - Eventdisplay - version v490.8

Eventdisplay is a reconstruction and analysis pipeline for data of the VERITAS Imaging Atmospheric Cherenkov Telescopes (IACT).

This is a maintenance release with no relevant updates for users and an intermediate release (no associated releases of scripts or auxiliary files).

For details on how to install the release, see this readme.

What's Changed

Remove leftover from CTA by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/262
code cleanup by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/264

Full Changelog: https://github.com/VERITAS-Observatory/EventDisplay_v4/compare/v490.7...v490.8

- C++
Published by GernotMaier almost 2 years ago

eventdisplay_v4 - Eventdisplay - version v490.7

Eventdisplay is a reconstruction and analysis pipeline for data of the VERITAS Imaging Atmospheric Cherenkov Telescopes (IACT).

This is a minor release with small improvements:

new application printCrabSensitivity for easier access to sensitivity vs time estimations from Crab fluxs
improved docker file

The usage of Eventdisplay requires code, scripts, and analysis files. For this release, the corresponding repositories are:

For details on how to install the release, see this readme.

What's Changed

v490.7 preprocessing docker with reduced docker image size by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/260
Add a simple tool to printout Crab sensitivity by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/261

Full Changelog: https://github.com/VERITAS-Observatory/EventDisplay_v4/compare/v490.6...v490.7

- C++
Published by GernotMaier almost 2 years ago

eventdisplay_v4 - Eventdisplay - version v490.6

Eventdisplay is a reconstruction and analysis pipeline for data of the VERITAS Imaging Atmospheric Cherenkov Telescopes (IACT).

This is a minor bug fix bug fix release addressing the following issues:

docker metadata
mscw quality cuts
DispAnalyzer - correct fui cut applied (this was a bug: loss value was used for direction reconstruction)

The usage of Eventdisplay requires code, scripts, and analysis files. For this release, the corresponding repositories are:

For details on how to install the release, see this readme.

What's Changed

In radial acceptance, skip runs with missing multiplicity by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/255
v490.6 dev2 docker updates by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/256
v490.6 - add additional cuts in mscw for event quality by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/257
v490.6-dev3 container metadata updates by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/258

Full Changelog: https://github.com/VERITAS-Observatory/EventDisplay_v4/compare/v490.5...v490.6

- C++
Published by GernotMaier about 2 years ago

eventdisplay_v4 - Eventdisplay - version v490.5

Eventdisplay is a reconstruction and analysis pipeline for data of the VERITAS Imaging Atmospheric Cherenkov Telescopes (IACT).

This is a minor bug fix bug fix release addressing the following issues:

fix compilation issue with newer root version (mathmore package)
improve Crab spectrum plotting

The usage of Eventdisplay requires code, scripts, and analysis files. For this release, the corresponding repositories are:

Eventdisplay v490.5
EventdisplayAnalysisScriptsVTS v490.5
EventdisplayAnalysisFilesVTS v490.5 (private repository)

For details on how to install the release, see this readme.

What's Changed

v490.5-dev1 by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/249

Full Changelog: https://github.com/VERITAS-Observatory/EventDisplay_v4/compare/v490.4...v490.5

- C++
Published by GernotMaier about 2 years ago

eventdisplay_v4 - Eventdisplay - version v490.4

Eventdisplay is a reconstruction and analysis pipeline for data of the VERITAS Imaging Atmospheric Cherenkov Telescopes (IACT).

This is an important bug fix bug fix release for v490.3 addressing the following issues:

relative gains and time offsets are not written to output files for flasher analysis

The usage of Eventdisplay requires code, scripts, and analysis files. For this release, the corresponding repositories are:

Eventdisplay v490.4
EventdisplayAnalysisScriptsVTS v490.4
EventdisplayAnalysisFilesVTS v490.3 (private repository; note that v490.3 is the correct version for the analysis files)

For details on how to install the release, see this readme.

What's Changed

critical bug fix in flasher gain calculation by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/248

Full Changelog: https://github.com/VERITAS-Observatory/EventDisplay_v4/compare/v490.3...v490.4

- C++
Published by GernotMaier over 2 years ago

eventdisplay_v4 - Eventdisplay - version v490.3

Eventdisplay is a reconstruction and analysis pipeline for data of the VERITAS Imaging Atmospheric Cherenkov Telescopes (IACT).

This is a very minor bug fix release for v490.2 addressing the following issues:

fix compilation issues for VTS.analyseMuonRings
improved printout for reflected region positions and sky map center

The usage of Eventdisplay requires code, scripts, and analysis files. For this release, the corresponding repositories are: - Eventdisplay v490.3 - EventdisplayAnalysisScriptsVTS v490.3 - EventdisplayAnalysisFilesVTS v490.3 (private repository)

For details on how to install the release, see this readme.

What's Changed

v490.3 dev1 - minor improvements by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/244
v490.3-dev by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/246

Full Changelog: https://github.com/VERITAS-Observatory/EventDisplay_v4/compare/v490.2...v490.3

- C++
Published by GernotMaier over 2 years ago

eventdisplay_v4 - Eventdisplay - version v490.2

Eventdisplay is a reconstruction and analysis pipeline for data of the VERITAS Imaging Atmospheric Cherenkov Telescopes (IACT).

This is a minor bug fix release for v490.1 addressing the following issues:

Epochs where not updated correctly when using printRunParameter with the -updated-runinfo options.

The usage of Eventdisplay requires code, scripts, and analysis files. For this release, the corresponding repositories are: - Eventdisplay v490.2 - EventdisplayAnalysisScriptsVTS v490.2 - EventdisplayAnalysisFilesVTS v490.2 (private repository)

For details on how to install the release, see this readme.

What's Changed

v490.2 - dev1 by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/239
bug fix in updating epoch string by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/240

Full Changelog: https://github.com/VERITAS-Observatory/EventDisplay_v4/compare/v490.1...v490.2

- C++
Published by GernotMaier over 2 years ago

eventdisplay_v4 - Eventdisplay - version v490.1

Eventdisplay is a reconstruction and analysis pipeline for data of the VERITAS Imaging Atmospheric Cherenkov Telescopes (IACT).

This is a minor bug fix release for v490.0 addressing the following issues:

adapt radial acceptance code to new directory structure of gamma/hadron BDTs (elog 2427)
replace in Display the plotting average by sum of traces (see ELog 2403)
minor axes changes for plotting of Crab spectra
add printout on downloading status for bin/VTS.getRunListFromDB
bug fix in pedvar calculation

Full Changelog: https://github.com/VERITAS-Observatory/EventDisplay_v4/compare/v490.0...v490.1

The usage of Eventdisplay requires code, scripts, and analysis files. For this release, the corresponding repositories are: - Eventdisplay v490.1 - EventdisplayAnalysisScriptsVTS v490.1 - EventdisplayAnalysisFilesVTS v490.1a (private repository)

For details on how to install the release, see this readme.

- C++
Published by GernotMaier over 2 years ago

eventdisplay_v4 - Eventdisplay - version v490.0

Eventdisplay v490 is a significant update with many changes to the code compared to previous versions.

This is the first release of v490 tested for V6 analysis. Eventdisplay v490 is ready for publication-type analysis

The default analysis methods for v490 are: - AP cleaning (new default in scripts) - dispBDT direction reconstruction - gamma/hadron separation using BDTs (for reduced high-voltage operations: soft box cuts)

The introduction of these methods provide a significant improvement in reconstruction quality and sensitivity, see the comparison with v487 for several targets.

The usage of Eventdisplay requires code, scripts, and analysis files. For this release, the corresponding repositories are: - Eventdisplay v490.0 - EventdisplayAnalysisScriptsVTS v490.0 - EventdisplayAnalysisFilesVTS v490.0

For details on how to install the release, see the this readme.

Main changes.

Eventdisplay v490 adds significant improvements to the analysis

New analysis methods:

AP cleaning (image/border cleaning with improved afterpulsing robustness)
optimised next-neighbour cleaning (NN cleaning; improved low-energy threshold)
dispBDT direction reconstruction (improved angular resolution at low elevations / large offsets)
improvements for DL3 / gammapy interface
improved gamma/hadron separation code (including critical bug fixes)

Code maintenance: - code cleanup and removal of lot of obsolete code (EventDisplay_v4) - parameter files cleanup EventdisplayAnalysisFilesVTS) - scripts improvements (EventdisplayAnalysisScriptsVTS)

Detailed release notes:

Afterpulsing cleaning (image/border cleaning with improved robustness)

removes islands consisting of a single image and a single border pixel
based on Jamie's suggestion presented in this VERITAS memo
see VERITAS memo on AP cleaning in Eventdisplay
implementation of median and pedestal 68% containment for testing of image/border cleaning settings. These values are calculated and written to the pedestal output file, but not used in the analysis.
AP cleaning is the new default cleaning method
switch this on by adding one parameter 1 to * -1 IMAGECLEANINGTHRESHOLDS 5. 2.5 1 in the runparameter file
use runparameter file EVNDISP.reconstruction.runparameter.AP.v4x

Optimised next-neighbour cleaning for improved low-energy reconstruction

based on M.Shayduk et al, arXiv:1307.4939
for VERITAS implementation, see Maria's presentations (e.g. (Winter 2023 collaboration meeting)[https://veritas.sao.arizona.edu/GammaWiki/images/5/54/VCMWinter2023NNKherlakian.pdf]
use runparameter file EVNDISP.reconstruction.runparameter.AP.v4x
integration deals with short runs (~5 min long) with low number of events required for IPR Graphs by averaging over all four telescopes
note that this requires to a separate evndisp analysis run, dedicated lookup tables and instrument response functions.
significant (factor x2!) improvement in run time

Epoch and throughput definitions

Epochs are now defined in terms of dates and not run numbers (see VERITAS.Epochs.runparameter)
Remove duplication of date/MJD in season definition. Having both MJD and SQL-strings lead to errors in definitions of yearly seasons. Removed the MJD, which is now calculated in VEvndispRunParameter from the sql string.

BDT gamma/hadron separation

vastly improved efficiency for BDT training
significantly improved cut optimisation with consistent background rate calculation
use MC-generated effective areas for signal and any set of background runs for background calculation (before: restriction to Crab Nebula runs); this increases significantly the available statistics
overlapping energy bins for training; this removes steps in effective areas

Overwrite of runinfo DB values

Command line parameters to overwrite target runinfo DB values: Add possibility to overwrite the DB settings for target information (read usually from the runinfo db table): -overwritedb_rightascension, -overwritedb_declination, -overwritedb_target (this complements the existing command line options for the wobble offset: overwritedb_wobblenorth and overwritedb_wobbleeast)

evndisp robustness against missing events

ncreased robustness for reading number of samples of readout window from VBF files. This recovers those runs, where for some reason the more than 10,000 events in the beginning of the run are empty, but the rest of the run is ok.

New dispBDT direction reconstruction algorithms for improved direction reconstruction at low zenith angles and large offsets

not available in this beta.1 release
see VERITAS memo on dispBDTs
dispBDT reconstruction is integrated in the mscw_energy binary, meaning that to redo the reconstruction a re-run of evndisp is not necessary.
improved gamma/hadron separation:
significant improvement to efficiency of trainTMVAforGammaHadronSeparation with a large reduction of the memory footprint (which was beyond 20 GB) and improvements in execution speed
add Size2ndMax and NImages as training parameters (improved signal/background efficiency, see ELOG 2357)
write a temporary root file with events after signal/background pre-cuts is written to disk and used in a second step for the training (TMVA unfortunately reads in all events before cuts)
a random fraction of events is selected from each input file to match the number of training events requested
if there are less events, the training option string is adjusted and the training proceeds with a lower number of events than requested (!!)

Support for DL3 export using V2DL3 and gammapy

see VERITAS memo by Sonal
- added propagation of VPM info to DL3 level (allows to select runs with / without VPM monitoring data)

Improved MC / Data comparison with additional plots for theta2 and disp

weighting of events according to the azimuth of the data runs
Improvements in plotting of instrument response functions
re-enable plotting of instrument response functions vs true energy: Effective area files contain since v497 two trees: fEffArea (with reconstructed energy axis) and fEffAreaH2F (with true energy axis). Change reading routines to re-enable plotting of IRFs vs true energy.
improve angular resolution plotting and fix issue with plotting cumulative theta2 distributions.
improved reading and generation of energy thresholds and lookup table plots

Evndisp trace integration

performance updates in trace integration and TZero calculation

Lookup tables

Energy axes for energy reconstruction lookup tables
removed dynamic setting of energy axes limits (1D histogram for tables) in lookup table code and fixed it to 150 TeV (well beyond what we do in VERITAS). Dynamic settings were introduced for CTA (telescope types), but the version in Eventdisplay_v4 had a minor bug
more flexibility to find correct NSB bins when combing lookup tables; add command line parameters noise tolerance in combineEffectiveAreas
Remove obsolete TableEnergy filling

Addition of cluster cleaning algorithms

use EVNDISP.reconstruction.runparameter.CC.v4x
not part of the standard analysis (but feel free to test)

Automatic container generation

integration of some automatic docker container generation with each release (see dockerfiles directory
newest images are available for each release from here: https://github.com/VERITAS-Observatory/EventDisplayv4/pkgs/container/eventdisplayv4
(this needs a bit further testing)

Code maintenance

improved code documentation
use consistently now a code linter
- requires installation of the unix tool astyle. Run make formatSourceCode before submitting any code
removal of lots of obsolete and not maintained code
- CTA related code (use this Eventdisplay for CTA Analysis)
- frogs and Model3d (code very much out of date)
cleanup of compilation warnings and C++17
- General cleanup of compilation warnings and changes to allow for C++ 14 and C++ 17compilation.
- test this by CI matrix on Github (which consist of testing compilation and linking on root:6.24.06-centos7, root:6.22.08-ubuntu20.04, root:6.26.06-ubuntu22.04, root:6.26.06-fedora35, root:latest)

Image/border pixel writer

writing of image/border pixel lists on evndisp stage (optional, very useful debugging of cleaning studies)
- Option -writeimagepixellist to evndisp allows to write out list of pixel (with integrated charge and timing) surviving the image cleaning.
- This option increases the output files of evndisp by approximately a factor 2.5-3. It is therefore switched off by default.
- Written are image, border pixels, and one additional ring of pixels around them: PixelListN : PixelListN/i PixelID : PixelID[PixelListN]/i PixelType : PixelType[PixelListN]/i PixelIntensity : PixelIntensity[PixelListN]/F PixelTimingT0 : PixelTimingT0[PixelListN]/F PixelType indicates the following:
PixelType == 1: image pixel
PixelType == 2: border pixel
PixelType == 3: neighbour pixel to image/border

DB text functionality

Use DB text files instead of direct query to VERITAS data base
- allows to download information from VERITAS DB (e.g.,run data, VPM data, etc) and re-use it
- avoids overloading the database connection when analysing hundreds of runs simultaneously
- allows to use a basic root docker image for compilation and no need of integration of mysql while compiling Eventdisplay
- requires reading of DB information using the scripts available in the dbscripts directory of EventdisplayAnalysisScripts_VTS
- used automatically by ANALYSIS.evndisp.sh when a run directory is found in ${VERITAS_DATA_DIR}/DBTEXT

Sofa replacement of slalib

Replacement of slalib routines by sofa for all coordinate transformations
- see PR #180
- Eventdisplay has been using a copy of the slalib code since the very beginning (VASlalib). All astronometry routines are replaced by the sofa package (same authors as slalib; but permissive licence) for the following reasons:
  - slalib (as used in Eventdisplay) is not maintained since many years and the status of the routines is unclear (e.g. means that timing routines do not include leap seconds introduced recently)
  - slalib's licence unclear and very likely the current usage Eventdisplay is not allowed
  - sofa is a well maintained IAU package
- #PR 180 lists some tests listing differences between using the two different astronometry packages: generally consistent results with max differences in reconstructions observations of <2 arcseconds.
- Note that this involves the installation of the sofa package before compiling eventdisplay. Use the script install_sofa.sh to download and install the package.

mergeVBF

addition of mergeVBF binary
- mergeVBF is used for the simulation production to merge several (possibly 1000s) of simulation vbf files into one file.

Readout window

Increased robustness for reading number of samples of readout window from VBF files. This recovers those runs, where for some reason the more than 10,000 events in the beginning of the run are empty, but the rest of the run is ok.

- C++
Published by GernotMaier almost 3 years ago

eventdisplay_v4 - Eventdisplay - version v490-beta.2

Eventdisplay v490 is a significant update with many changes compared to previous version.

This is beta.2 release - very close to final release but still not ready for publication-type analysis

Available for analysis testing: - AP cleaning (new default in scripts) - dispBDT direction reconstruction - gamma/hadron separation using BDTs. - testing should be done only for nominal/red HV observations at wobble offset 0.5 for V6 epochs.

This release should be used together with - Eventdisplay v490-beta.2 - EventdisplayAnalysisScriptsVTS v490-beta.2 - EventdisplayAnalysisFilesVTS v490-beta.2.1

Main changes.

Eventdisplay v490 adds significant improvements to the analysis

New analysis methods:

AP cleaning (image/border cleaning with improved afterpulsing robustness)
optimised next-neighbour cleaning (NN cleaning; improved low-energy threshold)
dispBDT direction reconstruction (improved angular resolution at low elevations / large offsets)
improvements for DL3 / gammapy interface
improved gamma/hadron separation code (including critical bug fixes)

Code maintenance: - code cleanup and removal of lot of obsolete code (EventDisplay_v4) - parameter files cleanup EventdisplayAnalysisFilesVTS) - scripts improvements (EventdisplayAnalysisScriptsVTS)

Detailed release notes:

Afterpulsing cleaning (image/border cleaning with improved robustness)

removes islands consisting of a single image and a single border pixel
based on Jamie's suggestion presented in this VERITAS memo
see VERITAS memo on AP cleaning in Eventdisplay
implementation of median and pedestal 68% containment for testing of image/border cleaning settings. These values are calculated and written to the pedestal output file, but not used in the analysis.
AP cleaning is the new default cleaning method
switch this on by adding one parameter 1 to * -1 IMAGECLEANINGTHRESHOLDS 5. 2.5 1 in the runparameter file
use runparameter file EVNDISP.reconstruction.runparameter.AP.v4x

Optimised next-neighbour cleaning for improved low-energy reconstruction

based on M.Shayduk et al, arXiv:1307.4939
for VERITAS implementation, see Maria's presentations (e.g. (Winter 2023 collaboration meeting)[https://veritas.sao.arizona.edu/GammaWiki/images/5/54/VCMWinter2023NNKherlakian.pdf]
use runparameter file EVNDISP.reconstruction.runparameter.AP.v4x
integration deals with short runs (~5 min long) with low number of events required for IPR Graphs by averaging over all four telescopes
note that this requires to a separate evndisp analysis run, dedicated lookup tables and instrument response functions.

Epoch and throughput definitions

Epochs are now defined in terms of dates and not run numbers (see VERITAS.Epochs.runparameter)
Remove duplication of date/MJD in season definition. Having both MJD and SQL-strings lead to errors in definitions of yearly seasons. Removed the MJD, which is now calculated in VEvndispRunParameter from the sql string.

BDT gamma/hadron separation

vastly improved efficiency for BDT training (f
significantly improved cut optimisation with consistent background rate calculation
use MC-generated effective areas for signal and any set of background runs for background calculation (before: restriction to Crab Nebula runs); this increases significantly the available statistics
overlapping energy bins for training; this removes steps in effective areas

Overwrite of runinfo DB values

Command line parameters to overwrite target runinfo DB values: Add possibility to overwrite the DB settings for target information (read usually from the runinfo db table): -overwritedb_rightascension, -overwritedb_declination, -overwritedb_target (this complements the existing command line options for the wobble offset: overwritedb_wobblenorth and overwritedb_wobbleeast)

New dispBDT direction reconstruction algorithms for improved direction reconstruction at low zenith angles and large offsets

not available in this beta.1 release
see VERITAS memo on dispBDTs
dispBDT reconstruction is integrated in the mscw_energy binary, meaning that to redo the reconstruction a re-run of evndisp is not necessary.
improved gamma/hadron separation:
significant improvement to efficiency of trainTMVAforGammaHadronSeparation with a large reduction of the memory footprint (which was beyond 20 GB) and improvements in execution speed
add Size2ndMax and NImages as training parameters (improved signal/background efficiency, see ELOG 2357)
write a temporary root file with events after signal/background pre-cuts is written to disk and used in a second step for the training (TMVA unfortunately reads in all events before cuts)
a random fraction of events is selected from each input file to match the number of training events requested
if there are less events, the training option string is adjusted and the training proceeds with a lower number of events than requested (!!)

Support for DL3 export using V2DL3 and gammapy

see VERITAS memo by Sonal
- added propagation of VPM info to DL3 level (allows to select runs with / without VPM monitoring data)

Improved MC / Data comparison with additional plots for theta2 and disp

weighting of events according to the azimuth of the data runs
Improvements in plotting of instrument response functions
re-enable plotting of instrument response functions vs true energy: Effective area files contain since v497 two trees: fEffArea (with reconstructed energy axis) and fEffAreaH2F (with true energy axis). Change reading routines to re-enable plotting of IRFs vs true energy.
improve angular resolution plotting and fix issue with plotting cumulative theta2 distributions.
improved reading and generation of energy thresholds and lookup table plots

Evndisp trace integration

performance updates in trace integration and TZero calculation

Lookup tables

Energy axes for energy reconstruction lookup tables
removed dynamic setting of energy axes limits (1D histogram for tables) in lookup table code and fixed it to 150 TeV (well beyond what we do in VERITAS). Dynamic settings were introduced for CTA (telescope types), but the version in Eventdisplay_v4 had a minor bug
more flexibility to find correct NSB bins when combing lookup tables; add command line parameters noise tolerance in combineEffectiveAreas
Remove obsolete TableEnergy filling

Addition of cluster cleaning algorithms

use EVNDISP.reconstruction.runparameter.CC.v4x
not part of the standard analysis (but feel free to test)

Automatic container generation

integration of some automatic docker container generation with each release (see dockerfiles directory
newest images are available for each release from here: https://github.com/VERITAS-Observatory/EventDisplayv4/pkgs/container/eventdisplayv4
(this needs a bit further testing)

Code maintenance

improved code documentation
use consistently now a code linter
- requires installation of the unix tool astyle. Run make formatSourceCode before submitting any code
removal of lots of obsolete and not maintained code
- CTA related code (use this Eventdisplay for CTA Analysis)
- frogs and Model3d (code very much out of date)
cleanup of compilation warnings and C++17
- General cleanup of compilation warnings and changes to allow for C++ 14 and C++ 17compilation.
- test this by CI matrix on Github (which consist of testing compilation and linking on root:6.24.06-centos7, root:6.22.08-ubuntu20.04, root:6.26.06-ubuntu22.04, root:6.26.06-fedora35, root:latest)

Image/border pixel writer

writing of image/border pixel lists on evndisp stage (optional, very useful debugging of cleaning studies)
- Option -writeimagepixellist to evndisp allows to write out list of pixel (with integrated charge and timing) surviving the image cleaning.
- This option increases the output files of evndisp by approximately a factor 2.5-3. It is therefore switched off by default.
- Written are image, border pixels, and one additional ring of pixels around them: PixelListN : PixelListN/i PixelID : PixelID[PixelListN]/i PixelType : PixelType[PixelListN]/i PixelIntensity : PixelIntensity[PixelListN]/F PixelTimingT0 : PixelTimingT0[PixelListN]/F PixelType indicates the following:
PixelType == 1: image pixel
PixelType == 2: border pixel
PixelType == 3: neighbour pixel to image/border

DB text functionality

Use DB text files instead of direct query to VERITAS data base
- allows to download information from VERITAS DB (e.g.,run data, VPM data, etc) and re-use it
- avoids overloading the database connection when analysing hundreds of runs simultaneously
- allows to use a basic root docker image for compilation and no need of integration of mysql while compiling Eventdisplay
- requires reading of DB information using the scripts available in the dbscripts directory of EventdisplayAnalysisScripts_VTS
- used automatically by ANALYSIS.evndisp.sh when a run directory is found in ${VERITAS_DATA_DIR}/DBTEXT

Sofa replacement of slalib

Replacement of slalib routines by sofa for all coordinate transformations
- see PR #180
- Eventdisplay has been using a copy of the slalib code since the very beginning (VASlalib). All astronometry routines are replaced by the sofa package (same authors as slalib; but permissive licence) for the following reasons:
  - slalib (as used in Eventdisplay) is not maintained since many years and the status of the routines is unclear (e.g. means that timing routines do not include leap seconds introduced recently)
  - slalib's licence unclear and very likely the current usage Eventdisplay is not allowed
  - sofa is a well maintained IAU package
- #PR 180 lists some tests listing differences between using the two different astronometry packages: generally consistent results with max differences in reconstructions observations of <2 arcseconds.
- Note that this involves the installation of the sofa package before compiling eventdisplay. Use the script install_sofa.sh to download and install the package.

mergeVBF

addition of mergeVBF binary
- mergeVBF is used for the simulation production to merge several (possibly 1000s) of simulation vbf files into one file.

Readout window

Increased robustness for reading number of samples of readout window from VBF files. This recovers those runs, where for some reason the more than 10,000 events in the beginning of the run are empty, but the rest of the run is ok.

- C++
Published by GernotMaier almost 3 years ago

eventdisplay_v4 - Eventdisplay - version v490.alpha.16

This update includes significant changes which break the analysis of older versions at certain stages.

Running this version requires updates to the newest versions of EventdisplayAnalysisScriptsVTS and EventdisplayAnalysisFilesVTS (private repository).

This is a pre-release with substantial changes expected to be implemented in near future. Please try it out, but don't use it for science publications.

Changes with relation to v490-alpha.15 are:

improvements for BDT gamma/hadron separation code (include optimisation code and especially calculateCrabRateFromMC.

- C++
Published by GernotMaier about 3 years ago

eventdisplay_v4 - Eventdisplay - version v490-beta.1

Eventdisplay v490 is a significant update with many changes compared to previous version.

This is the the first beta release - beta.1 - with limited functionality and intended for first testing only.

Available for analysis testing: - AP cleaning (new default in scripts) - nominal direction reconstruction (intersection method) - moderate and soft box cuts at wobble offsets 0.5 degrees - no radial acceptances available (run anasum scripts with option IGNOREACCEPTANCE)

This release should be used together with - Eventdisplay v490-beta.1 - EventdisplayAnalysisScriptsVTS v490-beta.1 - EventdisplayAnalysisFilesVTS v490-beta.1

Main changes.

Eventdisplay v490 adds significant improvement to the analysis

New analysis methods:

AP cleaning (image/border cleaning with improved afterpulsing robustness)
optimised next-neighbour cleaning (NN cleaning; improved low-energy threshold)
dispBDT direction reconstruction (improved angular resolution at low elevations / large offsets)
improvements for DL3 / gammapy interface
improved gamma/hadron separation code

Code maintenance: - code cleanup and removal of lot of obsolete code (EventDisplay_v4) - parameter files cleanup EventdisplayAnalysisFilesVTS) - scripts improvements (EventdisplayAnalysisScriptsVTS)

Detailed release notes:

Afterpulsing cleaning (image/border cleaning with improved robustness)

removes islands consisting of a single image and a single border pixel
based on Jamie's suggestion presented in this VERITAS memo
see VERITAS memo on AP cleaning in Eventdisplay
implementation of median and pedestal 68% containment for testing of image/border cleaning settings. These values are calculated and written to the pedestal output file, but not used in the analysis.
AP cleaning is the new default cleaning method
switch this on by adding one parameter 1 to * -1 IMAGECLEANINGTHRESHOLDS 5. 2.5 1 in the runparameter file
use runparameter file EVNDISP.reconstruction.runparameter.AP.v4x

Optimised next-neighbour cleaning for improved low-energy reconstruction

based on M.Shayduk et al, arXiv:1307.4939
for VERITAS implementation, see Maria's presentations (e.g. (Winter 2023 collaboration meeting)[https://veritas.sao.arizona.edu/GammaWiki/images/5/54/VCMWinter2023NNKherlakian.pdf]
use runparameter file EVNDISP.reconstruction.runparameter.AP.v4x
integration deals with short runs (~5 min long) with low number of events required for IPR Graphs by averaging over all four telescopes
note that this requires to a separate evndisp analysis run, dedicated lookup tables and instrument response functions.

Epoch and throughput definitions

Epochs are now defined in terms of dates and not run numbers (see VERITAS.Epochs.runparameter)
Remove duplication of date/MJD in season definition. Having both MJD and SQL-strings lead to errors in definitions of yearly seasons. Removed the MJD, which is now calculated in VEvndispRunParameter from the sql string.

Overwrite of runinfo DB values

Command line parameters to overwrite target runinfo DB values: Add possibility to overwrite the DB settings for target information (read usually from the runinfo db table): -overwritedb_rightascension, -overwritedb_declination, -overwritedb_target (this complements the existing command line options for the wobble offset: overwritedb_wobblenorth and overwritedb_wobbleeast)

New dispBDT direction reconstruction algorithms for improved direction reconstruction at low zenith angles and large offsets

not available in this beta.1 release
see VERITAS memo on dispBDTs
dispBDT reconstruction is integrated in the mscw_energy binary, meaning that to redo the reconstruction a re-run of evndisp is not necessary.
improved gamma/hadron separation:
significant improvement to efficiency of trainTMVAforGammaHadronSeparation with a large reduction of the memory footprint (which was beyond 20 GB) and improvements in execution speed
add Size2ndMax and NImages as training parameters (improved signal/background efficiency, see ELOG 2357)
write a temporary root file with events after signal/background pre-cuts is written to disk and used in a second step for the training (TMVA unfortunately reads in all events before cuts)
a random fraction of events is selected from each input file to match the number of training events requested
if there are less events, the training option string is adjusted and the training proceeds with a lower number of events than requested (!!)

Support for DL3 export using V2DL3 and gammapy

see VERITAS memo by Sonal
- added propagation of VPM info to DL3 level (allows to select runs with / without VPM monitoring data)

Improved MC / Data comparison with additional plots for theta2 and disp

weighting of events according to the azimuth of the data runs
Improvements in plotting of instrument response functions
re-enable plotting of instrument response functions vs true energy: Effective area files contain since v497 two trees: fEffArea (with reconstructed energy axis) and fEffAreaH2F (with true energy axis). Change reading routines to re-enable plotting of IRFs vs true energy.
improve angular resolution plotting and fix issue with plotting cumulative theta2 distributions.

Evndisp trace integration

performance updates in trace integration and TZero calculation

Lookup tables

Energy axes for energy reconstruction lookup tables
removed dynamic setting of energy axes limits (1D histogram for tables) in lookup table code and fixed it to 150 TeV (well beyond what we do in VERITAS). Dynamic settings were introduced for CTA (telescope types), but the version in Eventdisplay_v4 had a minor bug
more flexibility to find correct NSB bins when combing lookup tables; add command line parameters noise tolerance in combineEffectiveAreas
Remove obsolete TableEnergy filling

Addition of cluster cleaning algorithms

use EVNDISP.reconstruction.runparameter.CC.v4x
not part of the standard analysis (but feel free to test)

Automatic container generation

integration of some automatic docker container generation with each release (see dockerfiles directory
newest images are available for each release from here: https://github.com/VERITAS-Observatory/EventDisplayv4/pkgs/container/eventdisplayv4
(this needs a bit further testing)

Code maintenance

improved code documentation
use consistently now a code linter
- requires installation of the unix tool astyle. Run make formatSourceCode before submitting any code
removal of lots of obsolete and not maintained code
- CTA related code (use this Eventdisplay for CTA Analysis)
- frogs and Model3d (code very much out of date)
cleanup of compilation warnings and C++17
- General cleanup of compilation warnings and changes to allow for C++ 14 and C++ 17compilation.
- test this by CI matrix on Github (which consist of testing compilation and linking on root:6.24.06-centos7, root:6.22.08-ubuntu20.04, root:6.26.06-ubuntu22.04, root:6.26.06-fedora35, root:latest)

Image/border pixel writer

writing of image/border pixel lists on evndisp stage (optional, very useful debugging of cleaning studies)
- Option -writeimagepixellist to evndisp allows to write out list of pixel (with integrated charge and timing) surviving the image cleaning.
- This option increases the output files of evndisp by approximately a factor 2.5-3. It is therefore switched off by default.
- Written are image, border pixels, and one additional ring of pixels around them: PixelListN : PixelListN/i PixelID : PixelID[PixelListN]/i PixelType : PixelType[PixelListN]/i PixelIntensity : PixelIntensity[PixelListN]/F PixelTimingT0 : PixelTimingT0[PixelListN]/F PixelType indicates the following:
PixelType == 1: image pixel
PixelType == 2: border pixel
PixelType == 3: neighbour pixel to image/border

DB text functionality

Use DB text files instead of direct query to VERITAS data base
- allows to download information from VERITAS DB (e.g.,run data, VPM data, etc) and re-use it
- avoids overloading the database connection when analysing hundreds of runs simultaneously
- allows to use a basic root docker image for compilation and no need of integration of mysql while compiling Eventdisplay
- requires reading of DB information using the scripts available in the dbscripts directory of EventdisplayAnalysisScripts_VTS
- used automatically by ANALYSIS.evndisp.sh when a run directory is found in ${VERITAS_DATA_DIR}/DBTEXT

Sofa replacement of slalib

Replacement of slalib routines by sofa for all coordinate transformations
- see PR #180
- Eventdisplay has been using a copy of the slalib code since the very beginning (VASlalib). All astronometry routines are replaced by the sofa package (same authors as slalib; but permissive licence) for the following reasons:
  - slalib (as used in Eventdisplay) is not maintained since many years and the status of the routines is unclear (e.g. means that timing routines do not include leap seconds introduced recently)
  - slalib's licence unclear and very likely the current usage Eventdisplay is not allowed
  - sofa is a well maintained IAU package
- #PR 180 lists some tests listing differences between using the two different astronometry packages: generally consistent results with max differences in reconstructions observations of <2 arcseconds.
- Note that this involves the installation of the sofa package before compiling eventdisplay. Use the script install_sofa.sh to download and install the package.

mergeVBF

addition of mergeVBF binary
- mergeVBF is used for the simulation production to merge several (possibly 1000s) of simulation vbf files into one file.

- C++
Published by GernotMaier about 3 years ago

eventdisplay_v4 - Eventdisplay - version v490-alpha.v15

This update includes significant changes which break the analysis of older versions at certain stages.

Running this version requires updates to the newest versions of EventdisplayAnalysisScriptsVTS and EventdisplayAnalysisFilesVTS (private repository).

This is a pre-release with substantial changes expected to be implemented in near future. Please try it out, but don't use it for science publications.

Changes with relation to v490-alpha.14 are:

significant improvement to efficiency of trainTMVAforGammaHadronSeparation with a large reduction of the memory footprint (which was beyond 20 GB) and improvements in execution speed
main change is that a temporary root with events after signal/background pre-cuts is written to disk and used in a second step for the training (TMVA unfortunately reads in all events before cuts)
a random fraction of events is selected from each input file to match the number of training events requested
if there are less events, the training option string is adjusted and the training proceeds with a lower number of events than requested (!!)

- C++
Published by GernotMaier about 3 years ago

eventdisplay_v4 - Eventdisplay - version v490-alpha.v14

This update includes significant changes which break the analysis of older versions at certain stages.

Running this version requires updates to the newest versions of EventdisplayAnalysisScriptsVTS and EventdisplayAnalysisFilesVTS (private repository).

This is a pre-release with substantial changes expected to be implemented in near future. Please try it out, but don't use it for science publications.

Changes with relation to v490-alpha.13 are:

v490 - alpha.14 - fix include types by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/222
v490 - alpha.14 - TMVA training for gamma/hadron separation by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/220

Full Changelog: https://github.com/VERITAS-Observatory/EventDisplay_v4/compare/v490-alpha.13...v490-alpha.14

- C++
Published by GernotMaier about 3 years ago

eventdisplay_v4 - Eventdisplay - version v490-alpha.v13

This update includes significant changes which break the analysis of older versions at certain stages.

Running this version requires updates to the newest versions of EventdisplayAnalysisScriptsVTS and EventdisplayAnalysisFilesVTS (private repository).

This is a pre-release with substantial changes expected to be implemented in near future. Please try it out, but don't use it for science publications.

Changes with relation to v490-alpha.10 are:

v490 - alpha.12 - cleanup - Epoch reading by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/219
v490 - alpha.13 - critical bug fixes in time dependent pedvar69 calculation and dead time calculation by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/221

Full Changelog: https://github.com/VERITAS-Observatory/EventDisplay_v4/compare/v490-alpha.10...v490-alpha.13

- C++
Published by GernotMaier about 3 years ago

eventdisplay_v4 - Eventdisplay - version v490-alpha.10

This update includes significant changes which break the analysis of older versions at certain stages.

Running this version requires updates to the newest versions of EventdisplayAnalysisScriptsVTS and EventdisplayAnalysisFilesVTS (private repository).

This is a pre-release with substantial changes expected to be implemented in near future. Please try it out, but don't use it for science publications.

Changes with relation to v490-alpha.9 are:

integration and improvements of dispBDT direction reconstruction algorithms.

What's Changed

MC / Data comparision improvements by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/217
Add dispEnergy reconstruction by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/218
v490 - alpha.0.10 - dispBDT improvements adding dispSign by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/216

Full Changelog: https://github.com/VERITAS-Observatory/EventDisplay_v4/compare/v490-alpha.9...v490-alpha.10

- C++
Published by GernotMaier about 3 years ago

eventdisplay_v4 - Eventdisplay - version v490-alpha.9

This update includes significant changes which break the analysis of older versions at certain stages.

Running this version requires also updates to the newest versions of EventdisplayAnalysisScriptsVTS and EventdisplayAnalysisFilesVTS (private repository).

This is a pre-release with substantial changes expected in near future. Please try it out, but don't use it for science publications.

Changes with relation to v490-alpha.8 are:

added pedestal median and 68% containment calculation for pedestal variation. This is more outlier robust, especially to the increase contributions of after pulses to the background rate. Added also a plotting function (macros/VTS/plot_pedestals.C) for cross checks
removed printout of long debug statements in evndisp stage for MC analysis
improved performance of pixel parameter calculation in calcTZeroSums used for NN cleaning in case of identical first and second summation window (which is generally the case). Notable improvement in performance for NN cleaning.
minor code cleanup in trace analyzer
improved documentation for dispBDT training code

Full Changelog: https://github.com/VERITAS-Observatory/EventDisplay_v4/compare/v490-alpha.8.1...v490-alpha.9

- C++
Published by GernotMaier about 3 years ago

eventdisplay_v4 - Eventdisplay - version v490-alpha.8.1

Eventdisplay v490 is a major update of the reconstruction and analysis code used for VERITAS. This update includes significant changes which break the analysis of older versions at certain stages.

Running this version requires also updates to the newest versions of EventdisplayAnalysisScriptsVTS and EventdisplayAnalysisFilesVTS (private repository).

This is a pre-release with substantial changes expected in near future. Please try it out, but don't use it for science publications.

This is release is testing packaging.

- C++
Published by GernotMaier over 3 years ago

eventdisplay_v4 - Eventdisplay - version v490-alpha.8

Eventdisplay v490 is a major update of the reconstruction and analysis code used for VERITAS. This update includes significant changes which break the analysis of older versions at certain stages.

Running this version requires also updates to the newest versions of EventdisplayAnalysisScriptsVTS and EventdisplayAnalysisFilesVTS (private repository).

This is a pre-release with substantial changes expected in near future. Please try it out, but don't use it for science publications. Changes with relation to v490-alpha.7 are:

Combine lookup tables

More flexibility to find correct NSB bins when combing lookup tables; add command line parameters noise tolerance in combineEffectiveAreas.

Propagation of cluster cleaning from v5

Propagated cluster cleaning from v5 to v490.

removes islands with N pixels or with size smaller than a certain value
settings see EVNDISP.reconstruction.runparameter.CC.v4x

Addition of AP cleaning

Addition of removal of islands consisting of a single image and a single border pixel, triggered by Jamie's study on increased afterpulsing contributions.

switch this on by adding one parameter 1 to * -1 IMAGECLEANINGTHRESHOLDS 5. 2.5 1
settings see EVNDISP.reconstruction.runparameter.AP.v4x

Improvement of optimized next-neighbour cleaning

Improved handling of short runs when there are not enough entries in the pedestal histograms to determine correct cleaning cuts. In this case, a single average histogram is filled to calculate the cleaning thresholds.

Other changes

improved installation hints
critical bug fix in filling lookup tables, using the correct direction values

What's Changed

Deal with short runs with low number of events required for IPR Grphs by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/209
v490 - alpha.0.8 by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/208

Full Changelog: https://github.com/VERITAS-Observatory/EventDisplay_v4/compare/v490-alpha.7...v490-alpha.8

- C++
Published by GernotMaier over 3 years ago

eventdisplay_v4 - Eventdisplay - version v490-alpha.7

Eventdisplay v490 is a major update of the reconstruction and analysis code used for VERITAS. This update includes significant changes which break the analysis of older versions at certain stages.

Running this version requires also updates to the newest versions of EventdisplayAnalysisScriptsVTS and EventdisplayAnalysisFilesVTS (private repository).

This is a pre-release with substantial changes expected in near future. Please try it out, but don't use it for science publications. Changes with relation to v490-alpha.6 are:

Cleanup of compilation warnings and C++17

General cleanup of compilation warnings and changes to allow for C++ 14 compilation.

VLowGainCalibrator not working on all systems

VLowGainCalibrator depends on TSpectrum, which is depreciated. To allow compilation of slib, this is handled now automatically through the Makefile. Meaning that on some newer systems (e.g., 6.26.06-fedora35) the low-gain calibration scripts are not working (see issue #207)

What's Changed

v490 - alpha.0.7 by @GernotMaier in https://github.com/VERITAS-Observatory/EventDisplay_v4/pull/205

Full Changelog: https://github.com/VERITAS-Observatory/EventDisplay_v4/compare/v490-alpha.6...v490-alpha.7

- C++
Published by GernotMaier over 3 years ago

eventdisplay_v4 - Eventdisplay - version v490-alpha.6

Includes changes fromp v490-alpha.4, v490-alpha.5, v490-alpha.6

Eventdisplay v490 is a major update of the reconstruction and analysis code used for VERITAS. This update includes significant changes which break the analysis of older versions at certain stages.

Running this version requires also updates to the newest versions of EventdisplayAnalysisScriptsVTS and EventdisplayAnalysisFilesVTS (private repository).

This is a pre-release with substantial changes expected in near future. Please try it out, but don't use it for science publications. Changes with relation to v490-alpha.3 are:

Use Pointing status bit

VPM report success of pointing reconstruction. This is used now, events without useful pointing are discarded (generally very few events; often at the very end of a run)

Improvements in plotting of instrument response functions

Re-enable plotting of instrument response functions vs true energy: Effective area files contain since v497 two trees: fEffArea (with reconstructed energy axis) and fEffAreaH2F (with true energy axis). Change reading routines to re-enable plotting of IRFs vs true energy.

Improve angular resolution plotting and fix issue with plotting cumulative theta2 distributions.

Energy axes for energy reconstruction lookup tables

Removed dynamic setting of energy axes limits (1D histogram for tables) in lookup table code and fixed it to 150 TeV (well beyond what we do in VERITAS). Dynamic settings were introduced for CTA (telescope types), but the version in Eventdisplay_v4 had a minor bug.

DISP reconstruction

Improvement how head/tail uncertainty is resolved. Apply minimum angle cut for 2-tel events only.

Remove duplication of date/MJD in season definition

Having both MJD and SQL-strings lead to errors in definitions of yearly seasons. Removed the MJD, which is now calculated in VEvndispRunParameter from the sql string.

Remove obsolete TableEnergy filling

Remove filling of unused tables with energy / R axes

Unweighted simulated histograms

Add unweighted histograms to effective area output files (mostly for debugging now; later useful for DL2).

Command line parameters to overwrite target runinfo DB values

Add possibility to overwrite the DB settings for target information (read usually from the runinfo db table): -overwritedb_rightascension, -overwritedb_declination, -overwritedb_target (this complements the existing command line options for the wobble offset: overwritedb_wobblenorth and overwritedb_wobbleeast

- C++
Published by GernotMaier over 3 years ago

eventdisplay_v4 - Eventdisplay - version v487e

v487d patch release of Eventdisplay

Very minor patch release

printing of Non/Noff events to ecsv files

Wiki page: https://veritas.sao.arizona.edu/wiki/Eventdisplay_v487

- C++
Published by GernotMaier over 3 years ago

eventdisplay_v4 - Eventdisplay - version v490-alpha.3

Eventdisplay v490 is a major update of the reconstruction and analysis code used for VERITAS. This update includes significant changes which break the analysis of older versions at certain stages.

Running this version requires also updates to the newest versions of EventdisplayAnalysisScriptsVTS and EventdisplayAnalysisFilesVTS (private repository).

This is a pre-release with substantial changes expected in near future. Please try it out, but don't use it for science publications. Changes with relation to v490-alpha.1 are:

dispBDT angular resolution

add correct addition of VPM pointing monitor correction to dispBDT direction reconstruction

Writing of image/border pixel lists on evndisp stage

Option -writeimagepixellist to evndisp allows to write out list of pixel (with integrated charge and timing) surviving the image cleaning.

This option increases the output files of evndisp by approximately a factor 2.5-3. It is therefore switched off by default.

Written are image, border pixels, and one additional ring of pixels around them.

Written the pars trees: PixelListN : PixelListN/i PixelID : PixelID[PixelListN]/i PixelType : PixelType[PixelListN]/i PixelIntensity : PixelIntensity[PixelListN]/F PixelTimingT0 : PixelTimingT0[PixelListN]/F

PixelType indicates the following: + PixelType == 1: image pixel + PixelType == 2: border pixel + PixelType == 3: neighbour pixel to image/border

Use DB text files instead of direct query to VERITAS data base

New functionality to read DB text files. This allows faster parallel processing of many runs without opening up too many connections to the VERITAS database. Requires reading of DB information using the scripts available in the db_scripts directory of Eventdisplay_AnalysisScripts_VTS. Possibly also a step to remove the ROOT-mysql dependency and allow for easier compilation.

Others

DOI badge

- C++
Published by GernotMaier over 3 years ago

eventdisplay_v4 - Eventdisplay - version v490-alpha.2

(consistency release - no major updates to v490-alpha.1)

- C++
Published by GernotMaier over 3 years ago

eventdisplay_v4 - Eventdisplay - version v490-alpha.1

Eventdisplay v490 is a major update of the reconstruction and analysis code used for VERITAS. This update includes significant changes which break the analysis of older versions at certain stages.

Running this version requires also updates to the newest versions of EventdisplayAnalysisScriptsVTS and EventdisplayAnalysisFilesVTS (private repository).

This is a pre-release with substantial changes expected in near future. Please try it out, but don't use it for science publications.

Eventdisplay v490 is now public

Similar to the Eventdisplay version used for CTA (see here), this code is now public.

Optimised next-neighbour cleaning

(back ported from v5)

Improves the low-energy response. - requires to run Evetndisplay with the runparameter file EVNDISP.reconstruction.runparameter.NN.v4x

dispBDT angular resolution

(back ported from v5)

Improves direction reconstruction at low elevation angles (issue #184 and PR #190).

Note that dispBDT direction reconstruction is applied on the mscw_energy stage.

dispBDT files are located in $VERITAS_EVNDISP_AUX_DIR/DispBDTs
in alpha release: must be switched on in IRF.mscw_energy_MC.sh and ANALYSIS.mscw_energy.sh (search for DISPBDT)

Replacement of slalib routines by sofa

see PR #180

Eventdisplay has been using a copy of the slalib code since the very beginning (VASlalib). All astronometry routines are replaced by the sofa package (same authors as slalib; but permissive licence) for the following reasons:

slalib (as used in Eventdisplay) is not maintained since many years and the status of the routines is unclear (e.g. means that timing routines do not include leap seconds introduced recently)
slalib's licence unclear and very likely the current usage Eventdisplay is not allowed
sofa is a well maintained IAU package

PR 180 lists some tests listing differences between using the two different astronometry packages: generally consistent results with max differences in reconstructions observations of <2 arcseconds.

Note that this involves the installation of the sofa package before compiling eventdisplay. Use the script install_sofa.sh to download and install the package.

Addition of mergeVBF binaries

mergeVBF is used for the simulation production to merge several (possibly 1000s) of simulation vbf files into one file.

Improved MC/Data comparison plots

Improvements to MC/Data comparison includes: - weighting of events according to the azimuth of the data runs - energy-dependent theta2 comparision

Code cleanup

Eventdisplay-code base includes a lot of obsolete code which makes maintenance more difficult than necessary. Try and remove anything which will not be used in future: CTA-related code (uses v5), frogs and 3D model (will realistically never by used and would require a re-implementation); other smaller bits.

Code formatting (astyle) has been applied. For git diffs, best to remove white space comparision.

- C++
Published by GernotMaier over 3 years ago

eventdisplay_v4 - Eventdisplay - version v487d

v487d patch release of Eventdisplay

Very minor patch release

bug fix in reading IRFs as function of true energy (relevant for using likelihood method, see issue #164)
re-enable writing of data_on and data_off trees in anasum when user sets this flag in the anasum runparameter file: * WRITEDATAONOFFEVENTS 1. Note that these trees are a subset of the DL3EventTree.
expand environmental variables when writing to cvs in VEnergySpectrum
add further options for the storage of log files in the evndisp root files (./bin/logFile)

Wiki page: https://veritas.sao.arizona.edu/wiki/Eventdisplay_v487

- C++
Published by GernotMaier over 3 years ago

eventdisplay_v4 - Eventdisplay - version v487c

v487c patch release of Eventdisplay

Very minor patch release

improved printout in combineEffectiveArea help
additional variables written to pointing tree for DB pointing (useful for debugging)
removal of duplicated entries from Makefile targets.

Release wiki page including the list of major changes: https://veritas.sao.arizona.edu/wiki/Eventdisplay_v487

- C++
Published by GernotMaier almost 4 years ago

eventdisplay_v4 - Eventdisplay - version v487b

v487b patch release of Eventdisplay

Addresses critical issue as described in #170

Release wiki page including the list of major changes: https://veritas.sao.arizona.edu/wiki/Eventdisplay_v487

- C++
Published by GernotMaier about 4 years ago

eventdisplay_v4 - Eventdisplay - version v486d

Patch release v486d of Eventdisplay

Release wiki page: https://veritas.sao.arizona.edu/wiki/Eventdisplay_v486

Changes:

fix of light curve data plotting for run numbers > 100000

- C++
Published by GernotMaier about 4 years ago

eventdisplay_v4 - Eventdisplay - version v487

v487 release of Eventdisplay

Release wiki page including the list of major changes: https://veritas.sao.arizona.edu/wiki/Eventdisplay_v487

- C++
Published by GernotMaier about 4 years ago

eventdisplay_v4 - Eventdisplay - version v486c

Patch release v486c of Eventdisplay

Release wiki page: https://veritas.sao.arizona.edu/wiki/Eventdisplay_v486

Changes: * allow for the analysis of non-standard wobble directions (issue #166)

- C++
Published by GernotMaier about 4 years ago

eventdisplay_v4 - Eventdisplay - version v486b

Patch release v486b of Eventdisplay

Release wiki page: https://veritas.sao.arizona.edu/wiki/Eventdisplay_v486

Changes:

bug fix in mscw_energy compiled with ROOT6 (closing of files took ~20 min; see issue #146)
changes to combineLookupTables and combineEffectiveAreas for reduced file-size IRFs (when running anasum analysis and no gammapy analysis; issue #148 and #145)

- C++
Published by GernotMaier about 4 years ago

eventdisplay_v4 - Eventdisplay - version v486a

v486a release of Eventdisplay

Release wiki page including the list of major changes: https://veritas.sao.arizona.edu/wiki/Eventdisplay_v486

- C++
Published by GernotMaier about 4 years ago

eventdisplay_v4 - Eventdisplay - version v485c

Patch release of Eventdisplay version v485c

Release wiki page: https://veritas.sao.arizona.edu/wiki/Eventdisplay_v485

Main changes:

makefile fix to allow compiling with newer gcc versions
bug fix applied in writing out mean energy bias graph in anasum stage
addition of mean_pedvars to DL3 tree

- C++
Published by GernotMaier over 4 years ago

eventdisplay_v4 - Eventdisplay - version v485b

Patch release of Eventdisplay version v485b

Release wiki page: https://veritas.sao.arizona.edu/wiki/Eventdisplay_v485

Main changes:

bug fix to allow analysis of runs with run numbers >= 10000
added small tool to print anasum run parameters (printAnasumRunParameter)

- C++
Published by GernotMaier over 4 years ago

eventdisplay_v4 - Eventdisplay - version v485a

v485 release of Eventdisplay.

Release wiki page: https://veritas.sao.arizona.edu/wiki/Eventdisplay_v485

Main updates:

separation of script directories into separate repository (easier maintenance; see https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/tree/v485)
binned likelihood ratio re-enabled (thanks Ste!) - see wiki page for how to use it
simplifications of event list writing in anasum (preparation for DL3 output and gammapy analysis); Note that tree names changed and some trees have been removed (as there was a lot of duplication)

event list tree is now called fDL3EventTree
removal of trees, which were all doing kind of the same (implemented by different authors): fTreeWithEventsForCtools, fTreeWithAllGamma, fCDataTreeClone
event lists are written by default to the anasum output file, when running anasum over individual runs

Improved plotting of fluxes
Bug fixes leading to seg faults

- C++
Published by GernotMaier almost 5 years ago

eventdisplay_v4 - Eventdisplay version v483c

v483c release of Eventdisplay.

This is a page release solving an issue in anasum for filling on-off (diff) histograms)

See release page for details:

https://veritas.sao.arizona.edu/wiki/index.php/Eventdisplay_v483

- C++
Published by GernotMaier almost 5 years ago

eventdisplay_v4 - Eventdisplay version v483b

v483b release of Eventdisplay.

See release page for details:

https://veritas.sao.arizona.edu/wiki/index.php/Eventdisplay_v483

- C++
Published by GernotMaier about 5 years ago

eventdisplay_v4 - Eventdisplay version v483

v483 release of Eventdisplay.

See release page for details:

https://veritas.sao.arizona.edu/wiki/index.php/Eventdisplay_v483

- C++
Published by GernotMaier over 5 years ago

Recent Releases of eventdisplay_v4

eventdisplay_v4 - Eventdisplay - version v490.7.2

eventdisplay_v4 - Eventdisplay - version v490.7.1

What's Changed

eventdisplay_v4 - Eventdisplay - release v491.0

Main Improvements

What's Changed

eventdisplay_v4 - Eventdisplay - beta release v491-rc2

Improvements

What's Changed

eventdisplay_v4 - Eventdisplay - beta release v491-beta.4

Improvements

What's Changed

eventdisplay_v4 - Eventdisplay - beta release v491-beta.3

Improvements

What's Changed

eventdisplay_v4 - Eventdisplay - beta release v491-beta.2

Improvements

What's Changed

eventdisplay_v4 - Eventdisplay - beta release v491-beta.1

Improvements:

Bug fixes:

What's Changed

eventdisplay_v4 - Eventdisplay - version v490.8

What's Changed

eventdisplay_v4 - Eventdisplay - version v490.7

What's Changed

eventdisplay_v4 - Eventdisplay - version v490.6

What's Changed

eventdisplay_v4 - Eventdisplay - version v490.5

What's Changed

eventdisplay_v4 - Eventdisplay - version v490.4

What's Changed

eventdisplay_v4 - Eventdisplay - version v490.3

What's Changed

eventdisplay_v4 - Eventdisplay - version v490.2

What's Changed

eventdisplay_v4 - Eventdisplay - version v490.1

eventdisplay_v4 - Eventdisplay - version v490.0

Main changes.

Detailed release notes:

Afterpulsing cleaning (image/border cleaning with improved robustness)

Optimised next-neighbour cleaning for improved low-energy reconstruction

Epoch and throughput definitions

BDT gamma/hadron separation

Overwrite of runinfo DB values

evndisp robustness against missing events

ncreased robustness for reading number of samples of readout window from VBF files. This recovers those runs, where for some reason the more than 10,000 events in the beginning of the run are empty, but the rest of the run is ok.

New dispBDT direction reconstruction algorithms for improved direction reconstruction at low zenith angles and large offsets

Support for DL3 export using V2DL3 and gammapy

Improved MC / Data comparison with additional plots for theta2 and disp

Evndisp trace integration

Lookup tables

Addition of cluster cleaning algorithms

Automatic container generation

Code maintenance

Image/border pixel writer

DB text functionality

Sofa replacement of slalib

mergeVBF

Readout window

eventdisplay_v4 - Eventdisplay - version v490-beta.2

Main changes.

Detailed release notes:

Afterpulsing cleaning (image/border cleaning with improved robustness)

Optimised next-neighbour cleaning for improved low-energy reconstruction

Epoch and throughput definitions

BDT gamma/hadron separation

Overwrite of runinfo DB values

New dispBDT direction reconstruction algorithms for improved direction reconstruction at low zenith angles and large offsets

Support for DL3 export using V2DL3 and gammapy

Improved MC / Data comparison with additional plots for theta2 and disp

Evndisp trace integration

Lookup tables

Addition of cluster cleaning algorithms

Automatic container generation

Code maintenance

Image/border pixel writer

DB text functionality

Sofa replacement of slalib