Recent Releases of eventdisplay_analysisscripts_vts
eventdisplay_analysisscripts_vts - Eventdisplay Analysis Scripts for VERITAS - version v490.7.1
Eventdisplay is a reconstruction and analysis pipeline for data of the VERITAS Imaging Atmospheric Cherenkov Telescopes (IACT). Eventdisplay analysis scripts for VERITAS are developed for efficient analysis of many data runs and production of instrument response functions using EventDisplay_v4 for the reconstruction and analysis.
Version v490.7.1 is minor update of the long-term supported version v490.7. Note that analysis scripts are identical for v490.7, v491, and v492 release.
What's Changed
- v491-dev 4: fix hard 2tel cut optimisation by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/107
- v491.1-dev 1 by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/110
- Add (lost) reading from VERITASDATADIR by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/94
- v490.8 dev2 by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/112
- LTS support version v490.7.1. by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/113
- v490.7.1 - allow to run v490.7 from newer releases. by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/115
- v490.7.1 - Long-term support version by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/114
- V491.1 dev1 by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/116
- v492.0 dev1 - 3-tel reconstruction and efficiency improvements. by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/111
Full Changelog: https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/compare/v491.0...v490.7.1
- Shell
Published by GernotMaier about 1 year ago
eventdisplay_analysisscripts_vts - Eventdisplay Analysis Scripts for VERITAS - version v491.0
Eventdisplay is a reconstruction and analysis pipeline for data of the VERITAS Imaging Atmospheric Cherenkov Telescopes (IACT). Eventdisplay analysis scripts for VERITAS are developed for efficient analysis of many data runs and production of instrument response functions using EventDisplay_v4 for the reconstruction and analysis.
Release v491.0 include significant updates and efficiency improvements for the generation of instrument response functions and pre-processing of data files.
What's Changed
- v491.0 dev1 by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/100
- v491.0 dev1 by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/104
- v491.0 dev2 - fixes for IRF production. by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/105
- v491.0 dev3 - script efficiency by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/106
- Minor updates for v491 by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/108
- v491.0 dev6 - improved pre-processing scripts by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/109
Full Changelog: https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/compare/v490.8...v491.0
- Shell
Published by GernotMaier about 1 year ago
eventdisplay_analysisscripts_vts - Eventdisplay Analysis Scripts for VERITAS - version v490.8
Eventdisplay is a reconstruction and analysis pipeline for data of the VERITAS Imaging Atmospheric Cherenkov Telescopes (IACT). Eventdisplay analysis scripts for VERITAS are developed for efficient analysis of many data runs and production of instrument response functions using EventDisplay_v4 for the reconstruction and analysis.
This release adds improvements related to archival processing, especially the introduction of the usage of apptainers for the data analysis.
The usage of Eventdisplay requires code, scripts, and analysis files. For this release, the corresponding repositories are:
What's Changed
- improved v2dl3 submission settings by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/89
- v490.9-dev 1 - script improvements. by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/90
- Integrate Apptainers for IRF production; typos by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/91
- v490.9 dev2 - add apptainer support for part of the IRF production by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/93
- v490.9 dev4 - add IRF processing with apptainers by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/95
- v490.9 dev4 - new redHV MC set by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/96
- v490.9 dev5 - script submission improvements by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/97
- v490.9 dev5 - Minor additions and removal of SL7 related code by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/98
- v490.7 dev1 - minor typos and queue configuration updates. Script improvements. by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/99
- v490.8 - minor readme updates. by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/101
Full Changelog: https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/compare/v490.7...v490.8
- Shell
Published by GernotMaier over 1 year ago
eventdisplay_analysisscripts_vts - Eventdisplay Analysis Scripts for VERITAS - version v490.7
Eventdisplay is a reconstruction and analysis pipeline for data of the VERITAS Imaging Atmospheric Cherenkov Telescopes (IACT). Eventdisplay analysis scripts for VERITAS developed for efficient analysis of many data runs and production of instrument response functions using EventDisplay_v4 for the reconstruction and analysis.
This release adds improvements related to archival processing, especially the introduction of the usage of apptainers for the data analysis.
The usage of Eventdisplay requires code, scripts, and analysis files. For this release, the corresponding repositories are:
What's Changed
- V490.6 dev3 by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/78
- v490.7 preprocessing dev1 - efficiency improvments / apptainers by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/79
- Readme updates for DB text directory by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/82
- v490.7 Add hard2tel options. by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/81
- v490.7-dev 2 pre-processing updates. by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/83
- v490.7 preprocessing dev2 - minor script improvements. by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/84
- v490.7 preprocessing dev3 v2dl3 processing by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/86
- v490.7-dev4 minor bugfixes by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/87
- add archive v490.7 by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/88
Full Changelog: https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/compare/v490.6...v490.7
- Shell
Published by GernotMaier almost 2 years ago
eventdisplay_analysisscripts_vts - Eventdisplay Analysis Scripts for VERITAS - version v490.6
Eventdisplay is a reconstruction and analysis pipeline for data of the VERITAS Imaging Atmospheric Cherenkov Telescopes (IACT). Eventdisplay analysis scripts for VERITAS developed for efficient analysis of many data runs and production of instrument response functions using EventDisplay_v4 for the reconstruction and analysis.
This is a minor release with bug fixes and improvements related to archival processing.
The usage of Eventdisplay requires code, scripts, and analysis files. For this release, the corresponding repositories are:
What's Changed
- v490.6 dev1 - improved mscw and angular reconstruction scripts by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/76
- V490.6 dev2 camera rotation readout by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/77
Full Changelog: https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/compare/v490.5...v490.6
- Shell
Published by GernotMaier about 2 years ago
eventdisplay_analysisscripts_vts - Eventdisplay Analysis Scripts for VERITAS - version v490.5
Eventdisplay is a reconstruction and analysis pipeline for data of the VERITAS Imaging Atmospheric Cherenkov Telescopes (IACT). Eventdisplay analysis scripts for VERITAS developed for efficient analysis of many data runs and production of instrument response functions using EventDisplay_v4 for the reconstruction and analysis.
This is a minor release with bug fixes and improvements related to archival processing.
The usage of Eventdisplay requires code, scripts, and analysis files. For this release, the corresponding repositories are:
- Eventdisplay v490.5
- EventdisplayAnalysisScriptsVTS v490.5
- EventdisplayAnalysisFilesVTS v490.5 (private repository)
What's Changed
- v490.5 dev1 by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/75
Full Changelog: https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/compare/v490.4...v490.5
- Shell
Published by GernotMaier about 2 years ago
eventdisplay_analysisscripts_vts - Eventdisplay Analysis Scripts for VERITAS - version v490.4
Eventdisplay is a reconstruction and analysis pipeline for data of the VERITAS Imaging Atmospheric Cherenkov Telescopes (IACT). Eventdisplay analysis scripts for VERITAS developed for efficient analysis of many data runs and production of instrument response functions using EventDisplay_v4 for the reconstruction and analysis.
This is a minor release with bug fixes and improvements related to archival processing.
The usage of Eventdisplay requires code, scripts, and analysis files. For this release, the corresponding repositories are:
- Eventdisplay v490.4
- EventdisplayAnalysisScriptsVTS v490.4
- EventdisplayAnalysisFilesVTS v490.3 (private repository; note that v490.3 is the correct version for the analysis files)
For details on how to install the release, see this readme.
What's Changed
- v490.4 dev1 - improved docs and printouts by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/73
Full Changelog: https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/compare/v490.3...v490.4
- Shell
Published by GernotMaier over 2 years ago
eventdisplay_analysisscripts_vts - Eventdisplay Analysis Scripts for VERITAS - version v490.3
Eventdisplay is a reconstruction and analysis pipeline for data of the VERITAS Imaging Atmospheric Cherenkov Telescopes (IACT). Eventdisplay analysis scripts for VERITAS developed for efficient analysis of many data runs and production of instrument response functions using EventDisplay_v4 for the reconstruction and analysis.
This is a minor release with bug fixes and improvements related to archival processing.
The usage of Eventdisplay requires code, scripts, and analysis files. For this release, the corresponding repositories are: - Eventdisplay v490.3 - EventdisplayAnalysisScriptsVTS v490.3 - EventdisplayAnalysisFilesVTS v490.3 (private repository)
For details on how to install the release, see this readme.
What's Changed
- v490.3 - dev1 bug fixes; by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/67
- v490.3 dev2 - anasum and v2dl3 preprocessing by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/68
- v490.3 dev4 by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/69
- typos and formatting by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/70
- Adds a small script to write DB tables into a fits file; summary of DQM for DL3 by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/71
- v490 - Improved anasum processing with pre-processed files by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/72
Full Changelog: https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/compare/v490.2...v490.3
- Shell
Published by GernotMaier over 2 years ago
eventdisplay_analysisscripts_vts - Eventdisplay Analysis Scripts for VERITAS - version v490.2
Eventdisplay is a reconstruction and analysis pipeline for data of the VERITAS Imaging Atmospheric Cherenkov Telescopes (IACT). Eventdisplay analysis scripts for VERITAS developed for efficient analysis of many data runs and production of instrument response functions using EventDisplay_v4 for the reconstruction and analysis.
This is a minor release with bug fixes and improvements related to archival processing of a large number of observation runs and processing of runs using the optimised next-neighbour cleaning.
Changes: - now download script to get pre-processed data from DESY: ARCHIVE.preparedownloadfrom_DESY.sh - updated directory structure for processing to take AP and NN cleaning into account - add supersoft cuts into scripts used in NN cleaning - improved scripts for preprocessing and db access - removal of run-wise radial acceptance code in anasum scripts (should not be used) - scripts/ANALYSIS.mscw_energy.sh with changed command line sequence (!!) - first mscw output directory and then optional input directory (easier for typing when having the evndisp files in a default place) - add scripts to handle very old V4 laser files read from a scripts/dbscripts/laserruns
The usage of Eventdisplay requires code, scripts, and analysis files. For this release, the corresponding repositories are: - Eventdisplay v490.2 - EventdisplayAnalysisScriptsVTS v490.2 - EventdisplayAnalysisFilesVTS v490.2 (private repository)
For details on how to install the release, see this readme.
What's Changed
- v490.2 dev1 - NN cleaning changes by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/63
- ancient laser runs by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/64
Full Changelog: https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/compare/v490.1...v490.2
- Shell
Published by GernotMaier over 2 years ago
eventdisplay_analysisscripts_vts - Eventdisplay Analysis Scripts for VERITAS - version v490.1
Eventdisplay is a reconstruction and analysis pipeline for data of the VERITAS Imaging Atmospheric Cherenkov Telescopes (IACT). Eventdisplay analysis scripts for VERITAS developed for efficient analysis of many data runs and production of instrument response functions using EventDisplay_v4 for the reconstruction and analysis.
This is a minor release with bug fixes and improvements related to archival processing of a large number of observation runs and V4/V5 instrument response function generation.
Changes:
- improved efficiency of laser analysis script scripts/ANALYSIS.evndisp_laser.sh
- much improved job submission efficiency for scripts/ANALYSIS.mscw_energy.sh
- adaption of radial acceptance scripts to new directory structure
- scripts/RUNLIST.whichRunsAreOnDisk.sh prints now also the directory of the VBF run to the screen (option
-p) - scripts to handle db_text entries for very old V4 laser runs (without correct entries in the DB)
- scripts for preprocessing (scripts/preprocessing/ directory)
The usage of Eventdisplay requires code, scripts, and analysis files. For this release, the corresponding repositories are: - Eventdisplay v490.1 - EventdisplayAnalysisScriptsVTS v490.1 - EventdisplayAnalysisFilesVTS v490.1a (private repository)
- Shell
Published by GernotMaier over 2 years ago
eventdisplay_analysisscripts_vts - Eventdisplay Analysis Scripts for VERITAS - version v490.0
Eventdisplay analysis scripts for VERITAS developed for efficient analysis of many data runs and production of instrument response functions using EventDisplay_v4 for the reconstruction and analysis.
The usage of Eventdisplay requires code, scripts, and analysis files. For this release, the corresponding repositories are: - Eventdisplay v490.0 - EventdisplayAnalysisScriptsVTS v490.0 - EventdisplayAnalysisFilesVTS v490.1
Changes
ANALYSIS.anasum_combine.sh
ANALYSIS.anasum_combine.sh is not submitted jobs to the queue and not running locally anymore (unless you run all your analysis locally). Note that the script does not write out a complete run list but uses a simple run list now (simple = list of runs).
AP and NN Analysis support
Script support different analysis types, including the afterpulsing and optimised next-neighbour cleaning.
Best to set a environmental variable $VERITAS_ANALYSIS_TYPE to either AP (afterpulsing threshold cleaning) or NN (optimised next neighbour cleaning).
Scripts should pick up the correct settings (e.g., IRF.production.sh will set the eventdisplay reconstruction parameter file to EVNDISP.reconstruction.runparameter.NN.v4x for NN cleaning and EVNDISP.reconstruction.runparameter.AP.v4x for AP cleaning)
dispBDT Angular Reconstruction support
Scripts to train and use dispBDT regression trees for the angular reconstruction.
Training scripts are IRF.trainTMVAforAngularReconstruction.sh and reconstruction scripts are IRF.mscw_energy_MC.sh and IRF.generate_mscw_effective_area_parts.sh. Note that at this point dispBDT reconstruction is switched on by changing a flag in the IRF.production.sh script.
HT Condor support
Introduce HTCondor support (as this has been introduced at DESY in 2022).
Only change necessary to use the the HTCondor system is to set * condor_submit in the submissionCommands.dat file.
(and just not to forget it, queue status is checked with condor_status, job status with condor_q at DESY)
Slurm support
Addition of slurm as possible job submission system.
BDT gamma/hadron separation
Scripts for more efficient gamma/hadron separation include scripts to select background runs and optimise cuts.
Support for zstd-compressed evndisp MC files
Support zstd compressed MC evndisp root files.
This involves a slightly different directory structure for IRF production, where MC evndisp files are now located in $VERITAS_IRFPRODUCTION_DIR/v4N (all other IRF data products are stored in a similar directory structure as before)
Scripts to extract DB text files and use them in the evndisp analysis
Scripts to extract DB information are in the dbscripts directory. Users generally need only the dbrun.sh script. Allows large-scale parallel process without DB overload.
Large-scale processing
Optimise directory structure to avoid having files from all runs of VERITAS in a single directory. Runs are sorted according to the first digit, e.g. run 64080 into ./6/ and run 104500 into ./10/.
Significant improved submission efficiency and reduced footprint in terms of number of files.
Log files
To reduce the number of files on disk, log files are written into the corresponding root files.
e..g. the log file from the eventdisplay processing can be retrieved by the following command:
$EVNDISPSYS/bin/logFile evndispLog 64080.root
(same for all other stages; includes files used for IRF production)
- Shell
Published by GernotMaier almost 3 years ago
eventdisplay_analysisscripts_vts - Eventdisplay Analysis Scripts for VERITAS - version v490-beta.2
Eventdisplay analysis scripts for VERITAS, see EventDisplay_v4 for the reconstruction and analysis codes.
This release should be used together with - Eventdisplay v490-beta.2 - EventdisplayAnalysisScriptsVTS v490-beta.2 - EventdisplayAnalysisFilesVTS v490-beta.2.1
Changes (includes beta.1 changes):
AP and NN Analysis support
Script support different analysis types, including the afterpulsing and optimised next-neighbour cleaning.
Best to set a environmental variable $VERITAS_ANALYSIS_TYPE to either AP (afterpulsing threshold cleaning) or NN (optimised next neighbour cleaning).
Scripts should pick up the correct settings (e.g., IRF.production.sh will set the eventdisplay reconstruction parameter file to EVNDISP.reconstruction.runparameter.NN.v4x for NN cleaning and EVNDISP.reconstruction.runparameter.AP.v4x for AP cleaning)
dispBDT Angular Reconstruction support
Scripts to train and use dispBDT regression trees for the angular reconstruction.
Training scripts are IRF.trainTMVAforAngularReconstruction.sh and reconstruction scripts are IRF.mscw_energy_MC.sh and IRF.generate_mscw_effective_area_parts.sh. Note that at this point dispBDT reconstruction is switched on by changing a flag in the IRF.production.sh script.
HT Condor support
Introduce HTCondor support (as this is slowly introduced at DESY in 2022).
Only change necessary to use the the HTCondor system is to set * condor_submit in the submissionCommands.dat file.
(and just not to forget it, queue status is checked with condor_status, job status with condor_q at DESY)
Slurm support
Addition of slurm as possible job submission system.
BDT gamma/hadron separation
Scripts for more efficient gamma/hadron separation include scripts to select background runs and optimise cuts.
Support for zstd-compressed evndisp MC files
Support zstd compressed MC evndisp root files.
This involves a slightly different directory structure for IRF production, where MC evndisp files are now located in $VERITAS_IRFPRODUCTION_DIR/v4N (all other IRF data products are stored in a similar directory structure as before)
Scripts to extract DB text files and use them in the evndisp analysis
Scripts to extract DB information are in the dbscripts directory. Users generally need only the dbrun.sh script. Allows large-scale parallel process without DB overload.
Large-scale processing
Optimise directory structure to avoid having files from all runs of VERITAS in a single directory. Runs are sorted accordingi to the first digit, e.g. run 64080 into ./6/ and run 104500 into ./10/.
Log files
To reduce the number of files on disk, log files are written into the corresponding root files.
e..g. the log file from the eventdisplay processing can be retrieved by the following command:
$EVNDISPSYS/bin/logFile evndispLog 64080.root
(same for all other stages; includes files used for IRF production)
- Shell
Published by GernotMaier almost 3 years ago
eventdisplay_analysisscripts_vts - Eventdisplay Analysis Scripts for VERITAS - version v490-beta.1
Eventdisplay analysis scripts for VERITAS, see EventDisplay_v4 for the reconstruction and analysis codes.
Eventdisplay v490 is a major update with significant changes. This release should be used together with - Eventdisplay v490-beta.1 - EventdisplayAnalysisScriptsVTS v490-beta.1 - EventdisplayAnalysisFilesVTS v490-beta.1
Changes:
AP and NN Analysis support
Script support different analysis types, including the afterpulsing and optimised next-neighbour cleaning.
Best to set a environmental variable $VERITAS_ANALYSIS_TYPE to either AP (afterpulsing threshold cleaning) or NN (optimised next neighbour cleaning).
Scripts should pick up the correct settings (e.g., IRF.production.sh will set the eventdisplay reconstruction parameter file to EVNDISP.reconstruction.runparameter.NN.v4x for NN cleaning and EVNDISP.reconstruction.runparameter.AP.v4x for AP cleaning)
dispBDT Angular Reconstruction support
Scripts to train and use dispBDT regression trees for the angular reconstruction.
Training scripts are IRF.trainTMVAforAngularReconstruction.sh and reconstruction scripts are IRF.mscw_energy_MC.sh and IRF.generate_mscw_effective_area_parts.sh. Note that at this point dispBDT reconstruction is switched on by changing a flag in the IRF.production.sh script.
HT Condor support
Introduce HTCondor support (as this is slowly introduced at DESY in 2022).
Only change necessary to use the the HTCondor system is to set * condor_submit in the submissionCommands.dat file.
(and just not to forget it, queue status is checked with condor_status, job status with condor_q at DESY)
Support for zstd-compressed evndisp MC files
Support zstd compressed MC evndisp root files.
This involves a slightly different directory structure for IRF production, where MC evndisp files are now located in $VERITAS_IRFPRODUCTION_DIR/v4N (all other IRF data products are stored in a similar directory structure as before)
Scripts to extract DB text files and use them in the evndisp analysis
Scripts to extract DB information are in the dbscripts directory. Users generally need only the dbrun.sh script. Allows large-scale parallel process without DB overload.
- Shell
Published by GernotMaier about 3 years ago
eventdisplay_analysisscripts_vts - Eventdisplay Analysis Scripts for VERITAS - version v490-alpha.8
Eventdisplay analysis scripts for VERITAS, see EventDisplay_v4 for the reconstruction and analysis codes.
Eventdisplay v490 is a major update with significant changes.
This v490-alpha.8 release adds to the v490-alpha.4:
- allow to run v487 analysis
- compress evndisp files after processing for IRFs
- improved printout for HTCondor usage
- improved DISP analysis usage
What's Changed
- Renaming of V2DL3 script by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/45
- v490-alpha.7 - disp-related additions by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/44
- Add adjustment to allow running of older Eventdisplay version (v487) by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/46
- v490-alpha.8 - add automatic compression for IRFs; improved HTCondor printout by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/47
Full Changelog: https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/compare/v490-alpha.4...v490-alpha.8
- Shell
Published by GernotMaier about 3 years ago
eventdisplay_analysisscripts_vts - Eventdisplay Analysis Scripts for VERITAS - version v490-alpha.4
Eventdisplay analysis scripts for VERITAS, see EventDisplay_v4 for the reconstruction and analysis codes.
Eventdisplay v490 is a major update with significant changes.
This v490-alpha.4 release adds to the v490-alpha.3:
dispBDT
Improved handling of dispBDT training and analysis
UV filter analysis
Added new UV filter analysis
AP cleaning
Add support of AP cleaning and choice of cleaning methods
Condor submission scripts
Improved efficiency of submission scripts and avoidance of overwriting of log/submission files by using UUIDs for directory names.
What's Changed
- v490 - alpha.4 by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/38
- v490-alpha.5 by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/39
- v490-alpha.5 UV analysis: by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/40
- UV filter additions and UUID for analysis by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/43
- V490 dev v0.6 by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/42
Full Changelog: https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/compare/v490-alpha.3...v490-alpha.4
- Shell
Published by GernotMaier about 3 years ago
eventdisplay_analysisscripts_vts - Eventdisplay Analysis Scripts for VERITAS - version v490-alpha.3
Eventdisplay analysis scripts for VERITAS, see EventDisplay_v4 for the reconstruction and analysis codes.
Eventdisplay v490 is a major update with significant changes.
This v490-alpha.3 release adds to the v490-alpha.1 (note: ignore v490-alpha.2):
4. Scripts to extract DB text files and use them in the evndisp analysis
Scripts to extract DB information are in the db_scripts directory. Users generally need only the db_run.sh script. Allows large-scale parallel process without DB overload.
What's Changed
- Add functionality to write DB data from disk by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/36
- V490 dev v0.2 by @GernotMaier in https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/pull/37
Full Changelog: https://github.com/VERITAS-Observatory/EventdisplayAnalysisScriptsVTS/compare/v490-alpha.1...v490-alpha.3
- Shell
Published by GernotMaier over 3 years ago
eventdisplay_analysisscripts_vts - Eventdisplay Analysis Scripts for VERITAS - version v490-alpha.1
Eventdisplay analysis scripts for VERITAS, see EventDisplay_v4 for the reconstruction and analysis codes.
Eventdisplay v490 is a major update with significant changes.
1. NN Analysis support
Script support different analysis types, including the optimised next-neighbour cleaning.
Best to set a environmental variable $VERITAS_ANALYSIS_TYPE to either TS (standard threshold cleaning) or NN (optimised next neighbour cleaning).
Scripts should pick up the correct settings (e.g., IRF.production.sh will set the eventdisplay reconstruction parameter file to EVNDISP.reconstruction.runparameter.NN.v4x for NN cleaning)
2. dispBDT Angular Reconstruction support
Scripts to train and use dispBDT regression trees for the angular reconstruction.
Training scripts are IRF.trainTMVAforAngularReconstruction.sh and reconstruction scripts are IRF.mscw_energy_MC.sh and IRF.generate_mscw_effective_area_parts.sh. Note that at this point dispBDT reconstruction is switched on by changing a flag in the IRF.production.sh script.
3. Condor support
Introduce HTCondor support (as this is slowly introduced at DESY in 2022).
Only change necessary to use the the HTCondor system is to set * condor_submit in the submissionCommands.dat file.
(and just not to forget it, queue status is checked with condor_status, job status with condor_q at DESY)
4. Support for zstd-compressed evndisp MC files
Support zstd compressed MC evndisp root files.
This involves a slightly different directory structure for IRF production, where MC evndisp files are now located in $VERITAS_IRFPRODUCTION_DIR/v4N (all other IRF data products are stored in a similar directory structure as before)
- Shell
Published by GernotMaier over 3 years ago
eventdisplay_analysisscripts_vts - Eventdisplay Analysis Scripts for VERITAS - version v487d
v487d release of Eventdisplay analysis scripts for VERITAS.
Release wiki page including the list of major changes: https://veritas.sao.arizona.edu/wiki/Eventdisplay_v487
Changes:
- add functionality to compress evndisp stage simulation files (including log files)
- allow evndisp files for mscw stages to be zstd compressed.
- bug fix in handling V2DL3 scripts in anasum
- Shell
Published by GernotMaier over 3 years ago
eventdisplay_analysisscripts_vts - Eventdisplay Analysis Scripts for VERITAS - version v487
v487 release of Eventdisplay analysis scripts for VERITAS.
Release wiki page including the list of major changes: https://veritas.sao.arizona.edu/wiki/Eventdisplay_v487
Changes:
- adaption for new command line options required by
combineEffectiveAreas
- Shell
Published by GernotMaier about 4 years ago
eventdisplay_analysisscripts_vts - Eventdisplay Analysis Scripts for VERITAS - version v486b
v486b release of Eventdisplay analysis scripts for VERITAS.
Release wiki page including the list of major changes: https://veritas.sao.arizona.edu/wiki/Eventdisplay_v486
Changes: - allow for the production of reduced-file size IRFs
- Shell
Published by GernotMaier about 4 years ago
eventdisplay_analysisscripts_vts - Eventdisplay Analysis Scripts for VERITAS - version v486a
v486a release of Eventdisplay analysis scripts for VERITAS.
Release wiki page including the list of major changes: https://veritas.sao.arizona.edu/wiki/Eventdisplay_v486
- Shell
Published by GernotMaier about 4 years ago
eventdisplay_analysisscripts_vts - Eventdisplay Analysis Scripts for VERITAS - version v485a
v485 release of Eventdisplay analysis scripts for VERITAS.
This is the first release of a separate repository for the analysis scripts. Main motivations for these steps are mostly to simplify maintenance, while changing as little as possible for the user.
Main changes:
- updated help texts; removed many outdated information
- removal of outdated and obsolete script options; Note: for some scripts, the sequence of command line parameters changed to remove obsolete parameters and to be more consistent among the scripts
- removal of outdated or obsolete scripts.
- Shell
Published by GernotMaier almost 5 years ago