Data

Tools

Indexes

Applications

Experiments

Open Source Science

Recent Releases of dl1-data-handler

dl1-data-handler - v0.14.5

What's Changed

  • Fix duplicates by @TjarkMiener in https://github.com/cta-observatory/dl1-data-handler/pull/164

Full Changelog: https://github.com/cta-observatory/dl1-data-handler/compare/v0.14.4...v0.14.5

- Jupyter Notebook
Published by TjarkMiener 11 months ago

dl1-data-handler - v0.14.4

What's Changed

  • Add option to force dl1 lookup by @TjarkMiener in https://github.com/cta-observatory/dl1-data-handler/pull/163

Full Changelog: https://github.com/cta-observatory/dl1-data-handler/compare/v0.14.3...v0.14.4

- Jupyter Notebook
Published by TjarkMiener 11 months ago

dl1-data-handler - v0.14.3

What's Changed

  • upgrade to ctapipe v0.25.0 by @TjarkMiener in https://github.com/cta-observatory/dl1-data-handler/pull/162

Full Changelog: https://github.com/cta-observatory/dl1-data-handler/compare/v0.14.2...v0.14.3

- Jupyter Notebook
Published by rcervinoucm 11 months ago

dl1-data-handler - v0.14.2

What's Changed

  • Join trigger table also in mono mode by @TjarkMiener in https://github.com/cta-observatory/dl1-data-handler/pull/154
  • Update python version according to SPEC0 (include 3.13, drop 3.10) by @rcervinoucm in https://github.com/cta-observatory/dl1-data-handler/pull/160
  • Bring python version 3.10 back by @rcervinoucm in https://github.com/cta-observatory/dl1-data-handler/pull/161
  • Upgrade ctapipe v0.24.0 by @TjarkMiener in https://github.com/cta-observatory/dl1-data-handler/pull/158
  • Add choice of focal length by @TjarkMiener in https://github.com/cta-observatory/dl1-data-handler/pull/155

Full Changelog: https://github.com/cta-observatory/dl1-data-handler/compare/v0.14.1...v0.14.2

- Jupyter Notebook
Published by TjarkMiener about 1 year ago

dl1-data-handler - v0.14.1

What's Changed

  • Bugfix pixrotation by @TjarkMiener in https://github.com/cta-observatory/dl1-data-handler/pull/152
  • Pre rel 0.14.1 by @nietootein in https://github.com/cta-observatory/dl1-data-handler/pull/153

Full Changelog: https://github.com/cta-observatory/dl1-data-handler/compare/v0.14.0...v0.14.1

- Jupyter Notebook
Published by TjarkMiener about 1 year ago

dl1-data-handler - v0.14.0

What's Changed

  • Adopt ctapipe coordinate frames by @TjarkMiener in https://github.com/cta-observatory/dl1-data-handler/pull/150

Full Changelog: https://github.com/cta-observatory/dl1-data-handler/compare/v0.13.0...v0.14.0

- Jupyter Notebook
Published by TjarkMiener about 1 year ago

dl1-data-handler - v0.13.0

What's Changed

  • Update release.yml by @rcervinoucm in https://github.com/cta-observatory/dl1-data-handler/pull/137
  • exampleidentifiersfile is not able to open due a permissions issue. by @Pablitinho in https://github.com/cta-observatory/dl1-data-handler/pull/139
  • Test release by @rcervinoucm in https://github.com/cta-observatory/dl1-data-handler/pull/138
  • Allow ctapipe data format version to be v5.0.0 for real data with dl1 image by @TjarkMiener in https://github.com/cta-observatory/dl1-data-handler/pull/140
  • Update build config by @rcervinoucm in https://github.com/cta-observatory/dl1-data-handler/pull/142
  • Define API for reader and image_mapper by @TjarkMiener in https://github.com/cta-observatory/dl1-data-handler/pull/143

New Contributors

  • @Pablitinho made their first contribution in https://github.com/cta-observatory/dl1-data-handler/pull/139

Full Changelog: https://github.com/cta-observatory/dl1-data-handler/compare/v0.12.0...v0.13.0

- Jupyter Notebook
Published by TjarkMiener about 1 year ago

dl1-data-handler - v0.12.0

What's Changed

  • CI&CD fixes by @rcervinoucm in https://github.com/cta-observatory/dl1-data-handler/pull/132
  • Support only CTAO data format v6.0.0 by @TjarkMiener in https://github.com/cta-observatory/dl1-data-handler/pull/133
  • Use astropy for sky offset and separation calculation by @TjarkMiener in https://github.com/cta-observatory/dl1-data-handler/pull/135
  • Allowing external algo to find trigger patch by @TjarkMiener in https://github.com/cta-observatory/dl1-data-handler/pull/134
  • Update README.rst by @TjarkMiener in https://github.com/cta-observatory/dl1-data-handler/pull/136

Full Changelog: https://github.com/cta-observatory/dl1-data-handler/compare/v0.11.1...v0.12.0

- Jupyter Notebook
Published by TjarkMiener over 1 year ago

dl1-data-handler - v0.11.1

DL1 Data Handler Release v0.11.1

Move back to old version of version.py #130

Bug Fixes and Other Changes

- Jupyter Notebook
Published by TjarkMiener almost 2 years ago

dl1-data-handler - v0.11.0

DL1 Data Handler Release v0.11.0

Enables data reading for the AI-based trigger system and data reading of LST1 observations #126 Convert absolute peak times to relative #128 CI fix #127 @rcervinoucm

Bug Fixes and Other Changes

Upgraded ctapipe to v0.20.0

- Jupyter Notebook
Published by TjarkMiener almost 2 years ago

dl1-data-handler - v0.10.11

What's Changed

  • Bugfix transformation by @TjarkMiener in https://github.com/cta-observatory/dl1-data-handler/pull/120

Full Changelog: https://github.com/cta-observatory/dl1-data-handler/compare/v0.10.10...v0.10.11

- Jupyter Notebook
Published by TjarkMiener almost 3 years ago

dl1-data-handler - v0.10.10

DL1 Data Handler Release v0.10.10

Added LSTSiPM camera #118

Bug Fixes and Other Changes

  • Upgraded to ctapipe v0.19.0

    Known Issues

  • Increase of RAM usage in the DL1DataReaderSTAGE1 subclass (Issue #104).

- Jupyter Notebook
Published by TjarkMiener almost 3 years ago

dl1-data-handler - v0.10.9

DL1 Data Handler Release v0.10.9

Upgrade to ctapipe v0.17.0 and CTA data format v5 #117

Bug Fixes and Other Changes

  • Version of the CTA data model was not tracked properly in the reader class.

    Known Issues

  • Increase of RAM usage in the DL1DataReaderSTAGE1 subclass (Issue #104).

- Jupyter Notebook
Published by TjarkMiener about 3 years ago

dl1-data-handler - v0.10.8

DL1 Data Handler Release v0.10.8

Upgrade to (latest) ctapipe v0.16.0 and CTA data format v4 #116

Known Issues

  • Increase of RAM usage in the DL1DataReaderSTAGE1 subclass (Issue #104).

- Jupyter Notebook
Published by TjarkMiener over 3 years ago

dl1-data-handler - v0.10.7

DL1 Data Handler Release v0.10.7

Major bug fixes w.r.t. v0.10.6 (highly recommended to upgrade to this version)

Major Features And Improvements

  • Modified the Stage1 subclass reader to read files without the shower simulation table, e.g. real data from LST-1
  • Automated handling of multiple classes with CTLearn v0.6.1
  • Moved class weight calculation from CTLearn into the dl1dh reader # Bug Fixes and Other Changes
  • Shuffle ex. ids before storing them. It caused troubles when resuming a CTLearn run.
  • Changed the dim of the parameter list in mono mode to make it work with the output handler of CTLearn v0.6.1.
  • Upgrade to (latest) ctapipe v0.15.0

Known Issues

  • Increase of RAM usage in the DL1DataReaderSTAGE1, when the DL1 files are split by telid. Issue #104. At the moment it is recommended to use the DL1 files, where the telescope tables are split by teltypes.

- Jupyter Notebook
Published by TjarkMiener over 3 years ago

dl1-data-handler - v0.10.6

DL1 Data Handler Release v0.10.6

Upgrade to (latest) ctapipe v0.14.0 #115

Known Issues

  • Increase of RAM usage in the DL1DataReaderSTAGE1, when the DL1 files are split by telid. Issue #104. At the moment it is recommended to use the DL1 files, where the telescope tables are split by teltypes.

- Jupyter Notebook
Published by TjarkMiener almost 4 years ago

dl1-data-handler - v0.10.5

DL1 Data Handler Release v0.10.5

Added quality cuts on parameters in stereo mode

Major Features And Improvements

  • Quality cuts in stereo reading #113

Bug Fixes and Other Changes

Known Issues

  • Increase of RAM usage in the DL1DataReaderSTAGE1, when the DL1 files are split by telid. Issue #104. At the moment it is recommended to use the DL1 files, where the telescope tables are split by teltypes.

- Jupyter Notebook
Published by TjarkMiener about 4 years ago

dl1-data-handler - v0.10.4

DL1 Data Handler Release v0.10.4

Upgrade to MARS v2-20-1 superstar files Aligned MAGIC files with ctapipe conventions to share the same CTLearn config

Major Features And Improvements

Bug Fixes and Other Changes

  • code clean up #112

Known Issues

  • Increase of RAM usage in the DL1DataReaderSTAGE1, when the DL1 files are split by telid. Issue #104. At the moment it is recommended to use the DL1 files, where the telescope tables are split by teltypes.

- Jupyter Notebook
Published by TjarkMiener about 4 years ago

dl1-data-handler - v0.10.3

DL1 Data Handler Release v0.10.3

Bug fix for stage1 datasets (split by tel_types)

Major Features And Improvements

Bug Fixes and Other Changes

  • fix the tel id select option for stage1 datasets (split by tel_types) #111

Known Issues

  • Increase of RAM usage in the DL1DataReaderSTAGE1, when the DL1 files are split by telid. Issue #104. At the moment it is recommended to use the DL1 files, where the telescope tables are split by teltypes.

- Jupyter Notebook
Published by TjarkMiener about 4 years ago

dl1-data-handler - v0.10.2

DL1 Data Handler Release v0.10.2

Update of dependencies #110

Major Features And Improvements

Bug Fixes and Other Changes

  • Fix hyperlinks in README for pypi upload

Known Issues

  • Increase of RAM usage in the DL1DataReaderSTAGE1, when the DL1 files are split by telid. Issue #104. At the moment it is recommended to use the DL1 files, where the telescope tables are split by teltypes.

- Jupyter Notebook
Published by TjarkMiener about 4 years ago

dl1-data-handler - v0.10.1

DL1 Data Handler Release v0.10.1

Support latest official CTA DL1 data model v2.1.0 (still supporting Prod5 files with v1.1.0) Compatibility with ctapipe v0.12.0

Major Features And Improvements

  • Support multiple versions of the official CTA DL1 data model #107
  • Moved the class weights calculation for unbalanced datasets from CTLearn to the DL1DataReader #108

Bug Fixes and Other Changes

  • Formatted via black
  • Updated the README with installation instructions
  • Enhance version handling

Known Issues

  • Increase of RAM usage in the DL1DataReaderSTAGE1, when the DL1 files are split by telid. Issue #104. At the moment it is recommended to use the DL1 files, where the telescope tables are split by teltypes.

- Jupyter Notebook
Published by TjarkMiener about 4 years ago

dl1-data-handler - v0.10.0

DL1 Data Handler Release v0.10.0

Support official CTA DL1 data model v1.1.0 Compatibility with ctapipe v0.10.5 Support MAGIC real data by @astrojarred Deprecating DL1DH writer

Major Features And Improvements

  • Subclassing the DL1Reader to support the official DL1 data model (for both versions split by telid and by teltypes) and the deprecated DL1DH DL1 data format
  • Updated DLEventSource for MAGIC real data files. @astrojarred
  • Added ability to save and load example identifiers in a separate file to skip the init of the DL1Reader.
  • Make 'generaterunlist' and 'writedata' scripts to command-line tools.
  • Updated example notebooks.
  • Updated conda package build files and setup.py dependencies. Removed light setup (without ctapipe), because ctapipe is now scritly required.

Bug Fixes and Other Changes

  • DL1a and/or DL1b data can be read by the subclasses of DL1Reader.

Known Issues

  • Increase of RAM usage in the DL1DataReaderSTAGE1, when the DL1 files are split by telid. Issue #104. At the moment it is recommended to use the DL1 files, where the telescope tables are split by teltypes.
  • Calling file.close() in the destructor of CTAMLDataDumper causes a KeyError on some data files due to an empty NodeManager registry. The cause of this exception is unclear. The exception resulting from the call is currently being caught and ignored.

- Jupyter Notebook
Published by TjarkMiener over 4 years ago

dl1-data-handler - DL1 Data Handler 0.8.3

Release 0.8.3

Compatibility with ctlearn v0.5.0. Bugfixes w.r.t. v0.8.2.

Major Features And Improvements

Bug Fixes and Other Changes

  • Fix counting of class examples for labels that are sequences. @aribrill
  • added electrons to the particle transformation. @TjarkMiener

- Jupyter Notebook
Published by TjarkMiener about 5 years ago

dl1-data-handler - DL1 Data Handler 0.9.0

Release 0.9.0

Numerous bugfixes and improvements. DLEventSource for MAGIC calibrated files by @sahilyadav27. DLEventSource for MAGIC superstar files. Expansion of the data format with image parameters by @LucaRomanato. Compatibility with ctapipe v0.9.1.

Major Features And Improvements

  • Updated DL1DataWriter to ctapipe v0.9.1 and expand the data format with image parameters.
  • Make ctapipe's imageextractor and gainselector configurable in the config.
  • Added multiple parameter tables for different cleaning settings. Use MARS default ones for MAGIC superstar files.
  • Added ability to only store selected tels (by tel_id).
  • Updated example notebook (also for MAGIC data).
  • Updated conda package build files and setup.py dependencies.

Bug Fixes and Other Changes

Known Issues

  • The image parameters can't be read by the current DL1DataReader. This is planned for the next release DL1DH v0.9.1.
  • Calling file.close() in the destructor of CTAMLDataDumper causes a KeyError on some data files due to an empty NodeManager registry. The cause of this exception is unclear. The exception resulting from the call is currently being caught and ignored.

- Jupyter Notebook
Published by TjarkMiener about 5 years ago

dl1-data-handler - DL1 Data Handler 0.8.2

Release 0.8.2

Compatibility with ctapipe v0.8.0. Numerous bugfixes and improvements.

DL1DataReader compatible for the file format of DL1DH >= v0.7.4.

Major Features And Improvements

  • Updated DL1DataWriter to ctapipe v0.8.0.
  • gain_selector in ctapipe now. Redundant unit tests deleted.
  • Numerous improvements for the ImageMapper (IM) class.
  • ctapipe and ctapipe-extra optional dependencies for DL1DataReader and IM.
  • Added and modified Transform functions for the data processor.
  • Added telescope pointing for alt/az transformation
  • Added 'sorting' transformation for stereo reading.
  • Added and modified filters for the data processor in filters.py.
  • Updated example notebook.
  • Updated conda package build files and setup.py dependencies.

Bug Fixes and Other Changes

  • Fixed FlashCam mapping bug with reading pixel positions from the data files.
  • Normalization for nearest interpolation in IM.
  • Removed rotation back to original orientation option in IM.
  • Speeded up the data reading.
  • Added parallel reads support.
  • Sorting transformation fixed performance drop from the CTLearn cnn-rnn (stereo) model.

Known Issues

  • Calling file.close() in the destructor of CTAMLDataDumper causes a KeyError on some data files due to an empty NodeManager registry. The cause of this exception is unclear. The exception resulting from the call is currently being caught and ignored.

- Jupyter Notebook
Published by TjarkMiener over 5 years ago

dl1-data-handler - DL1 Data Handler 0.8.1 Legacy Version

Release 0.8.1-legacy

"Legacy" release adapting DL1DataReader for the file format of DL1DH <0.6.0.

Major Features And Improvements

  • Added reader_legacy module containing a DL1DataReader adapted for the legacy file format.

Bug Fixes and Other Changes

  • Updated dl1_data_handler_demo notebook for legacy reader.
  • Updated transforms and utils modules to use legacy column names in relevant Transforms and filters.
  • Added SortTelescopes Transform to sort telescopes by trigger or image size.
  • Added event_multiplicity_filter to filter events based on multiplicity.

Known Issues

  • The current version of ctapipe (v0.6.2) does not correctly support a version tag that can be read by pkgresources (instead _VERSION__ = "unknown") so it is not being written correctly to the output file headers.
  • Calling file.close() in the destructor of CTAMLDataDumper causes a KeyError on some data files due to an empty NodeManager registry. The cause of this exception is unclear. The exception resulting from the call is currently being caught and ignored.

- Jupyter Notebook
Published by aribrill over 6 years ago

dl1-data-handler - DL1 Data Handler 0.8.0

Release 0.8.0

Integration of DL1DH Reader, Processor, and ImageMapper completed. Numerous bugfixes and improvements for compatibility with CTLearn.

Added option for serial/parallel writing modes to DL1DH Writer. Fixes to setuptools and conda build process.

Major Features And Improvements

  • Finished changes to reader and image mapper APIs to be compatible with CTLearn
  • Finished incorporating image mapper and DL1DH processor.
  • Added example notebook.
  • Added six new Transform functions for the data processor and moved to transform.py
  • Added single threaded/parallel writes modes as a config option for DL1DataWriter.
  • Updated conda package build files and setup.py dependencies.

Bug Fixes and Other Changes

  • Fixed output_dir option of DataWriter to correctly write output files to directory.
  • Added outputdir option to generaterunlists.py script.
  • Moved example config files and runlists to /config folder
  • Added separate expectedmcevents config setting for optimizing chunking on the MC_Events table.
  • Fixed various issues with setuptools dependencies.
  • Fixed FACT camera shape mismatch by upgrading to ctapipe-extra 0.2.17.
  • Changed mapping table indices to start at 0 (removed 0 padding).
  • Added option for camera rotation for LST, NectarCam, MAGICCam in image mapper.
  • Fixed outputdir option in writedata.py script.
  • Moved config files to config folder
  • Changed indexing convention in image mapper.
  • Changed get_image() to return a vector when map method is axial addressing
  • Fixed status badge

Known Issues

  • The current version of ctapipe (v0.6.2) does not correctly support a version tag that can be read by pkgresources (instead _VERSION__ = "unknown") so it is not being written correctly to the output file headers.
  • Calling file.close() in the destructor of CTAMLDataDumper causes a KeyError on some data files due to an empty NodeManager registry. The cause of this exception is unclear. The exception resulting from the call is currently being caught and ignored.

- Jupyter Notebook
Published by bryankim96 almost 7 years ago

dl1-data-handler - DL1 Data Handler 0.7.4

Release 0.7.4

Added mode to write non-triggered MC events using Eventio. Some optimizations/performance improvements.

Note: although the Reader and Processor/ImageMapper have been added, they are still under development. A finalized version of both will be part of the next major release (v0.8.0).

Major Features And Improvements

  • Added savemcevents flag to allow saving of MC event information in a separate table (MC_Events).
  • Added single-threaded write data script for easier testing/profiling.

Bug Fixes and Other Changes

  • Reduced write time by ~12% per event for triggered events (events with images) by making various optimizations.
  • Cleaned up and simplified class definitions/function signatures.

Known Issues

  • The current version of ctapipe (v0.6.2) does not correctly support a version tag that can be read by pkgresources (instead _VERSION__ = "unknown") so it is not being written correctly to the output file headers.
  • Calling file.close() in the destructor of CTAMLDataDumper causes a KeyError on some data files due to an empty NodeManager registry. The cause of this exception is unclear. The exception resulting from the call is currently being caught and ignored.
  • There appears to be an ongoing issue with processing SCT data. This is being investigated.

- Jupyter Notebook
Published by bryankim96 almost 7 years ago

dl1-data-handler - DL1 Data Handler 0.7.3

Release 0.7.3

New release to fix missing gain/channel selection for processing the Prod3 NSB dataset.

Note: although the Reader and Processor have been added, they are still under development. A finalized version of both will be part of the next major release (v0.8.0).

Major Features And Improvements

  • Added gain channel selection functionality to DL1DataWriter.
  • DL1DataReader and Processor functionality added but still under development.

Bug Fixes and Other Changes

  • Fix to filename parsing in generate_runlist.py script.
  • Fixes/improvements to installation and dependency management with setuptools and conda.
  • Fixed Travis/pytest integration.

Known Issues

  • prodsitearray, prodsitecoord, and prodsitesubarray are included in the output files but defaulted to None, as they are included in the ctapipe MC header container but not supported in event_source as of https://github.com/cta-observatory/ctapipe/commit/5ef63ef2a7a093455f4b0683a68653cc42e84df7.
  • Calling file.close() in the destructor of CTAMLDataDumper causes a KeyError on some data files due to an empty NodeManager registry. The cause of this exception is unclear. The call has been removed as it has no effect on the output files.
  • There appears to be an ongoing issue with processing SCT data. This is being investigated.

- Jupyter Notebook
Published by bryankim96 almost 7 years ago

dl1-data-handler - DL1 Data Handler 0.7.2

Release 0.7.2

This release includes an update to the version in setup.py.

Major Features And Improvements

  • None.

Bug Fixes and Other Changes

  • Version in setup.py updated to 0.7.2 to avoid version confusion by users.

Known Issues

  • prodsitearray, prodsitecoord, and prodsitesubarray are included in the output files but defaulted to None, as they are included in the ctapipe MC header container but not supported in event_source as of https://github.com/cta-observatory/ctapipe/commit/5ef63ef2a7a093455f4b0683a68653cc42e84df7.
  • Calling file.close() in the destructor of CTAMLDataDumper causes a KeyError on some data files due to an empty NodeManager registry. The cause of this exception is unclear. The call has been removed as it has no effect on the output files.
  • There appears to be an ongoing issue with processing SCT data. This is being investigated.

- Jupyter Notebook
Published by nietootein about 7 years ago

dl1-data-handler - DL1 Data Handler 0.7.1

Release 0.7.1

This release includes fixes to the pip/setuptools installation as well as a minor fix to the generate_runlist.py script.

Major Features And Improvements

  • Setuptools installation process streamlined so it does not depend on manual installation of dependencies from source/pip. Now the entire installation can be done via setup.py/pip. requirements.txt is no longer required and was therefore removed.
  • Due to issues with exporting conda environments across systems, conda environment-based installation was removed. A conda package release is planned for the near future.

Bug Fixes and Other Changes

  • scripts/generaterunlist.py was modified to handle filenames with underscores in their prodinfo fields.

Known Issues

  • prodsitearray, prodsitecoord, and prodsitesubarray are included in the output files but defaulted to None, as they are included in the ctapipe MC header container but not supported in event_source as of https://github.com/cta-observatory/ctapipe/commit/5ef63ef2a7a093455f4b0683a68653cc42e84df7.
  • Calling file.close() in the destructor of CTAMLDataDumper causes a KeyError on some data files due to an empty NodeManager registry. The cause of this exception is unclear. The call has been removed as it has no effect on the output files.
  • There appears to be an ongoing issue with processing SCT data. This is being investigated.

- Jupyter Notebook
Published by bryankim96 about 7 years ago

dl1-data-handler - DL1 Data Handler 0.7.0

Release 0.7.0

This release includes only the writer (raw data to HDF5) portion of the full DL1 Data Handler package. The matching data reader and processor will be included in future releases.

Major Features And Improvements

  • Major refactoring of image-extractor into DL1DataWriter class, based on a DL1DataDumper class and the ctapipe EventSource class.
  • Now uses on YAML configuration files and run lists for all processing settings
  • Python multiprocessing package used to allow parallel processing of runs.
  • Image mapping functionality in image.py removed.
  • Command line script moved out of main package and into separate write_data.py script.
  • generate_runlist.py script added to automatically create run lists from directories of files.
  • Telescope type names in output files (table columns and table names) modified from original ctapipe names to be valid Python identifiers (for use when doing selections).
  • Telescope optics and camera types added to telescope type table.
  • xmax column added to event table.
  • Telescope image multiplicity columns added to event table.

Bug Fixes and Other Changes

  • .gitignore updated to include build, dist directories.
  • TravisCI configuration instructions updated, Python version updated 3.5 -> 3.6.
  • AUTHORS list updated.
  • runid renamed to obsid to match ctapipe container field.
  • requirements.txt file reduced by using pipreqs instead of pip freeze.
  • Out of date scripts moved to scripts/deprecated.
  • Improvements to logging.
  • Example config and run list files added.
  • Fixed an issue where some tables were not being written correctly before files were closed.

Breaking Changes to the API

  • Various name changes to columns and tables (see wiki for full data format specification)
  • Telescope type names in output files (table columns and table names) modified from original ctapipe names to be valid Python identifiers (for use when doing selections).
  • Telescope optics and camera types added to telescope type table.
  • xmax column added to event table
  • Telescope image multiplicity columns added to event table

Known Issues

  • prodsitearray, prodsitecoord, and prodsitesubarray are included in the output files but defaulted to None, as they are included in the ctapipe MC header container but not supported in event_source as of https://github.com/cta-observatory/ctapipe/commit/5ef63ef2a7a093455f4b0683a68653cc42e84df7.
  • Calling file.close() in the destructor of CTAMLDataDumper causes a KeyError on some data files due to an empty NodeManager registry. The cause of this exception is unclear. The call has been removed as it has no effect on the output files.

- Jupyter Notebook
Published by bryankim96 about 7 years ago

dl1-data-handler - Image Extractor 0.6.0

Release 0.6.0

Major Features And Improvements

  • Updated telescope, optics, and camera naming conventions to match ctapipe formatting. Implemented distinction between camera (image)-dependent settings and telescope type settings.
  • Added configuration file support and example configuration file for ease of use.
  • Added support for compression via PyTables.
  • Added support for chunking via PyTables.
  • Added support for table column indexing via PyTables.
  • ImageExtractor is now available as a conda package at https://anaconda.org/bryankim96/image-extractor.

Bug Fixes and Other Changes

  • Updated to internally use new ctapipe.io.EventSource instead of hessioeventsource.
  • Updated name of event.r0.runid to event.r0.obsid.
  • Fixed overflow bug in tel_id column causing duplicated telescope ids.
  • Added environment.yml file in addition to requirements.txt for easy Anaconda environment setup.
  • On exception/failure while processing, ImageExtractor will now remove the incomplete file.

Breaking Changes to the API

  • Type change in telid column in arrayinfo table.
  • Changes to all telescope names (affects both event_info table rows and the names of image tables).

Known Issues

  • Some metadata parameters within the user attributes, specifically those coming from in the simtel history block, are still not filled because they are not yet supported by ctapipe's EventSource. See this issue and this issue

- Jupyter Notebook
Published by bryankim96 over 7 years ago

dl1-data-handler - Image Extractor 0.5.1

Release 0.5.1

Major Features And Improvements

Bug Fixes and Other Changes

  • Corrected release version in setup.py.

Breaking Changes to the API

Known Issues

  • Due to the new event.mc.showerprimaryid attribute of the ctapipe MC container being only recently added, the current ctapipe version on anaconda (https://anaconda.org/cta-observatory/ctapipe) does not include it. This functionality will not work with the conda installation of ctapipe. Instead, at the current time it is necessary to install either the dev version of ctapipe as described here (https://cta-observatory.github.io/ctapipe/getting_started/index.html#get-the-ctapipe-software) or any ctapipe release from 0.5.3 on.
  • Some metadata parameters within the user attributes, specifically those coming from in the simtel history block, are not filled. See this issue.

- Jupyter Notebook
Published by nietootein about 8 years ago

dl1-data-handler - Image Extractor 0.5.0

Release 0.5.0

Major Features And Improvements

  • Added compatibility with breaking changes in ctapipe's instrument container implemented in v0.5.3.
  • Added process_dataset.py script to merge, calibrate and dump to h5 datasets from CTA MC production "prod3b".

Bug Fixes and Other Changes

Breaking Changes to the API

  • Compatibility with ctapipe versions previous to 0.5.3 has not been considered, so the code will break if one of such older versions of ctapipe is used in conjunction with this release.

Known Issues

  • Due to the new event.mc.showerprimaryid attribute of the ctapipe MC container being only recently added, the current ctapipe version on anaconda (https://anaconda.org/cta-observatory/ctapipe) does not include it. This functionality will not work with the conda installation of ctapipe. Instead, at the current time it is necessary to install either the dev version of ctapipe as described here (https://cta-observatory.github.io/ctapipe/getting_started/index.html#get-the-ctapipe-software) or any ctapipe release from 0.5.3 on.
  • Some metadata parameters within the user attributes, specifically those coming from in the simtel history block, are not filled. See this issue.

- Jupyter Notebook
Published by nietootein about 8 years ago

dl1-data-handler - Image Extractor 0.4.1

Release 0.4.1

Major Features And Improvements

  • Added event_index column to image tables (mapping back to the event table) for use when reading data for single-telescope use

Bug Fixes and Other Changes

  • Removed LICENCE
  • Added .landscape.yml

Breaking Changes to the API

Known Issues

  • Due to the new event.mc.showerprimaryid attribute of the ctapipe MC container being only recently added, the current ctapipe version on anaconda (https://anaconda.org/cta-observatory/ctapipe) does not include it. This functionality will not work with the conda installation of ctapipe. Instead, at the current time it is necessary to install the dev version of ctapipe as described here (https://cta-observatory.github.io/ctapipe/getting_started/index.html#get-the-ctapipe-software).
  • Some metadata parameters still not fully implemented

- Jupyter Notebook
Published by bryankim96 over 8 years ago

dl1-data-handler - ImageExtractor 0.4.0

Release 0.4.0

Major Features And Improvements

  • Existing telescope types renamed (SCT -> MSTS) and additional telescope types now supported (LST, MSTF, MSTN, SST1, SSTA, SSTC). 2D image mode still only supported for MSTS tel type
  • Trace to image conversion for 2D image mode now handled by pre-generating a mapping table
  • Now uses telescope numpixels * opticalfoc_len to identify telescope types
  • Optical focal length and number of pixels now stored in the telescope table
  • Saving raw image vectors (charge and arrival time) now supported in '1D' image mode
  • '1D' image mode now saves raw images as float vectors, not uint vectors
  • '1D' image mode now saves a PyTables table containing the charge image vector and arrival time image vector as separate columns
  • All image tables now store a blank (all-zero) image at index 0. Index 0 is now used in event indices vectors to indicate a non-existant image, instead of index -1.
  • Telescope table renamed to Telescope_Info
  • Event table renamed to Event_Info
  • Image tables renamed to [teltype/telid]
  • Now supports varying image scale factors and image dtypes by telescope type
  • Units for event-level and telescope-level parameters now saved in the corresponding table attributes
  • Removed reconstructed energy parameter from Event Info table
  • Gammahadronlabel parameter renamed to particle_id
  • All float parameters (event-level and telescope-level) now saved as 32-bit floats
  • Preselection cuts now handled using a dictionary cutsdict (to be used in ctapipe) or a pre-generated EventDisplay cuts dictionary EDcuts_dict
  • File-level metadata now stored in root group attributes. Identical metadata fields are now enforced for all events in the output file
    • particle_id
    • zenith
    • azimuth
    • ImageExtractor version
    • ctapipe version
    • runlist
  • The following metadata fields are included but reading them from the data is not yet implemented
    • CORSIKA version
    • Simtel version
    • prod site array
    • prod site subarray
    • prod site B field
    • prod site altitude
    • spectral index
    • E min
    • E max
  • Config file support removed, along with unnecessary options. Example config file removed. All remaining configuration options, value checking, default values now handled through the ImageExtractor class constructor
    • 'mode'(ghclass,energyrecon) removed
    • 'storagemode' (all,mapped) -> storagemode (telid,teltype)
    • 'usepkldict' (True,False), energybins, preselectioncuts -> EDcutsdict,cuts_dict
    • image 'mode' (PIXELSTIMING3C, etc.) -> imgmode (1D,2D), imgchannels, include_timing
    • image 'scalefactor' -> imgscalefactors(dict {teltype: scale_factor})
    • image 'dtype' -> imgdtypes(dict {teltype: dtype})
    • image 'dimorder' -> imgdim_order
    • telescope 'typemode' -> teltypelist(list [teltypes, ...])
    • energy_recon removed
  • Spliting the data into train/val/test sets now supported
  • Shuffling the event data randomly now supported
  • '--split' command line flag now takes 3 command line arguments specifying the split fractions
  • '--shuffle' command line flag now takes a random seed argument
  • '--binscutsdict' command line argument renamed to '--EDcutsdict'
  • Command line program now takes a text file runlist as an input instead of a wildcard expression
  • Added TravisCI and test coverage monitoring. Related dependencies added

Bug Fixes and Other Changes

  • Removed unused import statements
  • Collected constants
  • Some PEP8 fixes (still ongoing)
  • Renamed rowdescriptors.py to rowtypes.py, trace_converter.py to image.py
  • Debug command line flag now works properly with logger
  • Debug output now gives the total number of events in the output file after each new input file processed
  • Binning, shuffling, selecting telescopes, writing metadata all moved to ImageExtractor methods
  • Some default ImageExtractor constructor arguments modified
  • Example pytest tests added

Breaking Changes to the API

  • SCT telescope type renamed to MSTS
  • Config file removed and config options moved to ImageExtractor constructor. See above for detailed changes
  • Images now stored in tables, not earrays
  • Default image dtype changed to float32 from uint16. Images now saved without truncation or rounding
  • Charge image vectors and arrival time image vectors now stored separately
  • All image tables now save a blank image at index 0. Index 0 now used in the event table to indicate a non-existent image/non-triggered camera
  • Telescope table renamed to Telescope_Info
  • Event table renamed to Event_Info
  • Image tables renamed to [teltype/telid]
  • Image scale factors and dtypes now determined by telescope type
  • Reconstructed energy parameter removed
  • Gammahadronlabel renamed to particle_id
  • All float parameters (event-level and telescope-level) now saved as 32-bit floats
  • Command line program now takes a text file runlist as an input instead of a wildcard expression

Known Issues

  • Due to the new event.mc.showerprimaryid attribute of the ctapipe MC container being only recently added, the current ctapipe version on anaconda (https://anaconda.org/cta-observatory/ctapipe) does not include it. This functionality will not work with the conda installation of ctapipe. Instead, at the current time it is necessary to install the dev version of ctapipe as described here (https://cta-observatory.github.io/ctapipe/getting_started/index.html#get-the-ctapipe-software).
  • Some metadata parameters still not fully implemented

- Jupyter Notebook
Published by bryankim96 over 8 years ago

dl1-data-handler - ImageExtractor 0.3.0

Release 0.3.0

Major Features And Improvements

  • Full reorganization of codebase to enhance modularity and adhere to standard development practices. Existing code split into new modules, classes, and methods.
    • imageextractor functionality and command line tool restructured into ImageExtractor class + associated class methods and moved to imageextractor.py module. Image_extractor's command line functionality now supports full dataset processing (multiple simtel files) as well as dataset shuffling and splitting (previously handled using external scripts).
    • Trace to image/array conversion functionality reorganized into TraceConverter class + associated class methods and moved to trace_converter.py module.
    • Custom Pytables row descriptor classes (Event, Tel) moved to row_descriptors.py module.
    • Configuration file spec and other config-related command line tools reorganized into config.py module.
    • Modules and scripts moved into new directory structure. __init__.py added to image_extractor package.
  • Changed gammahadronlabel to use CORSIKA particle ID codes (gamma = 0, proton = 101, etc.) instead of previous binary labels (gamma = 1, proton = 0).
  • Added support for Python package installation with setuptools (setup.py). Package not registered with Conda or PyPl, but can be installed locally using pip.
  • Added preliminary support for automatic unit testing and continuous integration with Travis-CI. Installation tests are all passing, but unit tests are not implemented (placeholder created in /tests). Coverage monitoring is also supported through Coveralls.
  • Changed requirements.txt to include full dependency list. Can now be used directly with pip install or conda install to install all dependencies.
  • Changed default configuration files to use 'mapped' storage mode, 'PIXELS1C' image mode, and a scalefactor of 1. With the new default configuration file, SCT images will be of shape (120,120,1) and will be stored in separate arrays outside of the event table.
  • Removed shufflesplitevents.py and generatedatasetpytables.sh from /scripts directory, as their functionality is now included in image_extractor.py
  • Changed shuffling and splitting functions (now in image_extractor.py) to write shuffled/split events to new tables in existing HDF5 file rather than to a temporary file.
  • Other refactoring and reorganization. Global constants unified and moved into appropriate classes/modules.
  • Added LICENCE file

Bug Fixes and Other Changes

  • Added imgdimordering parameter to default config files. Default value set to 'channels_last' to agree with TensorFlow standards.
  • Fixed improper assignment to binnumber, reconstructedenergy when not using binscutsdict.
  • Changed binscutsdict command line argument to imageextractor.py to be optional. imageextractor code modified to correctly handle case when binscutsdict is missing/None. Will throw exception if config file requires binscutsdict but none is provided.
  • Added maxevents optional command line argument to imageextractor.py specifying the maximum number of events to read from a given simtel file.
  • Added shower core position (x,y), height of first interaction, altitude (zenith) angle, and azimuth angle parameters to event table.
  • Added run array direction parameter to telescope table.
  • Renamed 'MCenergy' parameter in event table to 'mcenergy'.
  • Changed type of 'tel_map' parameter in event table from Int16 to Int32 to allow for large indices (more events) and avoid future overflow issues.

Breaking Changes to the API

  • gammahadronlabel convention changed from (gamma = 1, proton = 0) to (gamma = 0, proton = 101, etc.).
  • Naming change from 'MCenergy' to 'mcenergy' for parameter in event table.
  • Datatype change to 'tel_map' parameter in event table from Int16 to Int32.
  • Major change to default config file behavior to 'mapped' storage mode, 'PIXELS1C' image mode, and a scalefactor of 1.
  • Requirements.txt format now includes full list of dependencies.

Known Issues

  • Due to the new event.mc.showerprimaryid attribute of the ctapipe MC container being only recently added, the current ctapipe version on anaconda (https://anaconda.org/cta-observatory/ctapipe) does not include it. This functionality will not work with the conda installation of ctapipe. Instead, at the current time it is necessary to install the dev version of ctapipe as described here (https://cta-observatory.github.io/ctapipe/getting_started/index.html#get-the-ctapipe-software).

- Jupyter Notebook
Published by bryankim96 over 8 years ago

dl1-data-handler - Image Extractor 0.2.1

Release 0.2.1

Major Features And Improvements

  • Moved configspec out of imageextractor.py and into config.py. Moved config.py out of /scripts and into main directory.
  • Imageextractor now only saves triglist when in 'all' mode and only stores tel_map when in 'mapped' mode to avoid duplicate information.
  • Moved argument parsing, config file validation/reading, and and binscutsdict loading out of imageextractor and into if _name__ == 'main' block.

Bug Fixes and Other Changes

  • Gamma hadron label now determined using particle id from event.mc.showerprimaryid
  • Added basic docstrings
  • Replaced all print statements for logging and debugging with logger statements
  • Various minor naming and style changes to align closer with PEP8 standards
  • Removed unused import statements

Breaking Changes to the API

  • Imageextractor now only saves triglist when in 'all' mode and only stores tel_map when in 'mapped' mode to avoid duplicate information.

Known Issues

  • Due to the new event.mc.showerprimaryid attribute of the ctapipe MC container being only recently added, the current ctapipe version on anaconda (https://anaconda.org/cta-observatory/ctapipe) does not include it. This functionality will not work with the conda installation of ctapipe. Instead, at the current time it is necessary to install the dev version of ctapipe as described here (https://cta-observatory.github.io/ctapipe/getting_started/index.html#get-the-ctapipe-software).

- Jupyter Notebook
Published by bryankim96 over 8 years ago

dl1-data-handler - ImageExtractor 0.2.0

Release 0.2.0

Organizational update. All previous implementations archived and removed.

Major Features And Improvements

Bug Fixes and Other Changes

  • Added requirements.txt file of Python package dependencies (generated using pipreq)
  • imageextractorpytables.py renamed to image_extractor.py

Breaking Changes to the API

  • All deprecated implementations removed. See v0.1.0 if required.

Known Issues

- Jupyter Notebook
Published by bryankim96 over 8 years ago

dl1-data-handler - ImageExtractor 0.1.0

Release 0.1.0

Description

First pre-release version of imageextractor. Reads events from a simtel file in ctapipe, uses binscuts_dict from EventDisplay MSCW stage to apply pre-selection cuts and binning, writes images and other metadata parameters to Pytables formatted HDF5.

IMPORTANT NOTE: Release v0.1.0 contains deprecated imageextractor implementations (using ROOT and h5py) in various directories for reference. The newest, working version described above is located in image-extractor/ctapipe/v2pytables/ and is named imageextractorpytables.py. Starting from v0.2.0, all older implementations will be removed and only the current ctapipe/Pytables implementation will be supported.

Features

  • Requires a saved dict (pickled) from EventDisplay MSCW analysis to determine array-level parameter cuts and energy binning
  • Supports both 'all' (storage of metadata + images + blank images in a single table) and 'mapped' (storage of metadata and indices to images in table, images stored separately in arrays) modes.
  • Supports both gamma-hadron classification and energy reconstruction (classification) modes
  • Supports charge-only image arrays, charge + peak_pos image arrays, padded image arrays
  • Supports scaling of image sizes
  • Supports channelsfirst and channelslast image array dimension ordering
  • Saves telescope information (tel_id, type, position) in a table, one per file
  • Uses config_obj formatted config files to handle processing options
  • Utilizes default ctapipe data calibration. Charge values per pixel are re-scaled, truncated, and converted to int.
  • Saves the following event-level parameters:
    • event number
    • run number
    • gamma-hadron label (0 vs. 1)
    • MC energy
    • reconstructed energy (from EventDisplay MSCW)
    • trig_list
    • tel_map (for mapped storage method)

Notes

  • Does not meet internal style guidelines (ctapipe)
  • Only supports saving SCT telescope images to arrays. Although other telescope selection modes (including SST and LST) are included in the configuration file, they are not implemented and should not be used
  • Telescope types determined using num_pixels
  • Particle type determined by parsing input file name

- Jupyter Notebook
Published by bryankim96 over 8 years ago

dl1-data-handler - v0.1 - preliminary

Preliminary release for generation of first set of image data.

Includes support for the following metadata fields:

eventType (only gamma and proton supported) eventID impactParameter telescopeNumber telx tely MCprim MCe0 MCxcore MCycore MCze MCaz MCxoff MCyoff pedrms

- Jupyter Notebook
Published by bryankim96 over 9 years ago