tkp - Release 6.0

This is the stable python 3 release of TraP. See Release 6.0 release candidate for main changes

- Python
Published by AntoniaR about 2 years ago

tkp - Release 6.0 release candidate

There are significant improvements to TraP throughout this release. These improvements lead to some speed improvements and solves a source association issue seen in R5.0. I recommend setting up a new project and new jobs to ensure that you have the correct pipeline.cfb and job_params.cfg files.

Major changes

• Upgraded to support Python 3 only. I recommend using a Conda environment running Python 3.11 to install this release.
• Upgraded database to postgresql 14. This means you also need to upgrade the database running on your system. N.B. I have not tried this yet so be careful - I do not know what happens to your old databases at this time. I will be testing this priori to this becoming an official release.
• Significant improvements and parallelisation of PySE.

The documentation will be updated prior to this becoming an official release.

If you test this release candidate, please let me know about your experiences and if there are any issues with the new code.

- Python
Published by AntoniaR about 3 years ago

tkp - Release 5.0

Update to job_params.cfg

A new section for all quality checks was created. Generic quality control parameters in persistence and quality_lofar were moved to this new section.

Quality checks

The rms values of all fits images are checked to global minimum and maximum rms values. The automatic rms based image quality control check was given its own independent sigma threshold.

[quality] rmsrejsigma = 3 ; threshold for rejecting images using rms histogram

The restoring beam parameter checks, initially developed for LOFAR images, now applied to all fits images. Negative RA values Some imagers, e.g. WSClean, define the WCS in the fits files as -180<RA<180. This caused issues within the source association algorithms. A new simple fix has been applied so these images can be correctly processed.

Removal of MonetDB

MonetDB is no longer supported but was still present in code and documentation. All references to MonetDB have now been removed.

Fixing of errors

A number of outstanding errors have been fixed.

- Python
Published by AntoniaR about 4 years ago

tkp - Release 5.0 release candidate

Update to job_params.cfg

A new section for all quality checks was created. Generic quality control parameters in persistence and quality_lofar were moved to this new section.

Quality checks

The rms values of all fits images are checked to global minimum and maximum rms values. The automatic rms based image quality control check was given its own independent sigma threshold. ``` [quality] rmsrejsigma = 3 ; threshold for rejecting images using rms histogram

``` The restoring beam parameter checks, initially developed for LOFAR images, now applied to all fits images. Negative RA values Some imagers, e.g. WSClean, define the WCS in the fits files as -180<RA<180. This caused issues within the source association algorithms. A new simple fix has been applied so these images can be correctly processed.

Removal of MonetDB

MonetDB is no longer supported but was still present in code and documentation. All references to MonetDB have now been removed.

Fixing of errors

A number of outstanding errors have been fixed.

- Python
Published by AntoniaR over 4 years ago

tkp - 4.0

4.0

No changes since 4.0rc1.

Changes since 3.1.1

frequency band logic change

the band determination logic has changed. Before all bands where split into 1 MHz intervals and associated as such. With this release images are put in the same band if their bandwidths overlap.

We added an option to limit the bandwidth used for band association (#492). Limiting the bandwidth for an image is done by setting bandwidth_max in jobparams.cfg_ under the persistence section. E.g.:

[persistence] bandwidth_max = 0.0

Setting the value to 0.0 will use the bandwidth defined in the image headers, a non 0.0 value will override this value.

added streaming telescope support

The internals of TraP have been rewritten to support streaming AARTFAAC data (#483). There is now a new section in the job_params.cfg file with a mode setting. Setting this to batch will keep the old TraP behavior, but setting mode to streamwill enable switch TraP to streaming mode. TraP will connect to a network port and process images from the network until terminated.

The hosts and ports where to connect to is controlled with the hosts and ports settings:

[pipeline] mode = 'stream' hosts = 'struis.science.uva.nl,struis.science.uva.nl' ports = '6666,6667'

The batch mode should mostly be unaffected, only the order of actions has changed. TraP will process the full dataset now in chunks grouped by timestamp. The storing of images, quality checks and meta data extraction is now run together with the source extraction and association cycle, where before this was all done at the start of a TraP run. This makes it more similar to how we process streaming data and enabled other optimisations in the future (like keeping images in memory per timestamp group).

Removal of MongoDB image store

If you enable the copy_images setting in your pipeline.cfg file the images are now stored in the sql database (#534). This makes it much easier to manage the files, for example delete them. Also the images load faster in banana. This makes setting up and configuring MongoDB obsolete.

Add command line option to delete dataset

It is now possible to delete a dataset and its associated images (#533)::

``` $ trap-manage.py deldataset 5 -y

dataset 5 has been deleted! ```

Make TraP more resilient against faulty data

TraP often crashed on faulty image data. On popular request TraP will now try to continue, giving a warning. #522

Various other changes and bugfixes

• Fix Numpy 1.9+ compatibility #509
• TraP sourcefinder error on updated AARTFAAC images #505
• forced fits is not parallelised #526
• restructure logging, make less verbose. Also multiproc workers will log to stdout.
• fix multiprocess job cancelling problem (ctrl-c)

known issues

• Streaming mode gives a harmless error #536
• Alembic upgrade is not working yet #535

- Python
Published by gijzelaerr over 9 years ago

tkp - 4.0 release candidate 1

4.0 release candidate 1

Changes since 3.1.1

frequency band logic change

the band determination logic has changed. Before all bands where split into 1 MHz intervals and associated as such. With this release images are put in the same band if their bandwidths overlap.

We added an option to limit the bandwidth used for band association (#492). Limiting the bandwidth for an image is done by setting bandwidth_max in jobparams.cfg_ under the persistence section. E.g.:

[persistence] bandwidth_max = 0.0

Setting the value to 0.0 will use the bandwidth defined in the image headers, a non 0.0 value will override this value.

added streaming telescope support

The internals of TraP have been rewritten to support streaming AARTFAAC data (#483). There is now a new section in the job_params.cfg file with a mode setting. Setting this to batch will keep the old TraP behavior, but setting mode to streamwill enable switch TraP to streaming mode. TraP will connect to a network port and process images from the network until terminated.

The hosts and ports where to connect to is controlled with the hosts and ports settings:

[pipeline] mode = 'stream' hosts = 'struis.science.uva.nl,struis.science.uva.nl' ports = '6666,6667'

The batch mode should mostly be unaffected, only the order of actions has changed. TraP will process the full dataset now in chunks grouped by timestamp. The storing of images, quality checks and meta data extraction is now run together with the source extraction and association cycle, where before this was all done at the start of a TraP run. This makes it more similar to how we process streaming data and enabled other optimisations in the future (like keeping images in memory per timestamp group).

Removal of MongoDB image store

If you enable the copy_images setting in your pipeline.cfg file the images are now stored in the sql database (#534). This makes it much easier to manage the files, for example delete them. Also the images load faster in banana. This makes setting up and configuring MongoDB obsolete.

Add command line option to delete dataset

It is now possible to delete a dataset and its associated images (#533)::

``` $ trap-manage.py deldataset 5 -y

dataset 5 has been deleted! ```

Make TraP more resilient against faulty data

TraP often crashed on faulty image data. On popular request TraP will now try to continue, giving a warning. #522

Various other changes and bugfixes

• Fix Numpy 1.9+ compatibility #509
• TraP sourcefinder error on updated AARTFAAC images #505
• forced fits is not parallelised #526
• restructure logging, make less verbose. Also multiproc workers will log to stdout.
• fix multiprocess job cancelling problem (ctrl-c)

known issues

• Streaming mode gives a harmless error #536
• Alembic upgrade is not working yet #535

- Python
Published by gijzelaerr almost 10 years ago

tkp - R3.1.1 (2016-05-20)

Adds a 'generic' (i.e. not telescope-specific) quality check for flat images, which are clearly bad data since they contain no information (#507). Also makes some changes to the way image-rejection reasons are handled, closing #360 in the process.

- Python
Published by timstaley about 10 years ago

tkp - Release 3.1 (2016-03-29)

Adds an uncommon but potentially serious bug-fix, and some minor user-interface improvements.

User-interface changes

New boolean entry colorlog in pipeline.cfg (#502), this controls whether the console logging-output from trap-manage.py is colored according to message severity. E.g.:

[logging] colorlog = True

This is accompanied by some adjustments to the logging, we now output both INFO and DEBUG level logfiles (under the <jobdir>/logs/<run_datestamp> folders).

Bugfixes

• Catch some forced fitting bugs that could possibly have been encountered due to oversize 'skyregion' settings, or simply when reducing mosaic images with 'NaN' pixel regions. (#496)

Documentation

• Add 'features overview' section to docs.
• Document use (and units) of 'monitoring-coords' option to trap-manage.py. (#485)

- Python
Published by timstaley about 10 years ago

tkp -

Changes since 2.1

User-interface changes

New entry expiration in jobparams.cfg_ (#472):

[source_extraction] expiration = 10 ; number of forced fits performed after a blind fit

Features / Enhancements

• Added support for AARTFAAC format images (#444, #452).
• Expiration of forced-fit monitoring at locations with no recent blind detections (#472).
• Caching of additional variability metrics (varmetric table) to improve interactive queries via the 'Banana' web-interface (#469).

Refactoring / Infrastructure changes

• Started integrating SQLAlchemy as a future replacement for home-grown database-ORM code (#362).
• Migrate from PyFITS to astropy.io.fits (#355).
• Removed unused 'Celery' code (#433).

- Python
Published by gijzelaerr over 10 years ago

tkp - Release candidate for 3.0

R3.0rc (2015-12-14)

User-interface changes

New entry expiration in jobparams.cfg_ (#472):

[source_extraction] expiration = 10 ; number of forced fits performed after a blind fit

Features / Enhancements

• Added support for AARTFAAC format images (#444, #452).
• Expiration of forced-fit monitoring at locations with no recent blind detections (#472).
• Caching of additional variability metrics (varmetric table) to improve interactive queries via the 'Banana' web-interface (#469).

Refactoring / Infrastructure changes

• Started integrating SQLAlchemy as a future replacement for home-grown database-ORM code (#362).
• Migrate from PyFITS to astropy.io.fits (#355).
• Removed unused 'Celery' code (#433).

- Python
Published by gijzelaerr over 10 years ago