Releases: transientskp/tkp
Release 6.0
Release 6.0 release candidate
There are significant improvements to TraP throughout this release. These improvements lead to some speed improvements and solves a source association issue seen in R5.0. I recommend setting up a new project and new jobs to ensure that you have the correct pipeline.cfb and job_params.cfg files.
Major changes
- Upgraded to support Python 3 only. I recommend using a Conda environment running Python 3.11 to install this release.
- Upgraded database to postgresql 14. This means you also need to upgrade the database running on your system. N.B. I have not tried this yet so be careful - I do not know what happens to your old databases at this time. I will be testing this priori to this becoming an official release.
- Significant improvements and parallelisation of PySE.
The documentation will be updated prior to this becoming an official release.
If you test this release candidate, please let me know about your experiences and if there are any issues with the new code.
Release 5.0
Update to job_params.cfg
A new section for all quality checks was created. Generic quality control parameters in persistence and quality_lofar were moved to this new section.
Quality checks
The rms values of all fits images are checked to global minimum and maximum rms values.
The automatic rms based image quality control check was given its own independent sigma threshold.
[quality]
rms_rej_sigma = 3 ; threshold for rejecting images using rms histogram
The restoring beam parameter checks, initially developed for LOFAR images, now applied to all fits images.
Negative RA values
Some imagers, e.g. WSClean, define the WCS in the fits files as -180<RA<180. This caused issues within the source association algorithms. A new simple fix has been applied so these images can be correctly processed.
Removal of MonetDB
MonetDB is no longer supported but was still present in code and documentation. All references to MonetDB have now been removed.
Fixing of errors
A number of outstanding errors have been fixed.
Release 5.0 release candidate
Update to job_params.cfg
A new section for all quality checks was created. Generic quality control parameters in persistence and quality_lofar were moved to this new section.
Quality checks
The rms values of all fits images are checked to global minimum and maximum rms values.
The automatic rms based image quality control check was given its own independent sigma threshold.
[quality]
rms_rej_sigma = 3 ; threshold for rejecting images using rms histogram
The restoring beam parameter checks, initially developed for LOFAR images, now applied to all fits images.
Negative RA values
Some imagers, e.g. WSClean, define the WCS in the fits files as -180<RA<180. This caused issues within the source association algorithms. A new simple fix has been applied so these images can be correctly processed.
Removal of MonetDB
MonetDB is no longer supported but was still present in code and documentation. All references to MonetDB have now been removed.
Fixing of errors
A number of outstanding errors have been fixed.
4.0
4.0
No changes since 4.0rc1.
Changes since 3.1.1
frequency band logic change
the band determination logic has changed. Before all bands where split
into 1 MHz intervals and associated as such. With this release images
are put in the same band if their bandwidths overlap.
We added an option to limit the bandwidth used for band association
(#492). Limiting the bandwidth for an image is done by
setting bandwidth_max in job_params.cfg under the
persistence section. E.g.:
[persistence]
bandwidth_max = 0.0
Setting the value to 0.0 will use the bandwidth defined in the image
headers, a non 0.0 value will override this value.
added streaming telescope support
The internals of TraP have been rewritten to support streaming AARTFAAC
data (#483). There is now a new section in the job_params.cfg file
with a mode setting. Setting this to batch will keep the old TraP
behavior, but setting mode to streamwill enable switch TraP
to streaming mode. TraP will connect to a network port and process
images from the network until terminated.
The hosts and ports where to connect to is controlled with the hosts
and ports settings:
[pipeline]
mode = 'stream'
hosts = 'struis.science.uva.nl,struis.science.uva.nl'
ports = '6666,6667'
The batch mode should mostly be unaffected, only the order of actions
has changed. TraP will process the full dataset now in chunks grouped by
timestamp. The storing of images, quality checks and meta data
extraction is now run together with the source extraction and association
cycle, where before this was all done at the start of a TraP run.
This makes it more similar to how we process streaming data and enabled
other optimisations in the future (like keeping images in memory per
timestamp group).
Removal of MongoDB image store
If you enable the copy_images setting in your pipeline.cfg file
the images are now stored in the sql database (#534). This makes it
much easier to manage the files, for example delete them. Also the
images load faster in banana. This makes setting up and configuring
MongoDB obsolete.
Add command line option to delete dataset
It is now possible to delete a dataset and its associated images (#533)::
$ trap-manage.py deldataset 5 -y
dataset 5 has been deleted!
Make TraP more resilient against faulty data
TraP often crashed on faulty image data. On popular request TraP will
now try to continue, giving a warning. #522
Various other changes and bugfixes
- Fix Numpy 1.9+ compatibility #509
- TraP sourcefinder error on updated AARTFAAC images #505
- forced fits is not parallelised #526
- restructure logging, make less verbose. Also multiproc workers will
log to stdout. - fix multiprocess job cancelling problem (ctrl-c)
known issues
4.0 release candidate 1
4.0 release candidate 1
Changes since 3.1.1
frequency band logic change
the band determination logic has changed. Before all bands where split
into 1 MHz intervals and associated as such. With this release images
are put in the same band if their bandwidths overlap.
We added an option to limit the bandwidth used for band association
(#492). Limiting the bandwidth for an image is done by
setting bandwidth_max in job_params.cfg under the
persistence section. E.g.:
[persistence]
bandwidth_max = 0.0
Setting the value to 0.0 will use the bandwidth defined in the image
headers, a non 0.0 value will override this value.
added streaming telescope support
The internals of TraP have been rewritten to support streaming AARTFAAC
data (#483). There is now a new section in the job_params.cfg file
with a mode setting. Setting this to batch will keep the old TraP
behavior, but setting mode to streamwill enable switch TraP
to streaming mode. TraP will connect to a network port and process
images from the network until terminated.
The hosts and ports where to connect to is controlled with the hosts
and ports settings:
[pipeline]
mode = 'stream'
hosts = 'struis.science.uva.nl,struis.science.uva.nl'
ports = '6666,6667'
The batch mode should mostly be unaffected, only the order of actions
has changed. TraP will process the full dataset now in chunks grouped by
timestamp. The storing of images, quality checks and meta data
extraction is now run together with the source extraction and association
cycle, where before this was all done at the start of a TraP run.
This makes it more similar to how we process streaming data and enabled
other optimisations in the future (like keeping images in memory per
timestamp group).
Removal of MongoDB image store
If you enable the copy_images setting in your pipeline.cfg file
the images are now stored in the sql database (#534). This makes it
much easier to manage the files, for example delete them. Also the
images load faster in banana. This makes setting up and configuring
MongoDB obsolete.
Add command line option to delete dataset
It is now possible to delete a dataset and its associated images (#533)::
$ trap-manage.py deldataset 5 -y
dataset 5 has been deleted!
Make TraP more resilient against faulty data
TraP often crashed on faulty image data. On popular request TraP will
now try to continue, giving a warning. #522
Various other changes and bugfixes
- Fix Numpy 1.9+ compatibility #509
- TraP sourcefinder error on updated AARTFAAC images #505
- forced fits is not parallelised #526
- restructure logging, make less verbose. Also multiproc workers will
log to stdout. - fix multiprocess job cancelling problem (ctrl-c)
known issues
R3.1.1 (2016-05-20)
Release 3.1 (2016-03-29)
Adds an uncommon but potentially serious bug-fix, and
some minor user-interface improvements.
User-interface changes
New boolean entry colorlog in pipeline.cfg (#502),
this controls whether the console logging-output from trap-manage.py
is colored according to message severity. E.g.:
[logging]
colorlog = True
This is accompanied by some adjustments to the logging, we now output both
INFO and DEBUG level logfiles (under the <jobdir>/logs/<run_datestamp> folders).
Bugfixes
- Catch some forced fitting bugs that could possibly have been encountered
due to oversize 'skyregion' settings, or simply when reducing mosaic images
with 'NaN' pixel regions. (#496)
Documentation
- Add 'features overview'
section to docs. - Document use (and units) of 'monitoring-coords' option to trap-manage.py.
(#485)
r3.0
Changes since 2.1
User-interface changes
New entry expiration in job_params.cfg (#472):
[source_extraction]
expiration = 10 ; number of forced fits performed after a blind fit
Features / Enhancements
- Added support for AARTFAAC format images (#444, #452).
- Expiration of forced-fit monitoring at locations with no recent blind
detections (#472). - Caching of additional variability metrics (
varmetrictable) to improve
interactive queries via the 'Banana' web-interface (#469).
Refactoring / Infrastructure changes
Release candidate for 3.0
R3.0rc (2015-12-14)
User-interface changes
New entry expiration in job_params.cfg (#472):
[source_extraction]
expiration = 10 ; number of forced fits performed after a blind fit
Features / Enhancements
- Added support for AARTFAAC format images (#444, #452).
- Expiration of forced-fit monitoring at locations with no recent blind
detections (#472). - Caching of additional variability metrics (
varmetrictable) to improve
interactive queries via the 'Banana' web-interface (#469).