Blimpy - Fix the bl_scrunch utility

The bl_scrunch utility was failing to update the output file header nchans field (number of fine channel frequencies). This field is now calculated based on the -f parameter.

Also, removed the obsolete match_fils.py and all references to it.

Scientific Software - Peer-reviewed - Python
Published by texadactyl over 3 years ago

Blimpy - Enhance dsamp to write SIGPROC (.fil) output as well as .h5

Scientific Software - Peer-reviewed - Python
Published by texadactyl over 3 years ago

Blimpy - Bug fix to release 2.1.1

Release 2.1.1 had missing fields in the constructed Waterfall "container". This resulted in issue #270.

Scientific Software - Peer-reviewed - Python
Published by texadactyl over 3 years ago

Blimpy - Expanded Waterfall Capability

In this past, blimpy's Waterfall class was only entertaining input from a Filterbank file (.h5 or .fil). With this release, a data array + a header dictionary may be passed instead of a file path. Note that these 2 input methodologies are mutually exclusive.

Scientific Software - Peer-reviewed - Python
Published by texadactyl over 3 years ago

Blimpy - Introduce Filterbank file downsampling capability (dsamp)

The operator will supply an input path, output path, and a groupsize used for the purpose of summing. For every every groupsize time samples of the input data, the utility will sum the data element values into one total based on the polarisation index and fine-requency channel index. This summing is performed for each polarisation index and each channel index within a given polarisation.

E.g. if the group size = 3, then for polarisation index 1 (relative to 0) and channel index 42 (relative to 0), the sum for time samples 21 through 23 (relative to 0) for single element outdata[7, 1, 42] is given by: indata[21, 1, 42] + indata[22, 1, 42] + indata[23, 1, 42]

The number of output samples = input array time dimension integer-divided by group_size. If the remainder of that division > 0, then the excess samples from the input array are dropped.

E.g. if the group size = 3 and the input data has 62 time samples, then the last 2 input samples are discarded.

The summing group size must be >= 2 and < input data time dimension.

Note that the tsamp header field in the output is set = the group size multiplied by the input tsamp value.

Scientific Software - Peer-reviewed - Python
Published by texadactyl over 3 years ago

Blimpy - Fix misspelling of kurtosis in watutil.py

See issue #266.

Scientific Software - Peer-reviewed - Python
Published by texadactyl over 3 years ago

Blimpy - Cleanup Waterfall, blimpy plotting, and regression tests

The Waterfall object ccurrently can only support the input of a Filterbank file. Therefore, it now diagnoses the following as an error: - Missing input Filterbank path. - Presence of a data_array. - Presence of a header dict object (metadata).

If, in the future, issue #264 is resolved in terms of requirements, then the following might be the metadata/data input options: - Input Filterbank path for a file containing both metadata and a data array. - A metadata dict object + a data array matrix, both RAM-resident.

Scientific Software - Peer-reviewed - Python
Published by texadactyl almost 4 years ago

Blimpy - Fixed blimpy to show plots when the display supports it.

Fixed blimpy to show plots when the display supports it. See issue #263.

Scientific Software - Peer-reviewed - Python
Published by texadactyl almost 4 years ago

Blimpy - Help turbo_seti plot_event.py performance

In hdfreader.py examineh5 function, the amount of data matrix loaded was reduced.

Scientific Software - Peer-reviewed - Python
Published by texadactyl almost 4 years ago

Blimpy - v2.0.37 Enhanced h5diag utility

Sample output:

VERSION attribute: 2.0 header: {'DIMENSIONLABELS': array([b'time', b'feedid', b'frequency'], dtype=object), 'azstart': 0.0, 'datatype': 1, 'fch1': 6688.96484375, 'foff': -0.00286102294921875, 'ibeam': -1, 'machineid': 20, 'nbeams': 1, 'nbits': 32, 'nchans': 65536, 'nfpc': 1024, 'nifs': 1, 'rawdatafile': 'blc17guppi5799149318DIAGPSRJ0332+54340008.0000.raw', 'sourcename': 'DIAGPSRJ0332+5434', 'srcdej': , 'srcraj': , 'telescopeid': 6, 'tsamp': 1.073741823999999, 'tstart': 57991.570810185185, 'za_start': 0.0} data shape: (56, 1, 65536) Number of fine channels: 65536 NFPC: 1024 Number of coarse channels: 64 Rawspec version: 3.1.1 Librawspec version: 3.1.1 cuFFT version: 10.2.1.245 HDF5 library version: 1.8.16 Bitshuffle: ENABLED

No errors detected

Scientific Software - Peer-reviewed - Python
Published by texadactyl about 4 years ago

Blimpy -

In fil2h5, h52fil, dice, and bl_scrunch, it is possible to mangle the output file name if the input file path has an embedded "fil". In essence, the embedded "fil" will be inadvertently changed to "h5". E.g. "abc.filter.def.fil" --> "abc.h5ter.def.h5" (should be "abc.filter.def.h5"). This was caused by misuse of the Python string replace function.

Solution:

• Implemented utils.changetheext(path, oldext, newext) will change only the final oldext in the path or append the newpath if there is no old_ext present.
• fil2h5, h52fil, dice, and bl_scrunch call the new function.

Since the function is generic, it can also be leveraged by other projects such as turbo_seti (probably needed!).

Scientific Software - Peer-reviewed - Python
Published by texadactyl about 4 years ago

Blimpy - Enhance h5diag for FBH5 version 2.0

For a valid FBH5 version 1.0 file, h5diag reports like this: h5diag: Begin h5diag: VERSION attribute: 1.0 h5diag: header: {'DIMENSION_LABELS': array([b'frequency', b'feed_id', b'time'], dtype=object), 'az_start': 0.0, 'data_type': 1, 'fch1': 8421.386717353016, 'foff': -2.7939677238464355e-06, 'ibeam': 1, 'machine_id': 20, 'nbeams': 1, 'nbits': 32, 'nchans': 1048576, 'nifs': 1, 'rawdatafile': 'guppi_57650_67573_Voyager1_0002.0000.raw', 'source_name': 'Voyager1', 'src_dej': <Angle 12.183 deg>, 'src_raj': <Angle 17.16777333 hourangle>, 'telescope_id': 6, 'tsamp': 18.253611008, 'tstart': 57650.78209490741, 'za_start': 0.0} h5diag: data shape: (16, 1, 1048576) h5diag: No errors detected

For a valid FBH5 version 2.0 file (created by rawspec 3.0 or later, producing an .h5 file directly), h5diag reports like this: h5diag: Begin h5diag: VERSION attribute: 2.0 h5diag: header: {'DIMENSION_LABELS': array([b'time', b'feed_id', b'frequency'], dtype=object), 'az_start': 0.0, 'data_type': 1, 'fch1': 7438.96484375, 'foff': -2.7939677238464355e-06, 'ibeam': -1, 'machine_id': 20, 'nbeams': 1, 'nbits': 32, 'nchans': 67108864, 'nfpc': 1048576, 'nifs': 1, 'rawdatafile': 'blc13_guppi_57991_49836_DIAG_FRB121102_0010.0000.raw', 'source_name': 'DIAG_FRB121102', 'src_dej': <Angle 33.1479 deg>, 'src_raj': <Angle 5.53295333 hourangle>, 'telescope_id': 6, 'tsamp': 18.253611007999982, 'tstart': 57991.57680555555, 'za_start': 0.0} h5diag: data shape: (3, 1, 67108864) Rawspec version: 2.5.0+91@gd699853 Librawspec version: 2.5.0+91@gd699853 cuFFT version: 10.2.1.245 HDF5 version: 1.8.16 Bitshuffle: ENABLED h5diag: No errors detected

Scientific Software - Peer-reviewed - Python
Published by texadactyl about 4 years ago

Blimpy - New utility: srcname

Created a tool for patching the header source_name in an HDF5 file: srcname.

Scientific Software - Peer-reviewed - Python
Published by texadactyl over 4 years ago

Blimpy - New blimpy utility: peek

Scientific Software - Peer-reviewed - Python
Published by texadactyl over 4 years ago

Blimpy - Fix issue #243

Scientific Software - Peer-reviewed - Python
Published by texadactyl over 4 years ago

Blimpy - Improve clarity in file writing messages

Impacted source files: - io/filwriter.py - io/hdfwriter.py - fil2h5.py

Scientific Software - Peer-reviewed - Python
Published by texadactyl over 4 years ago

Blimpy - Fix issue #238 and a surpise bug in blimpy/utils.py

• There were miscellaneous bugs in stax.py w.r.t. frequency range specifications and large files.
• blimpy/util.py rebin function sometimes crashed due to unexpected dimension input.

Scientific Software - Peer-reviewed - Python
Published by texadactyl over 4 years ago

Blimpy - Fix issues with stix and stax

1. Recovered lost executable stix.
2. The entry point name for `stax was corrected to be consistent with the rest of blimpy.

Scientific Software - Peer-reviewed - Python
Published by texadactyl over 4 years ago

Blimpy - Fixed a lot of mishaps in the regression tests

Scientific Software - Peer-reviewed - Python
Published by texadactyl over 4 years ago

Blimpy - Centralise regression test requirements

Blimpy testing needs setigen for regression testing. Setigen needs blimpy to save data in Filterbank or HDF5 files.

So, when the blimpy dependency of setigen is installed and blimpy is not yet installed, testing proceeds with the previous release level from pypi.

Solution: Take setigen out of requirements.txt and put it and all the other test requirements in requirements_test.txt (new file).

Impacted files: * .github/workflows/python_tests.yml * Dockerfile

Scientific Software - Peer-reviewed - Python
Published by texadactyl over 4 years ago

Blimpy - Try to remove HDF5 files that cannot be fully written due to an exception

One amended source file: blimpy/io/hdf_writer.py

Scientific Software - Peer-reviewed - Python
Published by texadactyl over 4 years ago

Blimpy - Clarify HDF5 b-tree errors

Catch corrupt B-tree in an HDF5 file with a clearer message. The corrupt files are caused by crashes during HDF5 file creation.

Scientific Software - Peer-reviewed - Python
Published by texadactyl over 4 years ago

Blimpy - A betterfix to issue #226

Scientific Software - Peer-reviewed - Python
Published by texadactyl over 4 years ago

Blimpy - Diagnose false and broken HDF5 files

The io/hdf_reader.py was updated and a new stand-alone utility, h5diag, was introduced.

Scientific Software - Peer-reviewed - Python
Published by texadactyl over 4 years ago

Blimpy - New signal_processing source file, "dedoppler.py"

This is the first in a planned series of signal processing functions for blimpy. In this release: * blimpy/signalprocessing/dedoppler.py - first dedoppler function * tests/testdedoppler.py - test for dedoppler.py and usage sample

Scientific Software - Peer-reviewed - Python
Published by texadactyl over 4 years ago

Blimpy - New utility, stix

LOFAR folks have some really huge Filterbank files E.g. 140 GB. Today's blimpy cannot plot them because it tries to get the whole data matrix into memory first.

New approach: Slice them into manageable sized sticks by fine-channel frequency subrange and plot each one separately. Optionally, concatenate all of the generated PNG files horizontally or vertically into one PNG.

New utility function: stix ``` stix -h usage: stix [-h] [--plotdir PLOTDIR] [--sourcename SOURCENAME] [--stitch {n,v,h}] [--dpi DPI] [--width WIDTH] [--height HEIGHT] inputfile chunkcount

Make waterfall plots from a single huge file, viewed from top to bottom.

positional arguments: inputfile List of files to plot chunkcount Count of same-sized frequency chunks

optional arguments: -h, --help show this help message and exit --plotdir PLOTDIR, -p PLOTDIR Directory to receive the plot (.png). Default: current directory. --sourcename SOURCENAME, -n SOURCENAME Source name to override the header field. Default: None. --stitch {n,v,h}, -s {n,v,h} Stitch files: n(no stitching), v(vertical), or h(horizontal). Default: n. --dpi DPI, -d DPI Single image file dots per inch. Default: 200. --width WIDTH, -w WIDTH Single image file image width in inches. Default: 10. --height HEIGHT, -t HEIGHT Single image file image height in inches. Default: 8. ```

Scientific Software - Peer-reviewed - Python
Published by texadactyl over 4 years ago

Blimpy - Get fil2h5 to support very large filterbank files

Fixed issue #218

Scientific Software - Peer-reviewed - Python
Published by texadactyl over 4 years ago

Blimpy - Fix issue #212

Waterfall.blank_dc() caused numpy "RuntimeWarning: Mean of empty slice" when the ratio between the number of fine channels to the number of coarse channels was < 5. This is a boundary case.

Scientific Software - Peer-reviewed - Python
Published by texadactyl over 4 years ago

Blimpy - Release 2.0.17.1 crashed [?}

Release 2.0.17.1: Undetected editing mistake in blimpy/init.py led to a missing value for version detected in turbo_seti. Blimpy operations are unaffected.

For some unknown reason, release 2.0.17.1 crashed. So, release 2.0.17.2 hopes to rectify that.

Scientific Software - Peer-reviewed - Python
Published by texadactyl over 4 years ago

Blimpy - Emergency release to 2.0.17

Undetected editing mistake in blimpy/init.py led to a missing value for version detected in turbo_seti. Blimpy operations are unaffected.

Scientific Software - Peer-reviewed - Python
Published by texadactyl over 4 years ago

Blimpy - Expand testing and improve diagnostics

Scientific Software - Peer-reviewed - Python
Published by texadactyl over 4 years ago

Blimpy - Catching up to version 2.0.15

Scientific Software - Peer-reviewed - Python
Published by texadactyl over 4 years ago

Blimpy - JOSS release

Changelog:

• Added CONTRIBUTIONS.md
• Improved docstrings
• Fixed Travis-CI docker issues

Scientific Software - Peer-reviewed - Python
Published by telegraphic over 6 years ago

Blimpy -

Scientific Software - Peer-reviewed - Python
Published by gijzelaerr about 7 years ago

Blimpy -

Scientific Software - Peer-reviewed - Python
Published by gijzelaerr about 7 years ago