IDIA “processMeerKAT” Pipelines v.1.1 release

by

The IDIA Pipelines team is pleased to announce V1.1 of the IDIA Pipeline. The package uses CASA/MPICASA to do the initial cross-calibration (1GC, a’priori) of MeerKAT data on the ilifu SLURM Cluster. This release implements the concurrent cross-calibration of spectral windows on the ilifu cluster and includes several tweaks to improve performance and stability. This release brings several scientific improvements to the pipeline, such as improved broadband Stokes I and polarisation calibration, improved flux scaling, etc.

V1.1 was updated on ilifu’s Master branch on 19 August 2020, available from /idia/software/pipelines/master/. The previous release will also be available for your continued usage, as necessary, from /idia/software/pipelines/master-v1.0/.

Please have a look at the updated documentation on https://idia-pipelines.github.io. Known issues are listed, and issues that are resolved in v1.1 are also indicated, including fixes issued for previous flux scale issues, broadband polarisation, calculation of antenna statistics and exit codes.

Please report any issues, comments or suggestions to us by logging an issue on our repo: https://github.com/idia-astro/pipelines/issues/. For any SLURM, runtime or system errors, please contact ilifu support at support@ilifu.ac.za. For details on how to access the SLURM cluster, please visit http://docs.ilifu.ac.za/#/getting_started/request_access.

What’s New

    • Spectral Window (SPW) splitting, where each separate SPW is processed independently and concurrently, providing a speed-up for large (TB) datasets, better polarisation calibration, and better flux scaling
    • Quick-look continuum cube, across all SPWs
    • Pre-processing during initial partition of MS, including pre-averaging of frequency channels, removal of cross-hand correlations up front for Stokes I processing, and removal of autocorrelations
    • If running in full Stokes mode, setjy now includes the polarisation models for 3C286 and 3C138 if they are present in the data.
    • Improved default parameters, including a smaller RFI mask that removes the persistent RFI in the ranges 933~960, 1163~1299, and 1524~1630 MHz
    • Improved interaction with SLURM, including dependencies, the exclusion of nodes, account and reservation parameters, and graceful termination of pipeline after errors

Image credit: SARAO