1. EGI-ACE Project
IDIA has been a partner on the European-Funded EGI-ACE project, which began at the start of 2021 and ended in mid-2023. Its mission was to implement the Compute Platform of the European Open Science Cloud, by delivering a secure federation of Cloud compute and storage facilities that are nationally procured and delivered in collaboration with providers of the EGI Federation, commercial providers, data providers and international research infrastructures of pan-European relevance. The platform provides IaaS, PaaS and SaaS services with the contribution of research communities, and is being gradually expanded into a Global European Open Science Cloud with major non-EU e-Infrastructures. IDIA’s contribution was in federating Ilifu resources within the EGI Federation, enabling it to be accessed by South Africa researchers and EGI users that had been added to appropriate Virtual Organizations (VOs), with the appropriate authorization roles. Toolkits were made available and deployable on EGI cloud resources.
2. ASTRON, Netherlands
Data transport systems and cloud-based pipeline processing collaboration In 2015, IDIA signed a MoU with fellow research institutions in the Netherlands, IBM and ASTRON to collaborate in a ground-breaking research project entitled: “Precursor Regional Science Data Centres for the SKA (SKA- RSDC)”. The South African-Dutch agreement on data science aims to establish national and regional data centres – in both countries – to tackle one of the most significant challenges presented by SKA: how to manage, process, and make accessible the immense amount of data the telescope will generate. The data centres will provide astronomers around the world with access to the large-scale data infrastructures and associated high performance computing (HPC) needed to make sense of the data.
3. Chinese Academy of Sciences, National Astronomical Observatories, China
Bilateral workshops organised since 2016 to address the challenge of big data in astronomy. The FAST telescope in China and the MeerKAT in South Africa, IDIA and the Chinese Academy of Sciences have joined forces to address big data science education in rural universities. The main objectives of the workshops are to: Strengthen cooperation on the data challenge of SKA. Mutual learning from the Chinese FAST and the South African MeerKAT projects. Cooperation on virtual observatory tools. Hold astroinformatics and astrostatistics summer schools or lectures and seminars. Scientific collaboration between radio astronomers in South Africa and China. Exchange ideas on how Big Data and the Virtual Observatory could be used for development, particularly to strengthen the interest of young people in mathematics, science and technology.
4. Academia Sinica Institute for Astronomy and Astrophysics, Taipei
Cloud-based visualisation and visual analytics of big radio astronomy data Because of the size and complexity of astronomical data, effective visualisation tools are key to scientific discoveries. IDIA is collaborating with a team at the Academia Sinica Institute of Astronomy and Astrophysics in Taipei to upscale the Cube Analysis and Rendering Tool for Astronomy (CARTA) – a platform for viewing astronomical data – to enable it to handle the data requirements of MeerKAT and the SKA. CARTA was originally designed to visualise multidimensional data sets that vary in size and volume. As it was designed for much smaller data sets, this presents a challenge that has been worked on since 2018. For this purpose, a CARTA Working Group, face-to-face meeting was organised by IDIA in 2018.
5. Joint Institute for VLBI in Europe (JIVE), Netherlands
Cloud-based pipelines for Big data VLBI based on Jupyter notebooks. This collaboration was established in 2017 to develop software enabling astronomers to make use of the Common Astronomy Software Application (CASA) within a Jupyter notebook environment, increasingly used by astronomers on the cloud. Specifically, this is for the processing of interferometry data.
6. National Radio Astronomy Observatory (NRAO), USA
Software algorithms and systems for high dynamic range imaging with big data from dish antenna arrays. Cloud-based visualisation and visual analytics of big radio astronomy data. The National Radio Astronomy Observatory (NRAO) and IDIA signed a MoU in 2016 to collaborate on algorithm and software development in support of forthcoming surveys on the Jansky Very Large Array (jVLA) and MeerKAT. These surveys are pathfinders for the scientific direction and technical challenges of even larger surveys with the SKA in the future. Due to high sensitivity, wide field-of-view, and relatively low frequencies, these surveys require high fidelity and, in some cases, high dynamic range imaging. This requires compute-intensive imaging algorithms. These surveys also generate Petabyte-scale data volumes. The jVLA and MeerKAT have a large overlap in the parameters of the telescope and the planned surveys. Both telescopes therefore also have high synergy in the required post-processing technical solutions. NRAO and IDIA are working together to explore the R&D challenges, and to build the data processing solutions within NRAO’s CASA software environment.
7. Inter-University Centre for Astronomy and Astrophysics (IUCAA), India
Pipeline development for MeerKAT and for the uGMRT. MALS stands for MeerKAT Large Survey collaboration The Inter-University Centre for Astronomy and Astrophysics and IDIA play a key role in large surveys with modern radio telescopes such as MeerKAT, uGMRT, the extended Giant Metre-Wave Radio Telescope, and the jVLA. IDIA established a formal cooperation agreement in 2018 to collaborate on the development of a CASA-based software data analysis pipeline and various other tools to deal with large datasets from surveys conducted with modern radio telescopes.
8. DARA Big Data, bilateral United Kingdom – South Africa programme
This forms part of the Development in Africa using Radio Astronomy (DARA) programme of the UK’s Newton Fund Under the title of “A UK-Africa Data Science Network: Capturing the SKA-Driven Data Transformation”, Prof. Russ Taylor from IDIA and Prof. Anna Scaife at the University of Manchester (UK) proposed a skills-development focused programme to the Newton Fund. The context of the project is to use the big data challenge of next generation scientific facilities to build capacity for dealing with large data sets and to use this skills development as a vehicle for the transfer of knowledge and innovation to other research domains. Three domains are identified in the project: Data Analytics, Data Visualisation, and Data Systems and tools. Since the original proposal, graduate bursaries have been funded for African students to study in the UK and Big Data Africa research schools have been held, with SARAO co-organising the schools with IDIA. The next such research school will take place in October 2019.
9. Canadian Initiative for Radio Astronomy Data Analysis (CIRADA), Canada
Cloud-based visualisation of big data, algorithm and pipeline development for polarisation In 2017, scientists in Canada, South Africa and the United States began a collaboration called the Canadian Initiative for Radio Astronomy Data Analysis (“CIRADA”) to develop infrastructure that will enable Canadian and international scientists to meet the data challenge of radio astronomy. In 2018, IDIA and CIRADA signed a formal letter of agreement. IDIA supports CIRADA goals of creating science-ready data products, specifically from polarised continuum observations from radio telescopes. Another area of support is in the creation of custom visualisation software solutions for big data.
10. Instituto de Astrofisica de Andalucia, IAA, Spain
Cloud-based collaboration technologies for big data and reproducible science In 2018, The Instituto de Astrofísica de Andalucía (IAA) and IDIA signed a MOU to bring together their research and development teams. The aim of the collaboration is to develop new e-science technologies and systems for big data processing and analytics in astronomy. One of the specific goals of the collaboration, entitled “Data-intensive and reproducible research in the age of the SKA” is to help astronomers build reproducible data processing and analysis methods and workflows.
11. SKA Science Data Processor, SKA Organisation, UK
In the age of SKA, turning signals from telescopes into science data that astronomers can use poses new science data processing challenges. Prof. Rob Simmonds from IDIA was involved in the Science Data Processor (SDP) consortium of the SKA which was led by Professor Paul Alexander from Cambridge University. The SDP passed its critical design review in January 2019. Prof. Simmonds led the team DELIV responsible for the design of data management and delivery to the globally distributed SKA regional science data centres and was a member of the core SDP architecture team and SDP management team. Dr. David Aikema and Ms. Adrianna Pínska from IDIA were also involved in the data delivery design. Prof. Taylor served as a board member of the SDP. The DELIV team included participants from the Canadian Astronomy Data Centre (CADC), ASTRON in the Netherlands, the Instituto de Astrofísica de Andalucía in Spain and Oxford University. The larger SDP architecture team included participants from Australia and other European centres. The component and connector (C&C) developed by this collaboration is one of the outputs of this work that will be deployed at SKA processing centres in Cape Town and in Perth and will drive the delivery of data to the SKA Regional Centres (SRCs) that will be implemented at sites located in at least 6 countries across 5 continents.
12. Istituto Nazionale di Astrofisica, INAF, Italy
Italy and South Africa have a bilateral agreement “RADIO SKY 2020 – Fostering the cooperation between Italy and South Africa through radio astronomy” The Italy-South Africa joint research scheme started a wide range of collaborations in 2018. Workshops, researcher mobility, knowledge transfer and human capital development are at the heart of the collaboration, which was launched in 2018 with a high-level “Italy – South Africa Radio Astronomy Workshop”. Representatives from the respective ministries, SARAO and the radio astronomy communities came together to establish the cooperation and kick-start science and engineering collaborations between the two countries.
13. International Centre for Radio Astronomy Research (ICRAR), Perth, Australia
Collaboration on the development of key infrastructure for the SKA Prof. Rob Simmonds is working with the Australian team based at ICRAR that has taken on the role of developing the interface documents that will define how the SKA Science Data Processor (SDP) centres interact with the SKA Regional Centres (SRCs). This is a continuation of previous work in developing the SKA SDP delivery architecture (see above). The ICRAR team that is doing this work is led by Markus Dolensky.
14. International Astronomical Union Office of Astronomy for Development (IAU-OAD), Cape Town, South Africa
Research in graduate industry skills. Big Data Skills development with the IAU OAD and DARA Big Data. There is some evidence that investment in astronomy leads to economic growth, but no body of literature to support this for university graduates. IDIA has obtained a small research grant from the IAU-OAD to investigate this. We are also collaborating with the IAU OAD and DARA Big Data programme on organising and running big data skills development events, using our research cloud.