AUT University is part of a bid by Australian and New Zealand radio astronomers to host the world's biggest telescope comprising thousands of radio telescope antennas spread across thousands of kilometres. South Africa is the other contender to host what is known as the Square Kilometre Array (SKA) and the chosen location will be announced next year. Current computing technology won't cope with the massive amounts of data produced by this project so scientists are looking at new models of data management. AUT has been working with IBM on a solution.
The SKA will be so sensitive that it will be able to detect an airport radar on a planet 50 light years away. It will generate enough raw data to fill 15 million 64 GB iPods every day. The SKA dishes will produce 10 times the global internet traffic. The supercomputer that will handle this data will have to perform 1018 operations per second - equivalent to the number of stars in three million Milky Way galaxies. All of which goes to highlight the fact that astronomy these days is as much about computing as it is about the study of the cosmos.
The Australasian bid to host SKA, if successful, will see New Zealand dishes (like AUT's radio telescope in Warkworth) linked to radio telescopes across Australia to operate as a single gigantic instrument spanning 5,500 kilometres.
As well as taking part in successful experiments to link its radio telescope to its Australian peers for radio astronomy observations, AUT's Institute for Radio Astronomy and Space Research (IRASR) has been looking at the problem of data processing and storage.
In 2009 the IRASR received the internationally awarded IBM Shared University Research grant, the first time it has been awarded in New Zealand. For the past two years the IRASR team has been working with the IBM T.J. Watson Research Centre in New York to explore a new approach for data processing - stream computing.
Once the SKA is up and running, measuring the signals from 3000 antenna dishes over a period of hours will result in an enormous data set which will require a dynamic and rapid computing response.
"Conventionally people collect data from the environment, put it in a database and then mine the data," says IRASR researcher Mahmoud Mahmoud. "But the 21st century has seen an information overload. We are producing such large volumes of data that it's no longer feasible to store, so now we're looking at processing the data as it streams in. This is applicable to any situation where data is coming in that is immediate and time sensitive, for example financial analysis or video surveillance."
Working with IBM, the IRASR team explored using IBM InfoSphere Streams, a new data stream management system designed to ingest, filter, analyse and correlate enormous amounts of data streaming from any number of data sources. But in order to future-proof the system they needed InfoSphere Streams to run on commodity computer processors so that as the computing demands of SKA grow, more processors can be added as required.
"We wanted the system to be hardware agnostic," says Mahmoud. "In the past in astronomy you always had purpose-built hardware which calls for a lot of engineering. We wanted to be able to use off-the-shelf hardware."
The IRASR was able to successfully use InfoSphere Streams to run on multiple Cell BE processors - the same processor that is used in the Sony Playstation - as well as x86 processors.
"InfoSphere Streams does not support the Cell BE so we were very pleased to be able to get it to go across to another architecture and take advantages of its unique performance capabilities," says Mahmoud.
This work has culminated in development of a streaming spectrometer and correlator for radio astronomy with an eye on enormous data streams generated by future SKA. A review paper by Mahmoud and the IRASR team is in press as a chapter of the book on Data Provenance and Management to be published by Springer.
"It is a very fruitful collaboration," says Bruce Elmegreen from the IBM T.J.Watson Centre. "I have enjoyed watching the AUT researchers make such great progress. There is great potential in the acceleration of streaming data and they have become experts in this field."
Click here to find out more.
Surviving the data tsunami
AdvertisementAdvertise with NZME.