Cambridge Astronomical Survey Unit

Supporting Worldwide Research
Today’s wide-field digital surveys produce massive amounts of data. The Cambridge Astronomical Survey Unit (CASU) automatically processes and archives this legacy data, not only to enable its own members to carry out important research, but to support the work of researchers worldwide. Astronomers in many countries, particularly the former Soviet bloc, have limited access to telescopic facilities of their own and regularly request data from Cambridge University.

Previously, searching through archive material to find the requested information could have taken two or three weeks. Now that CASU is using Sybase IQ, the results are delivered in minutes. Sybase IQ condenses the enormous quantities of data while it is being downloaded and stores it in a fraction of the space. Its reliability has also cut down on the amount of time needed for IT maintenance.

Managing a Massive Archive
Within the Institute, a wide range of theoretical problems are studied, including the evolution of the universe and the formation of galaxies and stars. Observational work relies on several large telescopes, in Cambridge itself, abroad and in space.

The CASU unit has 11 group members with diverse research interests. In addition, it manages the massive archive of astronomical data and the unit’s data processing activities.

“No one actually sits and stares through a telescope anymore,” said Dr. Jim Lewis, a senior research associate, who oversees the data processing at the unit. “All the equipment has been digital for some time, which makes it easier to preserve all the data we collect for posterity.”

Lewis added, “The range of data generated every night is extreme, depending on the style of observation. If it’s something faint and you have to expose for an hour, then you may generate only 100 megabytes. But in other cases, when you cannot expose for more than 10 or 20 seconds, it could be a terabyte a night.”

The Volume of Data
In the early days of digitization, people began creating their own electronic databases to share their observations, and the rapid adoption of the Internet enabled astronomers to view and share data online. But the existing databases simply couldn’t cope with the volume of the data being collected and soon began to buckle under the weight.

“I figured that we needed to look at what major organizations were doing to solve their data storage and data mining problems—which were far worse than ours,” Lewis said. “We started to look for a commercial relational database that could store data in a compressed, binary way—as telescopes do—and I was recommended Sybase IQ.”

Sybase IQ data management software condenses data by 80 percent, making enormous savings on storage space. Because of the way it stores and retrieves data, complex and ad hoc searches—equally important in the worlds of business and academia—are straightforward and quick.

Hundreds of terabytes of data from multiple sources is consolidated in real time, so information is always up to date.

A Versatile Solution
Sybase is scalable and will run on any hardware platform, making it a very versatile solution. At Cambridge, it is now running on Sun Solaris servers as well as Linux, with the initial implementation taking Lewis only a couple of weeks.
 
“There’s no doubt that the amount of time I used to spend maintaining the system has been reduced,” he said. “Previously, if something had gone wrong with one of the original databases, I would have had to investigate and the archive would have been down for a couple of weeks—which was not good.”

No More Frustration
As the Sybase platform is so popular, developers have written many useful middleware products to layer on top. One of these enabled Web-based forms to be designed for astronomers’ queries.

Previously, people wanting information would have had to read a huge help file in order to construct the query in the right language—usually SQL or similar. Researchers often got it wrong the first time around and would have to start again—therefore wasting a lot of time.

“Now, with these Web forms, people just point at the area of sky or the color filter they are interested in, and push a button,” Lewis said. “We knew we had got it right when the number of requests went up by a factor of three, yet the amount of data requested only went up by a factor of two. This is because people can be more precise about what they want, rather than asking for the whole night’s data.

“Previously, when an archivist responded to a query, he would have to load each tape individually and visually check through it. It could take two or three weeks to get the data someone had requested and send it to them. When we first went online, we cut this down to an hour. Now that everything is online, it can take a matter of minutes. That’s a big saving.”

Lewis added, “We’re dealing with a worldwide audience and if we had to deal with requests manually and on a case-by-case basis, I just couldn’t do it. But Sybase IQ makes it possible.”
Today someone working in Australia, for instance, can look at the unit’s data catalog of observations in the Sybase IQ database, covering each individual telescoping facility that CASU works with, and scroll down to see what interests them. They put in a request, and the unit downloads it and puts it online for the researcher to pick up.

“From the feedback we have received from our many users, it’s clear they appreciate the much faster and much more accurate service we can provide,” Lewis said.