How Vera Rubin Telescope Scientists Will Deal With 60 Million Billion Bytes of Imagery

Not so long ago, astronomers would spend one night traveling a telescope, making careful observations of one or a few points of light.

Based on these few observations, they would extrapolate large generalizations on the universe.

“It was all people could really do at the time, because it was difficult to collect data,” said Leanne Guy, data management scientist from the New Vera C. Rubin Observatory.

Rubin, located in Chile and funded by the American Department of Energy and the National Science Foundation, will flood astronomers with data.

Each image taken by Rubin’s camera consists of 3.2 billion pixels which can contain asteroids, dwarf planets, supernovas and unattained galaxies. And each pixel records one of the 65,536 shades of gray. This represents 6.4 billion bytes of information on a single image. Ten of these images would contain about as much data as all the words that the New York Times published on paper during its 173 years of history. Rubin will capture around 1,000 images each night.

As the data of each image is quickly mixed with the computers of the observatory, the telescope will pivote towards the next sky patch, taking a photo every 40 seconds.

It will do it again and almost every night for a decade.

The final statement will total around 60 million billion bytes of image data. It is a “6” followed by 16 zeros: 60,000,000,000,000,000.

Astronomy follows the path of scientific fields such as biology, which is today flooded in DNA sequences and particle physics, in which scientists must browse torrents of particle collisions to unravel indications of something new.

“We produce a lot of data for everyone,” said William O’Mullane, associate director of data management of the observatory. “So, this idea of ​​coming to the telescope and making your observation does not exist, right? Your observation has already been made. You just need to find it.”

Astronomers will be able to do their research at any time, anywhere, relying on high -speed networks, cloud computing and artificial intelligence algorithms to eliminate discoveries.

All this data must be stored and processed.

To do this, Dr. O’Mullane supervised the construction of an ultra -modern data center in Rubin with enough storage to keep a month of images in the event of a network disturbance.

Maintaining nearly 60 miles of fiber optic cables which connect the observatory to the city of Serena, Chile, can be difficult. People have stolen equipment. A fire on the road and a truck hitting a post caused breakdowns. Dr. O’Mullane said that someone used a cable for shooting.

When the data is circulating, it is sent to the National Slac Accelerator Laboratory, a research center for the Department of Energy in Menlo Park, California, for calculations that go beyond the initial analysis to the observatory.

Although Rubin takes a thousand images per night, this is not what will be sent to the world at the beginning. On the contrary, SLAC computers will create small snapshots of what has changed compared to what the telescope has seen previously.

For each new image, obvious imperfections, such as satellite sequences and spots generated by cosmic rays hitting the camera sensors, will be erased. “We are trying to filter non -astronomical garbage,” said Dr. O’Mullane.

Then, the software will compare the scene with a model that combines at least three previous observations of the same part of the sky.

When the model is removed from the last image, everything that is unchanged disappears. What remains are features that have changed. These include explosive stars called supernovas, variable stars that have brightened up or attenuated and asteroids that pass.

A single image will contain around 10,000 highlighted changes. An alert will be generated for each change – some 10 million alerts per night.

Is it like an astronomical version of “Where’s Waldo?”

To classify the objects identified outside the solar system, Rubin turns to nine external organizations called data brokers. These automated software systems will carry out an additional analysis, will withdraw interest data for individual astronomers and will identify intriguing events which justify monitoring observations by other telescopes.

There are differences in the orientation and approach of each data broker.

“It is best to send this to a global community of scientists with a lot of different skills and expertise to provide their knowledge,” said Dr. Guy.

A data broker named Antares, created by the National Research Laboratory on Infrared Optical Astronomy of the National Science Foundation, or Noillab, will direct alerts through 20 general filters to withdraw changes of interest, including certain supernovas.

His analysis is flexible. Astronomers will be able to write their own filters to find only the events they wish to study.

“We add contextual information from existing astronomical catalogs,” said Tom Matheson, who heads the Antares team. “When an alert comes into play, we say:” Does one of these catalogs have information on this object? ” And then we incorporate this into the alert so that people can find out more. »»

A Chilean data broker called Alerce – Automatic learning for the rapid classification of events – takes what could be considered a simpler and wider approach: sort all non -system alerts in 22 buckets. These include different supernovas flavors as well as radiation scratches from supermassive black holes, young stars and white dwarfs.

“They will be well organized by specialists in all areas,” said Francisco Förster, director of the Millennium Institute of Astrophysics in Chile and Principal Investigator of Alerce.

Alerce does not provide flexible data analysis like Antares, but it uses several types of classification techniques.

Two of these techniques use conventional automatic learning methods, which classify events based on preselected criteria. (This is equivalent to defining Waldo as a cartoon human with brown hair, wearing a striped shirt, glasses and a hat.)

Other techniques are based on networks of neural and other modern methods of in -depth learning. These draw raw data and independently invent their own criteria to identify different cosmic phenomena. (Imagine a computer understanding that Waldo is also often seen by holding a walking stick.)

Several of the data brokers, including Alerce, have tested their systems using the transitional installation of Zwicky near San Diego, which uses a smaller telescope.

A surprise, said Dr. Förster is that for years, the supposedly more sophisticated deep learning models failed when applied to Zwicky’s real -time data. But that could have been the result of limited training data.

“Everything indicates that in -depth learning should gain this in the future because you get more data,” said Dr. Förster.

Michael Wood-Vasey, professor of astronomy at the University of Pittsburgh, decided to create a data broker because he thought that large technological companies like Google had already resolved similar challenges.

“I said to myself, wait a minute, we have YouTube and content distribution networks for all big things,” said Dr. Wood-Vasey. “We should not technically try to reinvent this.”

He has teamed up with Ross Thomson, who works on high-performance computer projects at Google, to use the company’s cloud computing platform.

“Missing of brand consultants or anything, we just called it the Pitt-Google broker,” said Dr. Wood-Vasey.

Some other data brokers focus on specific data slices. One will compile information on asteroids and other small objects that take place in the solar system, calculating properties such as color and rotation rate. Another will follow the behavior of variable stars.

Beyond alerts, Rubin software will combine images for a more detailed analysis.

For each image, the telescope uses one of the six filters, which range from ultraviolet to infrared wavelengths. The filter changes the view, a bit like a pair of sunglasses. Three images taken through different filters can be combined in a color image.

The images can also be added, essentially a longer exposure to make weaker objects visible.

Complete images will be made public two years after taking them. Until then, their use has owned the scientists of the United States and Chile and other contributors to the project. Once a year, the Rubin project, with the help of an additional calculation power in France and in the United Kingdom, will retreat all the images and generate a more detailed catalog of its observations. This catalog also has a two -year owner period.

“We are going to withdraw all the data taken to date and combine it to extract as much scientific information as possible,” said Yusra Alsayyad, who oversees image processing at Rubin. “Because we provide calibrated images and measurements of all sources, this will increase the data of a factor of 10.”

The final data press release at the end of the 10-year survey, she said, could reach 500 million billion bytes.