Microsoft and the Papal State present a digital replica of St. Peter's Basilica that shows, in an immersive virtual tour, details and places of the emblematic church that are impossible to see in person

 

FRANCISCO DOMÉNECH Rome - NOV 16, 2024 -

 

In a city of Rome that is still immersed in preparations for the Jubilee of 2025 , the works are complicating traffic more than usual in the historic center, where scaffolding still overshadows visits to emblematic places such as the Trevi Fountain or Piazza Navona. Meanwhile, the Vatican is finalizing an unprecedented facelift for the already gleaming epicenter of these celebrations, St. Peter's Basilica. It is a technological restoration, which has used generative artificial intelligence to create a digital replica of the basilica for the first time. And with it, the Papal State hopes to offer "a new spiritual experience," in the words of Father Enzo Fortunato, who has participated in the project from the Fabbrica , the papal institution responsible for the conservation of the monumental building.

Microsoft is the technological partner of this initiative, in which its president, Brad Smith, has been personally involved: “This 3D model, of great realism and very high resolution, is a digital twin; something that we are much more accustomed to seeing in an industrial factory, where it is used to monitor operations and determine if there is something that may need repair. Here, however, we have an extraordinary opportunity to use a digital twin not to manage its operation, but to expand human knowledge,” explained the director of the American multinational last Monday, during the official presentation of the project together with Cardinal Mauro Gambetti, archpriest of St. Peter's Basilica. EL PAÍS has been invited to this presentation by Microsoft

With this philosophy in mind, pure technology took the first step. The French startup Iconem used two drones, controlled by artificial intelligence and equipped with cameras and lasers, to take more than 400,000 images using advanced photogrammetry techniques. It took three weeks of work, taking advantage of the hours of the day without visitors, to scan the entire basilica inside and out, recording every last detail. But the result was not just photos, but laser sensors ( LiDAR ) that measured the distance to each object and relief that was being photographed: more than 9 billion points were obtained, with which a first three-dimensional map was drawn, both of the basilica and of all its contents.

The total data exceeded 20 petabytes of information, which, as Smith points out, “would fill 5 million DVDs, which stacked together would form a tower 6,000 metres high.” The president of Microsoft explained to EL PAÍS that it would have been impossible to combine all that information – that is, to paste all those photos onto that map of countless points, to build an ultra-precise and realistic 3D model – without using the most advanced generative AI techniques, “which were not available just two or three years ago .”

Impossible without AI?

Juan Lavista, vice president and head of data science at Microsoft, recalls that “20 years ago, these types of processes for building 3D models were done by hand. But for a project of this size and complexity, I believe that it would not be possible or would take many years to complete without artificial intelligence.” The AI developed by the research laboratory that Lavista directs within the technology giant has been used not only to build the digital twin but to make it manageable and, despite the huge amount of information with which it is built, the 3D model can be rendered and moved with agility. Only in this way, with this last step, can the idea of visualizing the basilica from any point and angle be put into practice.

For Father Paolo Benanti, the question of whether this project would be possible without AI has a more philosophical answer: “Could Van Gogh have painted his art if he had not done so with synthetic colours, which were not available to previous artists such as Rubens? If the tools used condition a work, it is clear that without artificial intelligence the 3D model would simply be different: this is not a graphic representation, it is a representation generated with AI,” explains this Franciscan monk, who is an advisor to Pope Francis on the ethical use of technology and has collaborated on the project. Benanti clarifies that AI is essential to be able to simulate with this replica how the basilica would look under any light or weather conditions.

v

This digital twin is the heart of a new exhibition with which the Vatican will enrich visits to St. Peter's Basilica during the Jubilee of 2025. Just weeks before its inauguration, EL PAÍS has had access to this exhibition space — called Petros ení (Peter is here, in Greek) — which is located between two octagonal rooms inside the basilica that were not in use. Its main attraction is an immersive virtual visit. A giant, curved screen projects an ultra-high-definition video that runs through the digital twin: from the apostle's tomb, in the underground rooms, to the top of the dome, which was designed by Michelangelo and remains the highest interior dome — 118 meters — ever built. The video tour takes you to multiple places in the basilica that are impossible to see in person and zooms in to reveal details that are invisible when walking through the monumental building. Thus, the virtual visitor can see with his own eyes what the guides tell us during their visits to the basilica: none of what appear to be paintings are, they are all mosaics.

Lights and shadows of an immersive visit

The president of Microsoft describes it to this newspaper as a religious experience: “There is a moment when you start with your feet on the ground and look up at the top of the dome. It is so high that you cannot see what is there, but in this virtual space you can go up in the air and then you realize that what is there is an extraordinary mosaic and you are close enough to see the face of God in it; and then, you can look down from there and see the basilica as God sees it,” Brad Smith recounts. This is what those who built and decorated the great dome saw, completed in 1590, decades after Michelangelo's death. Smith points out that, literally, centuries have passed without any person having the opportunity to have that view, “which today we can contemplate using the power of AI.” Instead, Mauro Gambetti uses a scientific metaphor to illustrate the experience and the benefits of a digital twin built with artificial intelligence: “It is like looking at a starry sky on a summer night: the new tools act like a telescope or a spaceship, to see it better or from another perspective,” says the cardinal.

Enthusiastic about these achievements, the project leaders wanted to take on another challenge: to bring this immersive experience to any place in the world, to people who may not be able to travel to the Vatican but do have access to a computer, mobile phone or tablet. The result can already be seen on a new website, virtual.basilicasanpietro.va : there, the section 'Explore St. Peter's Basilica' gives access to an interactive web viewer of the digital twin. In its initial version, published this week, it has an impractical interface, which makes navigation through the basilica difficult and confusing (it is easy to get lost in unreal spaces between the walls). The loading is slow, even with a broadband connection, and the level of detail is much lower than in the videos of the physical exhibition: instead of perceiving each tile of the mosaics, what is seen are artefacts in the form of lines that blur these works of art.

Microsoft and the Vatican have also announced that in the coming months they will also bring a digital replica of the basilica to the educational space of the video game Minecraft , as a sign of an effort to connect with the new generations. Their plans also include the possibility of creating a version for virtual and augmented reality glasses; and looking to the future, a greater aspiration: “This 3D model will last forever,” says Juan Lavista, head of the Microsoft AI for Good laboratory , who highlights that his team was able to develop an AI tool that detects each of the missing pieces in the basilica’s mosaics, as well as the cracks in the walls, columns and vaults. This allowed them to create a detailed map of the structural vulnerabilities, which Microsoft and Iconem have made available to the Vatican to guide future interventions for the maintenance and restoration of this historical, artistic and religious complex.

“It was the first time we tackled this problem of detecting missing pieces and cracks, we didn’t know if we were going to be able to solve it. And now that we are sure, we are going to use it in other projects,” says Lavista, who also promises that if they prove that this utility can be generalized to other monuments, his company will release the AI algorithms used and make them open source to facilitate their use in heritage conservation.

The first 3D replica of St. Peter's Basilica, made with Microsoft AI

Microsoft already had some more modest experience in the field of digitising heritage. Also in collaboration with the company Iconem, it had previously worked on the recreation of Mont Saint-Michel (France) and the archaeological remains of Olympia (Greece) ; in both cases, they had already used AI to create 3D models that would serve as a basis for both immersive experiences and future restoration plans. But neither of these initiatives had the dimensions and complexity of the replica of St. Peter's Basilica, which Brad Smith assures that "without a doubt, it is one of the largest projects of its kind that anyone has undertaken." His company did not want to reveal how much money it has invested in the new digital twin, but it hopes that precisely the experience of having taken the idea to this enormous scale will serve to reduce the cost and duration of these projects, and thus lead to more being done.

Although the initiative by Microsoft and the Vatican serves to popularise and make the development of next-generation AI-powered digital twins cheaper, this is just one of many conservation initiatives that have sought to bring both technologies together in recent years, as James Hutson and Joseph Weber point out in their 2023 article Digital twins and the conservation of cultural heritage . The startup Iconem, founded in 2013, has specialised in the digitalisation of world heritage sites threatened by war in countries such as Syria, Iran and Afghanistan .

For Brad Smith, the fact that a digital twin can replicate a historical monument and allow a virtual visit, with certain advantages over a physical visit – or even keep it alive in some way, even if it is destroyed – will never replace in-person contemplation or lead to a loss of interest in its conservation: “In fact, if we had had a digital twin of Notre Dame Cathedral before its great fire, reconstruction would have been much easier and faster ,” declared Microsoft’s number two at the official presentation of the project. St. Peter's Basilica, an AI-enhanced experience .