When artificial intelligence helps tell the story of old manuscripts

Whether they are paintings, illustrations in old pharmacology manuscripts or astronomical diagrams dating back to the Middle Ages, images tell stories that Mathieu Aubry loves to bring to light. A researcher at the Gaspard Monge Computer Science Laboratory (LIGM*) at the École nationale des ponts et chaussées (ENPC), he uses artificial intelligence (AI) to track down the ‘visual structures’, and their repetitions and alterations, that tell the story of how knowledge spread.

As part of the ERC DISCOVER (Discovering and Analyzing Visual Structures) project, among others, Mathieu Aubry and his team are developing neural networks capable of extracting information from an image, sparing experts the tedious and time-consuming task of annotating documents beforehand. These neural networks, otherwise described as “functions applied to data”, identify correspondences between visual structures and highlight the coherence between them. “Unsupervised learning is an automatic learning process in which the algorithm is tasked to use raw data, including images, to obtain a result, for example based on the detection of similarities between some of these data. The aim is not necessarily to recognise the same visual structure elsewhere, but to identify one or more identical structures in a set, as well as their variations”, says the mathematician.

Detecting coherences between “visual structures”

For historians, artificial intelligence is proving to be a valuable tool. For example, it becomes possible to identify among paintings from the same workshop the parts that have been copied from each other and thus trace the genealogy of the paintings. One can also understand how ancient astronomical diagrams produced by generations of astronomers have interacted over time and spread.

“Our innovation makes it possible to objectify information and move up a scale,” explains Mathieu Aubry, who works with historians from institutions such as Sorbonne University, the Collège de France, the Observatoire de Paris and the Institut de Recherche et d'Histoire des Textes. “A historian can have access to a document, a few dozen or a hundred, but his scope remains limited. You can't study all the documents relating to a subject. If you want to have an overview on a larger scale, or if you want to carry out detailed analyses of variations, that becomes possible with digital tools.”

The computing power of neural networks is useful for illustrations of all kinds and from all eras. The ENPC researcher, who combines his love of mathematics with a penchant for art and history, is currently working with historians of science on the role of illustration in scientific publications in the Middle Ages and modern times.

AI within the grasp of historians

“In pharmacology, there are manuscripts with hundreds of illustrations of different plants,” says Mathieu Aubry. “Often, the person who copied the drawing of the plant has never seen the plant in real life. And the manuscript he's working on was also drawn up by someone who has never actually seen the plant. The drawing becomes distorted over time. Then at some point, someone redraws the illustration based on the plant, or the two illustrations are combined in the same manuscript. To understand these documents and what they tell us, we need to be able to analyse the images, find out which ones are similar and any breaks in representation, and thus trace the history of the different images.”

À propos :

Mathieu Aubry is a researcher in the IMAGINE group at the Gaspard Monge Computer Science Laboratory and at the École nationale des ponts et chaussées (ENPC). He is also a scholar at ELLIS (European Laboratory for Learning and Intelligent Systems) in the Paris unit. Mathieu Aubry is working on computer vision, more specifically the unsupervised analysis of image collection and applications to historical data and the analysis of Earth imagery.

>> Mathieu Aubry's personnal webpage

>> Mathieu Aubry on Google Scholar