Semantic Segmentation using Foundation Models for Cultural Heritage: an Experimental Study on Notre-Dame de Paris

Kévin Réby (1) , Anaïs Guillem (1) , Livio De Luca (1)

1 MAP - Modèles et simulations pour l'Architecture et le Patrimoine

The zero-shot performance of foundation models has captured a lot of attention. Specifically, the Segment Anything Model (SAM) has gained popularity in computer vision due to its label-free segmentation capabilities. Our study proposes using SAM on cultural heritage data, specifically images of Notre-Dame de Paris, with a controlled vocabulary. SAM can successfully identify objects within the cathedral. To further improve segmentation, we utilized Grounding DINO to detect objects and CLIP to automatically add labels from the segmentation masks generated by SAM. Our study demonstrates the usefulness of foundation models for zero-shot semantic segmentation of cultural heritage data.

Domains: Computer Vision and Pattern Recognition [cs.CV]; Machine Learning [stat.ML]

Kévin Réby, Anaïs Guillem, Livio De Luca. Semantic Segmentation using Foundation Models for Cultural Heritage: an Experimental Study on Notre-Dame de Paris. 4th ICCV Workshop on Electronic Cultural Heritage, Computer Vision Foundation, Oct 2023, Paris, France. https://openaccess.thecvf.com/content/ICCV2023W/e-Heritage/html/Reby_Semantic_Segmentation_Using_Foundation_Models_for_Cultural_Heritage_an_Experimental_ICCVW_2023_paper.html. ⟨hal-04275484⟩