PhD subject:
Contribution to the natural image segmentation: Application to the multimodal representation of painting images
Location: LITIS, STI team, University of Rouen, Madrillet site
Thesis Director : Prof. Edwige Pissaloux;
Co-supervisor : Katerine Romeo
Contact : Edwige.Pissaloux@univ-rouen.fr , Katerine.Romeo@univ-rouen.fr
Start Date : 2021
Financial support: ANR project
Subject description:
People with a visual impairment (PVI) do not have access to art, and painting in particular. Indeed, painting is currently treated as a purely visual art, ie. an art which must generate aesthetic impressions from what is actually seen. However, PVIs cannot see, and it is necessary to look for alternative ways to make art accessible to them (blindness gain [Cho 19]) and "to de-visualize art" [Ear 16, Tho 18].
Recent progress made by digital technologies, and knowledge of the sense of touch, suggests a new way of perceiving art that is multimodal: an ad hoc combination of natural segmentation of images and their haptic representation [Fur 14, Rom 18, Sou 20].
The concept of natural segmentation was launched by the University of Berkeley ([Arb 07, Arb 10], with the creation of BSDS 300 and BSDS 500 image databases). The natural segmentation of an image (into contours / regions ) is close to that performed by a human expert and taking into account the semantics of the scene. However, so far this concept has not found its effective implementation.
Recently, a so-called adaptive edge detector has been proposed [Fro 14]. Although it offers superior qualitative performance to other detectors evaluated (Sobel, Canny, Canny Color, Pyramidal), this detector aims to improve the display of contours superimposed on real images in order to improve the perception of all elements of the image by touch.
However, PVIs (and especially those who were born blind) need to perceive the content of images through a "2D visual gist", ie. a set of landmarks perceptible by touch and which would make it possible to quickly and comprehensively understand the content of the scene studied, the set would give a certain cognitive and cultural content to the painting.
It is therefore necessary to seek an approach to image segmentation, close to natural segmentation, and allowing to quickly understand the most relevant elements of a scene and to represent them haptically.
Goals :
The objectives are of three different kinds: theoretical research, design of the algorithm and experimental validation.
- The development of a new segmentation method (in contours and regions). Hierarchical (pyramidal) methods combined or not with methods based on machine learning (eg (D) CNN, [Afi 19]), or even bio-inspired methods modeled by a certain mixture of Gaussians (simulating the cognitive processes of human), or CRFs (Conditional Random Fields which take into account the context of an object) should be studied, synthesized and / or improved. In addition, a contour simplification method [Rom 18] suitable for displaying objects on an F2T force feedback tablet, from LITIS, must be proposed. Finally, an audio aid for exploring the represented tactile scene should be integrated.
These are the theoretical objectives of this doctoral work.
- The proposed methods should be integrated into the F2T software (design and implementation objectives) in order to prove their relevance and usefulness for the target user population.
- Realistic, or even real, scenarios must be tested with end users (PVI and museum curators), and finally integrated into the Inclusive Museum Guide (IMG). These are experimental validation objectives. It is very likely that experimental results will lead to the refinement of the proposed segmentation approach.
Required skills:
- Good general knowledge in computer science (Java, Python), image processing / vision, machine learning.
- Knowledge on tactile perception
- Good general culture in electronics and control engineering
- Good team player
- Perseverant.
Documents to send in support of your application:
- CV
- Master 1 and Master 2 grades or equivalent
- The names of two people who know your academic background
All candidates selected for an (e-)interview will have to make a presentation of
- Their academic training
- Their understanding of the subject making a link with their academic training.
Références :
[Afi 19] Mouna Afif, Riadh Ayachi, Yahia Said, Edwige Pissaloux, Mohamed Atri, An Evaluation of RetinaNet on Indoor Object Detection for Blind and Visually Impaired Persons Assistance Navigation, Neural Processing Letters, 51, 2265–2279, 2020
[Arb 07] Arbelaez, P. et al., The Berkeley Segmentation Dataset and Benchmark, 2007, https://www2.eecs.berkeley.edu/Research/Projects/CS/vision/bsds/
[Arb 10] Arbelaez, P. et al., Contour Detection and Hierarchical Image Segmentation, IEEE PAMI 2010.161
[Ear 16] Eardley, AF, Mineiro, C., Neves, J., Ride, P., Redefining Access: Embracing multimodality, memorability and shared experience in Museums, Curator: The Museum Journal 59 (3), 263-286, 2016
[Fro 14] Konik, H., Froissard, B., Dinet, E., Trémeau, A., Contribution of edges in augmented reality eyewear to assist visually impaired people in their mobility, Proc. of HCI : Universal Access in Human-Computer Interaction. Design for All and Accessibility Practice, Lecture Notes in Computer Science, Volume 8516, pp 182-191, 2014
[Fur 14], Furferi, R., Governi, L., Volpe, Y., Puggelli, L., Vanni, N., Carfagni, M., From 2D to 2 ;5D i.e. from paintings to tactile model, Graphical Models, 76, 706-723, 2014
[Tho 18] Thompson, H., « Blindness Arts » (co-ed. V. Warne), Disability Studies Quarterly, vol. 38, no.3, 2018
[Sou 20] A. Souradi, C. Lecomte, K. Romeo, S. Gay, M-A. Rivière, A. Elmoataz, E. Pissaloux, Towards the tactile discovery of cultural heritage with multi-approach segmentation, ICISP 2020, Image and Signal Processing, Ed. A. El Moataz, D. Mammass, A. Mansouri, F. Nouboud, Springer LNCS 12119, p14-23, June 4-6, 2020
[Rom 18] K. Romeo, M. Chottin, P. Ancet, C. Lecomte, E. Pissaloux. Simplification of Painting For Tactile Perception by Visually Impaired Persons. ICCHP 2018, 16th Int. Conf. Computers Helping People with Special Needs, Linz, Austria, July 11-13, Springer, p251-257, 2018
