Computer-based methodologies for semi-automatic
3D model generation from paintings
Y. Volpe*, R. Furferi and L. Governi
Department of Industrial Engineering, University of Firenze,
via S. Marta, 50139 Firenze, Italy E-mail: yary.volpe@unifi.it E-mail: rocco.furferi@unifi.it E-mail: lapo.governi@unifi.it *Corresponding author
G. Tennirelli
Department of Literature and Philosophy, University of Firenze,
Piazza Brunelleschi, 3-4 50121, Firenze, Italy E-mail: giovanna.tennirelli@unifi.it
Abstract: Over the last few years, technologies like 3D scanning and rapid
prototyping provided an extraordinary boost in improving reproductions of 3D artworks, like sculptures and historical buildings, all over the world. Physical 3D reproduction of subjects represented in paintings, is recognised to be one of the best ways to allow visually impaired people to enjoy such kind of artworks. However, the use of advanced technologies with the aim of realising 3D models starting from paintings has not been satisfactorily investigated yet. Though a number of algorithms coming from computer vision science exist to cope with similar issues, the specific problem of producing a 3D representation which is targeted at blind people tactile exploration has been only marginally investigated. Starting from these considerations, this work presents 1) a quite extensive review of the criteria proposed in literature for producing tactile models suitable for blind people and 2) four alternative computer-based methods for semi-automatic generation of tactile 3D models starting from RGB digital images of paintings. The outcomes of this study contribute new information to the field of visually impaired user-oriented 3D reconstruction and clearly indicate the strategy to be adopted in order to produce a meaningful reproduction of a bi-dimensional piece of artwork.
Keywords: haptic exploration; 3D computer-based modelling; blind; visually
impaired.
Reference to this paper should be made as follows: Volpe, Y., Furferi, R.,
Governi, L. and Tennirelli, G. (2014) ‘Computer-based methodologies for semi-automatic 3D model generation from paintings’, Int. J. Computer Aided
Engineering and Technology, Vol. 6, No. 1, pp.88–112.
Biographical notes: Y. Volpe received his PhD in Machine Design and
Construction from University of Florence, Italy in 2006. He graduated with MSc in Mechanical Engineering in University of Florence. He is currently working as an Assistant Professor at the Department of Mechanics and
Industrial Technologies, University of Florence. He is also an Adjunct Professor of the course ‘computational graphics’ from the Engineering Faculty of the University of Florence. His main scientific interests are: computer aided design, image processing, virtual prototyping, FE simulation, reverse engineering and rapid prototyping. He is author and co-author of many publications printed in international journals and participated to a number of international conferences. Some latest publications described techniques for comfort-oriented design, machine vision-based systems for industrial processes and spline-based approximation of point clouds.
R. Furferi received his PhD in Machine Design and Construction from University of Florence, Italy in 2005. He graduated with MSc in Mechanical Engineering in University of Florence. After working as a Post-Doctoral Researcher at the Department of Mechanics and Industrial Technologies of the University of Florence, in 2008 he assumed a faculty position as an Assistant Professor for the course ‘mechanical drafting’. His main scientific interests are: development of artificial vision systems for industrial and textile control, artificial neural networks, colourimetry, reverse engineering and rapid prototyping. He is author and co-author of many publications printed in journals and presented on international conferences. Some of his latest publications described methods for colour assessment of textiles, algorithms for 3D reconstruction of objects from orthographic views and ANN-based systems for industrial process.
L. Governi received his PhD in Machine Design and Construction from University of Florence, Italy in 2002. He graduated with MSc in Mechanical Engineering in University of Florence. After working as a Post-Doctoral Researcher at the Department of Mechanics and Industrial Technologies of the University of Florence, in 2005 he assumed a faculty position as an Assistant Professor for the courses of ‘reverse engineering’ and ‘design and modelling methods’. His main scientific interests are: machine vision and reverse engineering, colourimetry, tools and methods for product design and development. He is author and co-author of many publications printed in international journals and participated to a number of international conferences. Some of his latest publications described techniques oriented towards the 3D reconstruction from orthographic views, vision-based product and process assessment and spline-based approximation of point clouds.
G. Tennirelli received his PhD in Technologies and Management of Cultural Heritage from the IMT (Istituto Alti Studi) Lucca, Italy. She graduated with MSc in Conservation and Cultural Heritage in University of Florence. She is currently working as a Contract Professor at the Department of Literature and Philosophy, University of Florence. Since 2005, she has been working in the field of museum accessibility for blind and visually impaired people.
1 Introduction
Blind or visual impaired people are usually disadvantaged to enjoy visual artworks because these are created for sighted people. Even if several solutions have been proposed by some museums in order to increase the accessibility of impaired people to the cultural life, more museums are nowadays ‘behind the time’. A number of initiatives have spread over the world with the aim of developing services for blind and partially
sighted people. Such services, offered among others by the British Museum (Michelangelo Exhibition) and by the London Royal Academy, usually, include human assistance, audio guides, Braille descriptions and others. Awkwardly, if on one hand supporting systems allows blind and visually impaired people to ‘be informed’ about the artworks, they are not really able to sense and explore them thus being prevented to “enjoy greatest artworks of human times” (Turtle, 2012).
Tactile exploration is the primary means that blind people use to learn about their surroundings through the manipulation and all the sensations that contribute to the process that is called haptic exploration (from Greek haptomai, grab hold, touch with care) (Quatraro, 2008; Allegro, 1991; Grassini and Scichillone, 1997). Such a process generates a complex cognitive iter, aimed at creating a mental image that comes from two phases: an initial rapid and brief exploration of the whole and a second one, more detailed, addressed to analysing a small portion of the surface to define details and place them in the context of the whole ‘image’ (Grassini, 1998). Unfortunately, many of the artworks exposed in a museums cannot, obviously, be touched; this prevents a direct interaction between the blind people and the artworks even in the simple case of sculptures (for sculptures the sense of touch would provide an almost complete comprehension of the artwork to blind people). As a consequence blind and visually impaired people are generally unwilling to participate to artworks expositions not only of paintings, photos, digital art, graphics (which are inherently flat forms of art which cannot be explored by the sense of touch), but also of sculptures. In order to confront with these issues, tactile interaction with works of art in museums has been particularly studied in the last 20 year with the aim to encourage social inclusion and to provide a supplementary educational approach to gallery programming. Nowadays, numerous initiatives based on the interaction with sculptures and tactile three-dimensional reproductions or architectural aids on scale are quite commonly diffused. Some relevant examples of the interest demonstrated in this field are provided by the tactile display integrated in the British Galleries at the Victoria and Albert Museum in London (and the related handling activities) and by the touch tour organised with a selection of original sculptures or special replicas in a number of international museum such as Louvre, Tate Modern, British Museum, MOMA (Salzhauer Axel, 2003). From the early nineties, temporary or permanent initiatives dedicated to the blind were born in several European museums, through the creation of three-dimensional copies of works of art. Some of the most significant cultural institutions are: the Tactile Museum Omero (Ancona, Italy), the Tactile Center (Catania, Italy), the Haptic Museum of Sicily (Palermo, Italy), the Tiphlological Museum ONCE (Madrid, Spain) and the Tactual Museum (Athens, Greece).
Many of these museums present scaled copies of 3D objects like sculptures, monuments, archaeological finds and bas-reliefs. These objects are manually realised by artists by using materials like clay, plaster, wood, bronze, etc. Only in few cases some advanced technological systems like 3D scanning and rapid prototyping (RP) have been used. A relevant project where these techniques have been applied is the ‘Toccare l’Arte’ (‘Touch the Art’) project, carried out in the Egyptian Museum of Turin (Italy), where a number of small Egyptian statues have been reproduced. 3D acquisition systems and RP techniques allow a high level of accuracy in reproducing 3D objects and, consequently, should be preferred to the manual process for obtaining an accurate replica of a sculpture.
Obviously, the situation is completely different for pictorial artworks (paintings, photos, etc.)which are not three-dimensional. As a consequence they cannot be directly
reproduced in the form of a 3D object (like a sculpture) but they need to be translated into an appropriate ‘object’ to be touched. One of the most important experimental approaches applied to paintings has been developed by the Museum of Tactile Antique and Modern Painting, ‘Anteros’, founded by the Istituto Cavazza in Bologna (Italy). It exhibits three-dimensional reproductions of works of art by famous painters in art history (see Figure 1).
Figure 1 Examples of three-dimensional reproductions of works of art by famous painters in art
history (see online version for colours)
Source: http://www.cavazza.it
The reproductions are hand sculpted by artists of the School of Applied Sculpture of Bologna and are obtained in the form of bas-reliefs, accompanied by preparatory information material on historical styles and descriptive specification sheets (in Braille).
An important project is the ‘Chiaroscuro project’ of the ‘Centro Internazionale del libro parlato’ (Belluno, Italy). From a few years this institute is working with major museum institutions for making tactile copies of paintings.
Other relevant experiences include handmade reconstructions of the ‘Venere’ by Botticelli at the Uffizi Gallery (Firenze, Italy) and of the ‘Deposizione’ by Caravaggio at Musei Vaticani (Roma, Italy). A tactile section has also been realised at the ‘Museo Nazionale della Fotografia Alinari’ in Firenze (Italy) presenting tactile reproductions of ancient photographic images.
All these initiatives, however, entail the work of specialised artists (sculptors) thereby drastically reducing the number of representations practically built. From this point of view, a systematic engineering procedure oriented towards a possible industrial process for translating into tactile models pictorial artworks could dramatically improve blind people’s accessibility to artworks.
In addition to the examples provided above (mostly aiming at the creation of a bas-relief-like representation of the painting) some techniques based on commercial technologies are also available and quite commonly used. These include: serigraphy, micro-capsule Minolta paper, gaufrage (see Figure 2), each one entailing a different way of representing the starting image. One of the most widespread means among the ones
listed above is the Minolta technique, which typically produces a constant thickness relief superimposed on a paper sheet, usually representing the outlines of the image. This kind of relief is know as tactile diagram. Tactile diagrams are not the exact relief reproductions of visual images; rather, they are translations of visual images into a tactile language and are generally used in conjunction with verbal narratives that guide the user through the diagram in a logical and orderly manner.
Figure 2 Details of the Palazzo Ducale (Italy) Gothic lodges realised with different techniques:
(a) serigraph,( b) gaufrage, (c) micro-capsule minolta paper
(a) (b) (c)
Source: http://www.letturagevolata.it
2 Strategies for generating 3D models from paintings
The initiatives presented in the introductory section demonstrate a growing interest regarding the potentiality of paintings perception for blind and visually impaired people. Research in this field, however, is also focused on the definition of criteria and methods to reproduce tactile translations meeting the needs of blind people which is the crux of the matter (Salzhauer Axel, 2003; Hatwell and Martinez-Sarocchi, 2003; Kennedy, 2003; Kennedy and Igor, 2003). In fact, despite several techniques for tactile aids production exist (see Introduction), no guidelines for translating visual images into tactile language, shared at an international level, can be found. “Recognition mechanisms of 3D objects based on tactile information differ from those based on visual information” (Teshima, 2010).
From this point of view, a few indications are provided only for the case of tactile diagrams by the American Foundation for the Blind (AFB) and by the University of Torino (Italy). Some of the information contained in the guidelines produced by these two institutions are described below, since they have inspired some of the computer-based techniques presented in this paper (see Section 3). The guidelines defined by the AFB are up to date an important reference point for the experiences exchange and for a universal design perspective of cognitive support tools (Salzhauer Axel, 2003). The AFB guidelines suggest to transform paintings into one of the following categories of images (see Figure 3):
1 Simple image: Objects are represented by means of simple, outlined, shapes. In paintings, a thin line describes the outer edge, or border, of the image. In addition to this outlining function is possible to use thick lines to emphasise specific objects, or to describe the shapes that are closest to the viewer. For example, an overlapping shape may be thickly outlined to show that it is the most prominent form. 2 Complex image: Simple image convey only a limited amount of information;
accordingly, when the subject is represented by a complicated image it is necessary to break it down into multiple diagrams.
3 Complex images with enlarged detailed views: Images including too many objects or small details can be treated as complex images (schematic diagrams) followed by enlarged detailed views. This diagram consolidates the two preceding diagrams in a composite view of the complete image. With this method, the visual information is introduced in a series of steps. The final diagram should sum up the details in the preceding diagrams. This enables the user to assemble, piece by piece, a mental image of a highly complex work of art. This method also helps the user understand the relationship of the objects or figures to one another and to the surrounding space (Kardoulias, 2003).
Figure 3 Hope II by Gustav Klimt: original painting and diagrams of two detailed views
(see online version for colours)
Source: http://www.artbeyondsight.org
When multiple diagrams are used to illustrate one image, their tactile vocabulary must be consistent. For example, the same line widths and patterns must be maintained throughout the diagram sequence. In the sequence, there should be a gradual progression from basic information to more detailed information. The first diagram should be the simplest, and the final one the most complex. The most common way to break down an image is to separate the background and foreground, or the ground and figures, illustrating them in two separate diagrams or using two different patterns. More in general, outline-based representations may be enriched using different textures each one characterising different position in space and/or different surface properties. For instance (see Figure 4):
• solid-rough pattern can represent depth and can be used to identify shapes that are farther back in space, such as those in the background
• dot pattern can be used to represent depth, but also to identify shapes that are closer to the ‘observer’, such as those in the middle ground or foreground
• basket-weave pattern can serve to represent solid shapes, such as furniture or structural masses
• solid pattern can be used to emphasise important shapes or to represent objects in the foreground, which are closest to the viewer.
Figure 4 Example of outline-based representations enriched using different textures
Source: http://www.tactilevision.it
As mentioned before, some useful indication are also provided by a study carried out by the University of Torino (Italy), in collaboration with Tactile Vision Onlus, which for nearly 20 years has been working in collaboration with the Italian Union of the Blind. Tests with congenitally and late blind, older than ten years, highlighted that shape, extension and position of the various composition elements are the more understandable the more the following criteria are adopted:
1 tactile aids must keep the size of the explorable space including extension of the arms, and allow an easy handling
2 when possible, verbal description is to be used as a support
3 complex images have to be fragmented into a sequence of simple ones 4 a relief line is difficult to perceive if the line width is lower than 0.5 mm
5 relief lines cannot be too close the one to the other: this minimum distance needs to be around 2 mm
6 the minimum height of relief needs to be 0.4/0.6 mm in order to be correctly perceived by the user (Levi, 1993; Levi and Rolli, 1994).
While tactile diagram realisation is somehow guided by the works described above, no indications at all can be found for bas-relief representation, even if they are generally recognised to be more effective in conveying 3D information in a more ‘realistic’ way and in improving depth perception.
Summing up, on the basis of the state of the art, it is evident that:
• though several studies provide a few criteria for converting paintings (or images in general) into tactile models, a shared methodology is still missing and further work investigating actual blind people requirements and perception is still necessary • most of the tactile models which can be found in blind-dedicated exhibitions are
hand-crafted by artists; this kind of process is very expensive and time consuming, so that only a few pieces of artwork are actually manufactured.
As a consequence, computer-aided technologies for automatic or semi-automatic translation of paintings into tactile models would considerably help broadening the number of artworks accessible to blind and visually impaired people. These technologies, however, should be carefully tested in order to assess their outcome in terms of perceivability.
Though a number of algorithms coming from computer vision science exist to cope with the issue of 3D reconstruction from single images, the specific problem of producing a 3D representation which is targeted at blind people tactile exploration has been only marginally investigated. Some interesting examples of method based on computer vision for automatic conversion of image into its tactile form are those proposed by Hernandez and Barner (2000), Jayant et al. (2007) and Wang and Li (2010).
Moving from these considerations, this work aims to provide a series of computer-aided methodologies for the semi-automatic generation of four different kind of tactile 3D models (starting from paintings) and to investigate the effectiveness of each kind of representation according to the users’ requirements and actual perception. This work is consequently meant to provide useful information for anyone working in the field of tactile representation of paintings for blind people and, more in particular, for researchers willing to confront with the issue of automating the reconstruction process.
In the following section (Section 3) the devised methodologies will be presented with reference to the iconographic subject of ‘still life’, interpreted by Fernando Botero (1932–…) a great artist of the twentieth century whose selected work is shown in Figure 5(a).
Section 4 presents the results of tactile exploration performed by a panel of visually impaired people on the 3D models representing the Botero painting and, also, on the analogous models obtained for one of Giorgio Morandi’s (1890–1964) masterpieces [see Figure 5(b)].
Finally, in Section 5, a few concluding considerations are drawn and possible future work is discussed.
Figure 5 (a) F. Botero ‘still life with fruit’ (b) G. Morandi ‘still life I’ (see online version
for colours)
(a) (b)
3 3D reconstruction methodologies
Based on the state of the art analysis, four kinds of 3D tactile models (two tactile diagrams and two bas-relief-like representations) have been selected with an increasing reconstruction difficulty level:
• tactile outline: objects in the paintings are represented by relief lines following their contours
• texturised pattern: each object of the scene is characterised by a different pattern • flat layered bas-relief: each object of the scene is represented by flat surfaces at
different depths • bas-relief.
Starting from high resolution RGB digital images of the original work (painting), different image processing-based and CAD-based methodologies have been applied depending on the kind of 3D representation sought. The authors, in the past, have coped with the problem of reconstructing 3D geometry from single or multiple views (Bartolini et al., 2004; Furferi et al., 2011a, 2011b; Carfagni et al., 2012); however, these approaches cannot be successfully applied since most of paintings lack the necessary 3D information. As a consequence, alternative reconstruction strategies have been experimented. The resulting 3D virtual models have been, finally, prototyped using well established RP techniques, so to make them usable for the subsequent tactile exploration phase.
3.1 Tactile outline
In order to perform this kind of reconstruction, it is necessary to extract the edges of the digital image to be processed. Edges, together with other information such as regions delimited by edges themselves, are also necessary to perform the texturised pattern and
the flat layered bas-relief reconstructions. Accordingly, instead of applying techniques aimed only at contour extraction, segmentation algorithms have been considered. Image segmentation consists of partitioning a digital image into multiple segments (regions); more precisely, it is the process of assigning a label to every pixel in an image such that pixels with the same label share certain visual characteristics (Shapiro and Stockman, 2001). Note that once a region (set of pixels) is identified, its delimiting outline/s is/are immediately defined by the pixel lying on the region perimeter.
Over the years, a huge number of segmentation techniques have been proposed, however, as widely known within the scientific community, a general solution to the problem has not been found to date. Some of the most reliable segmentation techniques are: thresholding, clustering, region growing, region splitting and merging, boundary extraction, watershed and techniques based on edge detection (Ziou and Tabbone, 1998; Nadernejad et al., 2008). For each of these methods, pros (for instance the speed in case of thresholding method) and cons (for instance the need to connect possible separated contours in case of edge detection-based method) can be identified. A number of algorithms based on different methods have been implemented and tested by the authors; among them, the most suitable results for the selected paintings have been obtained using the watershed segmentation technique. However, the obtained segments were still unsatisfactory in order to be used for the 3D reconstruction process. As a consequence, an hybrid and interactive algorithm based on watershed segmentation (Beucher and Lantuéjoul, 1979) and Mortensen’s livewire methods (Mortensen and Barrett, 1996) has been used and is briefly outlined below.
Let then:
• I (size a × b) be the RGB image of the painting to be processed
• G (size a × b) be the greyscale image obtained by discarding colour information from I.
First, a marker-controlled watershed segmentation is applied to G in order to obtain a first attempt clustering. More in detail, the Prewitt-based gradient magnitude image Gm is used as a segmentation function (Beucher and Lantuéjoul, 1979) of G; moreover, in order to avoid an over-segmentation, the image is preliminarly ‘cleaned’ first by applying a 3 × 3 median filter and, then, by using a morphological opening procedure followed by a morphological closing one.
Depending on shape (circular, rectangular, linear, etc.) and dimensions (in pixels) of the structural element used for opening and closing, different results can be achieved. Referring, for instance, to circular shaped structural elements, lower dimensions may result in considerable over-segmentation [Figure 6(a)] while higher dimensions lead to under-segmentation [Figure 6(c)]. In the present work a circular shaped structural element with radius β is applied [Figure 6(b)], where:
0.02 min( , )a b
= ⋅
β (1)
Whichever are the value of β and the shape of the structural element, watershed-based procedure alone is not able to properly cluster all the objects in the image. In fact, generally speaking, some contours are missing while some objects are still over-segmented. As a consequence further processing is required. For this purpose, the Mortensen’s livewire method is applied to the gradient magnitude image Gm. Such a method allows to interactively modify the gradient values of the contours separating each
cluster (not been recognised by the watershed-based procedure). The result consists of a new gradient magnitude image G′m where the weak or even missing contours have been manually reinforced. Using this new image as a segmentation function of G a new clustering is produced [Figure 7(a)].
Figure 6 Results obtained with different β values, (a) Over-segmentation (b) Correct segmentation (c) Under-segmentation
(a) (b) (c)
If, on one hand, the desired contours are added, unfortunately the watershed procedure applied to G′m still results in a number of undesired contours (i.e., undesired clusters). As a consequence, an interactive algorithm capable of merging clusters has been used. In detail, the clusters to be merged are selected by the user by clicking on a generic point inside each one; the inner contours of the selected clusters are, then, removed. Clusters characterised by small areas (e.g., lower than β / 42
) are automatically merged with the nearest cluster (i.e., the one whose centroid is located at the minimum distance with respect to the small area centroid). The result of the application of the interactive algorithm is illustrated in Figure 7(b).
Finally, some contours are manually added in order to take into account possible relevant details describing the scene (e.g., the fruit stem) that are, due to their own nature, undetectable by the watershed procedure [see Figure 7(c)]. The final result of this procedure consists of two images: the contour image C and the clustered image L (i.e., a labelled image where different labels are assigned to different clusters).
Figure 7 (a) Contour image obtained using gradient magnitude image G′ (b) Contour image m
obtained after cluster merging procedure (c) Final result image C
Starting from image C, three more steps need to be carried out in order to obtain a depth map (image in which the grey tone of each pixel represents its elevation); these steps are: • C image inversion
• contour dilation of inverted image, in order to obtain a suitable contour width (at least equal to 0.5 mm once the model is physically prototyped)
• Gaussian smoothing, in order to smooth sharp edges.
The final depth map is shown in Figure 8(a). Starting from that, it is straightforward to obtain the surface reproducing the outlines in relief [see Figure 8(b)]. Since, as explained in Section 2, outlines are required to have a minimum height equal to 0.6 mm, a proper elevation scale (z axis) is chosen.
Figure 8 (a) Depth map and (b) surface (see online version for colours)
(a) (b)
Figure 9 (a) STL file and (b) physical prototype (see online version for colours)
(a) (b)
The resulting surface, using an appositely developed procedure, is finally converted in the form of a triangular mesh (STL file) and imported into a reverse engineering software application (Polyworks®). In this environment, the surface is provided with a suitable thickness in order to obtain the necessary stiffness of the tactile support as shown in
Figure 9(a). Such an STL model is finally ‘printed’ using a RP machine (EDEN 250 by Objet) as shown in Figure 9(b).
3.2 Texturised pattern
As already stated, the hybrid and interactive procedure described in Section 3.1 allows to determine not only the contours of the objects (matrix C) in the scene, but also the relevant regions (labelled matrix L with i labels). With the aim of obtaining a texturised pattern representation model, a properly defined image-texture Ti has been used for each of these i regions [represented in false colours in Figure 10(a)]. Each region of the texturised pattern model is required to satisfy the following requisites (Levi and Rolli, 1994):
1 texture height equal to, at least, 0.6 mm; this is due to the fact that the texture should be higher than the minimum required, in order to be correctly perceived by users, but also lower than the outlines
2 outlines height equal to, at least, 0.78 mm (i.e., 30% higher than texture relief) 3 texture elements and outlines width equal to, at least, 0.5 mm
4 distance between each relief element (both for outlines and textures) equal to, at least, 2 mm.
Accordingly, the following procedure has been adopted in order to build a representation model fully complying with the above requisites. First, a dilation is performed on the image C in order to obtain the desired outlines width; the result of this task is a new matrix C′. Then, for each generic ith
cluster in image L, a binary image Li is defined; such an image, whose size is equal to one of image L (i.e., a × b), has values equal to 1 for all the pixels belonging to the ith cluster. Each Li image is morphologically eroded in order to take into account the necessary distance that should subsist between texture and outlines. The result of this image erosion is a set of i matrices L′i. Finally, the texturised model is
described by the greyscale image T provided by the following equation: .
i i i
T =⎛⎜ L T′⋅ ⎞⎟⋅C′
⎝
∑
⎠ (2)where matrices Ti are selected from an appositely devised database of textures, defined according to the criteria mentioned before; note that matrices Ti are greyscale images whose maximum pixel value is equal to 70% of white pixel one (i.e., outlines pixel value).
Matrix T is, finally, processed with a Gaussian filter in order to obtain the corresponding depth map [Figure 10(b)] and the final surface. Such a surface is converted into a triangular mesh [Figure 10(c)], processed into Polyworks® and printed by the RP machine [Figure 10(d)].
Figure 10 (a) Segmented image, (b) depth map, (c) STL file and (d) physical prototype (see online
version for colours)
(a) (b)
(c) (d)
3.3 Flat layered bas-relief
The reconstruction process of the flat layered bas-relief representation starts from the segmentation results obtained in the previous one. With reference to Figure 11, the areas of the object approximately belonging to horizontal planes (HPs) are defined as regions with: a constant value of the vertical gradient of the grey level function, a zero value of the horizontal gradient and sign determined by the plane position with reference to the horizon line. Analogously to HPs, in case of side planes (SP) the areas are defined with a constant value of the horizontal gradient of the grey level function and with zero value of the vertical gradient. For regions approximately belonging to front planes (FPs) a constant value of the grey level function are used, manually ‘picking’ it from the HP or SP zone on which the FPs are attached.
Figure 11 Illustration of grey level for object approximately belonging to front (FP), horizontal
(HP) and side planes (SP) (see online version for colours)
Using the above described rules the depth map shown in Figure 12(a) has been interactively obtained. The corresponding 3D model and physical prototype are shown respectively in Figures 12(b) and 12(c).
Figure 12 (a) Depth map, (b) STL file and (c) physical prototype (see online version for colours)
(a) (b) (c)
3.4 Bas-relief
In order to obtain the last representation (bas-relief), a manual methodology based on CAD modelling has been used. Starting from the acquired image of the painting, each object in the scene is, individually, 3D modelled. By a way of example, referring to the Botero’s table in the painting, the image representing the painting is inserted in a 2D sketch (as background) with the aim of extracting the relevant construction curves [see Figure 13(a)]. Using the typical 3D CAD features, such as extrusions and revolutions, the solid model is then built [see Figure 13(b)]. In Figures 14(a) and 14(b), the model of the teapot and of one of the fruits is shown.
Figure 13 (a) 2D sketch superimposed onto the painting and (b) 3D model of table with napkin
(see online version for colours)
(a) (b)
Figure 14 (a) 3D model of the teapot and (b) of one of the fruits composing the scene (see online
version for colours)
(a) (b)
Figure 15 (a) 3D CAD model, (b) pseudo-3D surface and (c) physical prototype (see online
version for colours)
Each 3D model (CAD part) is, then, positioned into an assembly with the aim of reproducing the 3D scene as accurately as possible [see Figure 15(a)]. In order to place all the parts in their correct mutual position, it is first necessary to approximately identify the viewing direction and use the same one in the CAD environment when arranging the parts in the global assembly model. The reproduced 3D scene is finally converted, using an appositely devised algorithm whose details are omitted for brevity, in a 2,5D surface [Figure 15(b)] represented by means of a triangular mesh. Such a surface is provided with a suitable thickness and printed by the RP machine [Figure 15(c)].
4 Tactile exploration
4.1 Assessment
Tactile exploration assessment has been carried out in cooperation with the Italian Union of Blind and Visually Impaired People, located in Florence (Italy). A panel of 14 users characterised by total blindness, defined as the complete lack of form and visual light perception, has been selected. Among such 14 users (of different ages, see Table 1) eight are characterised by congenital total blindness while the remaining six acquired the visual deficit in the pre-teen.
Table 1 Users’ composition
Panel Congenital total blindness Acquired total blindness Age
User 01 X 67 User 02 X 54 User 03 X 73 User 04 X 31 User 05 X 36 User 06 X 47 User 07 X 73 User 08 X 62 User 09 X 35 User 10 X 57 User 11 X 57 User 12 X 58 User 13 X 34 User 14 X 25
The testing phase was performed by a professional who is specifically trained to guide people with visual impairments in tactile exploration.
Each test has been divided into two different phases: an introductory phase and an exploration phase. During the tests, a one to one relationship between interviewer and user has been adopted in order to put the subject at ease and not to hurry him/her. Each test required, on average, 55 minutes.
The introductory phase consists of a knowledge questionnaire followed by a verbal description about the general cultural context and the pictorial language of each artist. A
simple language together with essential information has been used by the professional (from now on ‘interviewer’) in order not to decrease tester attention or create conceptual difficulties. In fact, the aim was to simulate the informative input of a Braille or audio device, which could be provided by a tactile museum.
The exploration phase consists of a tactile exploration of the prototypes performed by the users according to a three level structure with decreasing difficulty. Each level ended with a set of evaluation questions, recorded into a questionnaire. Questions were related to the readability of the models and, generally, to the recognition of the iconographic subject, proceeding with more specific issues regarding tactile comprehension in relation to the composition of objects (and lines) and contour perception, until to investigate the potential to identify the location of objects on the representation space and the possibly perceived differences between the flat-layered bas-relief and the bas-relief representations. During question time, users were given the opportunity to explore the models in a comparative modality.
• Level 1 – Complete autonomy. First, the prototypes are experienced by the users in complete autonomy, i.e. without any interference from the interviewer, and without any limitation in terms of exploration time and modalities.
• Level 2 – Facilitated autonomy. On the basis of the users capability in recognising the general iconographic subject (performed in Level 1), the interviewer decides whether or not to progress to a subsequent level by providing additional information about the artist, its peculiarities and the concept of ‘still life’ in the art history. Further data about the types of represented objects, their position in representative space and the work title are also provided.
• Level 3 – Interviewer guided. In case of persisting difficulties related to perception/cognition, the third exploration level is performed. In this phase the interviewer adds the exact description of the objects and their mutual position. The verbal support is associated with a tactile guidance technique targeted to facilitate the reading process by: a preliminary identification of the composition, a deeper perception of each object/subject, preferring a formal and spatial reading with references – where appropriate – to the graphic deformation due to the perspective view.
In Figure 16, some of the exploration phases are shown.
Tactile assessment has been conducted on Botero’s ‘still life with fruit’ and Morandi’s ‘still life I’.
4.2 Results
In this section, some of the results coming from the analysis of questionnaires are briefly presented, especially pointing out the most relevant outcomes. In particular, the analysis is related to the questions listed in Table 2.
Table 2 Questions analysed
Panel Question
Q1 Which of the four models is more readable? Q2 Why the model you selected in Q1 is more readable? Q3 How difficult is to understand contours of objects? Q4 How difficult is to perceive object position in the 3D space? Q5 For each representation (i.e., model), how clear is the composition
of the represented scene?
In Table 3, the exploration level, in which users were able to perceive the paintings, is pointed out referring to the two case studies.
Table 3 Exploration level in which users were able to perceive the paintings
Morandi Botero Panel
Level 1 Level 2 Level 3 Level 1 Level 2 Level 3
User 01 X X User 02 X X User 03 X X User 04 X X User 05 X X User 06 X X User 07 X X User 08 X X User 09 X X User 10 X X User 11 X X User 12 X X User 13 X X User 14 X X
As depicted in Figure 17(a) only in few cases (21% for both paintings) the users were able to understand the scene at the first exploration level (complete autonomy) at least for one case study. Hence, the lack of information about the artist, the types of represented objects and their position in representative space (i.e., all the information provided after the first level) partly compromises the readability of the tactile models. By analysing the exploration level per deficit occurrence, i.e., people affected by congenital blindness
(C.B.) and acquired blindness (A.B.), it is possible to observe that 50% of C.B. affected people need a guided reading, while the other 50% is able to understand tactile paintings in a completely independent way or, at least, with verbal assistance. In case of A.B., the percentage of people capable to understand the model without interviewer guidance, reaches 75%. This result suggest that a painting description using audio guide or caption in form of Braille text has to be used together with the tactile models in order to make them understandable by blind people.
Figure 17 (a) Exploration level (b) Exploration level per deficit occurrence – congenital blind
people in blue and acquired blindness in red (see online version for colours)
(a)
(b)
Furthermore, Figure 18(a) shows that 92% of C.B. affected people and 65% of the ones affected by A.B. perceived bas-relief models as the ones characterised by the best ‘readability’. Accordingly, these models can be considered as the most effective translation of the two paintings chosen for testing the procedure. Flat bas-relief, on the other hand, was preferred by 25% (A.B.) against 8% (C.B.). Congenital blind subjects did not rate at all tactile outline and texturised pattern models.
Figure 18 (a) percentages of CB (blue) and AB people (red) related to the ‘readability’ of the
models and (b) related to the reasons of their preference (see online version for colours)
Which of the four models is more ‘readable’?
(a)
Why the model is more ‘readable’?
(b)
The reason of choosing bas-relief as a comprehensible representation of the selected paintings has been motivated using different considerations depending on the typology of visual disability [see Figure 18(b)]: the A.B. affected people believed that such models allow primarily a better perception of details (50%) and secondarily a better comprehension of represented objects shapes (34%). Quite the reverse, as many as the 75% of the C.B. affected people declared to prefer bas-relief thanks to a better perception of the shapes while only 6% declared to prefer it in terms of better detail perception. This can be explained by the fact that objects’ edges are smoother in bas-relief models while are generally quite irregular in flat-layered bas-relief ones. This second typology is indeed the one more closely resembling the original artwork (where outlines are actually irregular) and give the user the feeling of a more faithful representation. This aspect is particularly relevant for people who have experienced sight, while is far less important for congenital blinds. Figure 19(a) and Figure 19(b) show that the reason for discarding tactile outline and texturised pattern models seems to rely, both for C.B. and A.B.
affected people, on the objective difficulty (1 = less difficult, 5 = more difficult) in discriminating contours of objects and in perceiving object position in the 3D space. In particular, the average values in terms of difficulty in discriminating contours of objects for tactile outline resulted to be 4.13 and 4.33 for C.B. (red line) and A.B. affected people (blue line) respectively. Analogously, the average values in terms of difficulty in perceiving the object position in 3D space resulted to be 4.06 and 4.42 respectively. Similar objective difficulties (i.e., values greater than 4) occur for texturised pattern models. Additionally, texturised pattern proved even not to allow a proper discrimination of the objects in the scene; textures, that where expected to help the detection of shapes, generated on the contrary a misperception of represented objects. Finally, both bas-relief and flat layered bas-relief proved to allow a better comprehension of the composition of the represented scene, as demonstrated in Figure 19(c) where higher values are associated to higher difficulty.
Figure 19 (a) Difficulty in discriminating contours of objects (1 = less difficult, 5 = more difficult)
(b) Difficulty in discriminating position of objects in the 3D space (1 = less difficult, 5 = more difficult) (c) Clearness in understanding the composition (1 = less clearness, 5 = more clearness) (see online version for colours)
How difficult is to understand contours of objects? How difficult is to perceive object position in the 3D space
(a) (b)
For each representation (i.e., model), how clear is the composition of the represented scene?
5 Conclusions
Though some recent technologies, like 3D scanning and RP, have been successfully introduced to improve the 3D reproduction of sculptures, quite the opposite, no significant innovation can be found dealing with the realisation of 3D models starting from paintings or photographs. This is mainly due to the fact that there is a lack of high level systems and algorithms to create 3D models starting from 2D images; moreover, the criteria and methods to translate visual images into tactile language, thus meeting the needs of blind people, are not yet completely encoded at an international level. Moving from these considerations, the present work briefly described a series of computer-aided methodologies for the semi-automatic generation of four different kind of tactile 3D models (tactile outline, texturised pattern, flat bas-relief and bas-relief) starting from RGB digital images of paintings. The four kinds of tactile models have been realised 1 by modelling the painting subject (i.e., objects represented in the scene) using image
processing-based techniques integrated with CAD methods
2 by using well established RP techniques in order to physically produce tactile models.
Referring to modelling, hybrid and interactive algorithms based on the combination of watershed segmentation and Mortensen’s livewire methods proved to be effective for extracting consistent contours in RGB images so that the creation of tactile outlines model can be semi-automatically accomplished. Moreover, by using an appositely devised procedure, texture can be successfully superimposed on each cluster detected by the hybrid algorithm thus allowing the construction of a texturised pattern model. Further grey-level-based image processing of clustered images demonstrated its efficacy in creating flat layered bas-relief 3D models. Finally, modelling directly into CAD environment and reconstructing the 3D scene ‘as it would be for real’, showed its effectiveness in reproducing the paintings in the form of bas-relief. The devised methodology has been applied to the iconographic subject of ‘still life’, with particular reference to two masterpieces created by two modern painters: Fernando Botero and Giovanni Morandi. The physical prototypes have been evaluated by a panel of users in order to assess their effectiveness, according to user’s requirements and actual perception. Tactile assessment results show that the most effective strategies of developing a tactile representation of paintings are the flat-layered bas-relief and the bas-relief. Actually, the first model is preferred by acquired blindness people when a more faithful representation of the original artwork is an important requisite for the user. After all, the bas-relief is preferred by almost all users since it allows a better perception of details and an enhanced comprehension of represented objects shapes. It is authors’ opinion that the proposed study may offer useful information for anyone working in the field of tactile representation of paintings for blind people and, more precisely, for researchers willing to confront with the issue of automating the reconstruction process. It can also lay the bases for a systematic engineering procedure oriented towards a possible industrial process for translating into tactile models pictorial artworks. This could dramatically improve blind people’s accessibility to artworks. Moving from the results obtained so far, future work will be especially addressed to the improvement of the reconstruction of these two kind of models. In particular, the main aims are
1 to develop innovative computer-based algorithms capable to automate, as far as possible, the flat layered bas-relief generation
2 to translate paintings into bas-reliefs using semi-automatic procedures whereby, starting from the flat layered bas-relief representation, the shape of the objects is provided by their shading (e.g., shape from shading techniques)
3 to reconstruct different iconographic subjects, such as, for instance, realist paintings, in order to further investigate blind people tactile perception potential
4 to increase the number of experimental tests.
Acknowledgements
The authors wish to thank Prof. A. Quatraro for his valuable research assistance and all the persons of the Italian Union of Blind and Visually Impaired People in Firenze (Italy) who participated in this study. The authors wish to acknowledge the Tuscany Region (Italy) for co-funding the T-VedO project (PAR-FAS 2007-2013), which originated and made possible this research.
References
Allegro, M. (1991) L’educazione dei ciechi: storia concetti e metodi, Armando Editore, Roma,
Italy.
Bartolini, F., Carfagni, M. and Governi, L. (2004) ‘Model-based extraction of femoral medulla ducts from radiographic images’, Image and Vision Computing, Vol. 22, No. 3, pp.173–182. Beucher, S. and Lantuéjoul, C. (1979) ‘Use of watersheds in contour detection’, International
Workshop on Image Processing, Real-time Edge and Motion Detection, Rennes, France.
Carfagni, M., Furferi, R., Governi, L., Volpe, Y. and Tennirelli, G. (2012) ‘Tactile representation of paintings: an early assessment of possible computer based strategies’, Lecture Notes in
Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Vol. 7616, pp.261–270.
Furferi, R., Governi, L., Palai, M. and Volpe, Y. (2011a) ‘From unordered point cloud to weighted B-spline – a novel PCA-based method’, Applications of Mathematics and Computer Engineering – American Conference on Applied Mathematics, AMERICAN-MATH’11, 5th WSEAS International Conference on Computer Engineering and Applications, CEA’11,
pp.146–151.
Furferi, R., Governi, L., Palai, M. and Volpe, Y. (2011b) ‘Multiple incident splines (MISs) algorithm for topological reconstruction of 2D unordered point clouds’, International Journal
of Mathematics and Computers in Simulation, Vol. 5, No. 2, pp.171–179.
Grassini, A. (1998) ‘I ciechi e la formazione delle immagini: conoscere e valutare’, Convegno Alla
ricerca dei sensi perduti, Cosenza, Italy.
Grassini, A. and Scichillone, G. (1997) Dialogo nel buio. Una metafora dell’universo dei non
vedenti, Edizioni Quintilia, Roma, Italy.
Hatwell, Y. and Martinez-Sarocchi, F. (2003) ‘The tactile reading of maps and drawings and the access of blind people to works of art’, in Hatwell, Y., Streri, A. and Gentaz, E. (2003):
Touching for Knowing. Cognitive Psychology of Haptic Manual Perception, pp.255–273,
Benjamin Publishers, Amsterdam, The Netherlands.
Hernandez, S.E. and Barner, K.E. (2000) ‘Tactile imaging using watershed-based image segmentation’, Annual ACM Conference on Assistive Technologies, pp.26–33.
Jayant, C., Renzelmann, M., Wen, D., Krisnandi, S., Ladner, R. and Comden, D. (2007) ‘Automated tactile graphics translation: in the field’, ASSETS’07: Proceedings of the Ninth International ACM SIGACCESS Conference on Computers and Accessibility, pp.75–82.
Kardoulias, T. (2003) ‘Guidelines for making tactile diagrams and accompanying narratives’, in Salzhauer Axel, E. (2003): Art Beyond Sight. A Resource on Art, Creativity and Visual Impairment, pp.267–296, AFB Press, New York, NY, USA.
Kennedy, J.M. (2003) ‘Drawings from Gaia, a blind girl’, Perception, Vol. 32, No. 3, pp.321–340.
Kennedy, J.M. and Igor, J. (2003) ‘Haptics and projection: drawings from Tracy, a blind adult’,
Perception, Vol. 32, No. 9, pp.1059–1071.
Levi, F. (1993) Legittimità e utilità del disegno a rilievo’, Incontro Internazionale Strasburgo,
Strasbourg, France.
Levi, F. and Rolli, R. (1994) Manual of Tactile Graphics, Zamorani Editore, Roma, Italy.
Mortensen, E.N. and Barrett, W.A. (1996) ‘Interactive live-wire boundary extraction’, Medical Image Analysis, Vol. 1, No. 4, pp.331–341.
Nadernejad, E., Sharifzadeh, S. and Hassanpour, H. (2008) ‘Edge detection techniques: evaluations and comparison’, Journal of Applied Mathematical Sciences, Vol. 31, No. 2, pp.1507–1520. Quatraro, A. (2008) ‘La condizione di cecità ed ipovision’, in Cioppi, E. (2008): La scienza a
portata di mano, percorsi museali per non vedenti ed ipovedenti, pp.15–25, Polistampa,
Firenze, Italy.
Salzhauer Axel, E. (2003) Art Beyond Sight. A Resource Guide to Art, Creativity, and Visual
Impairment, AFB Press, New York, NY, USA.
Shapiro, L.G. and Stockman, G.C. (2001) Computer Vision, pp.279–325, Prentice-Hall,
New Jersey.
Teshima, Y. (2010) ‘Three-dimensional tactile models for blind people and recognition of 3D objects by touch: introduction to the special thematic session’, Lecture Notes in Computer
Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Vol. 6180, LNCS (Part 2), pp.513–514.
Turtle, M. (2012) [online] http://www.timetravelturtle.com/2012/07/tactual-museum-for-blind-athens (accessed 01/06/2012).
Wang, Z. and Li, B. (2010) ‘A Bayesian approach to automated creation of tactile facial images’,
IEEE Transactions on Multimedia, Vol. 12, No. 4, pp.233–246.
Ziou, D. and Tabbone, S. (1998) ‘Edge detection techniques – an overview’, International Journal
