MLIC-Synthetizer: a Synthetic Multi-Light Image Collection Generator

(1)

STAG: Smart Tools and Applications in Graphics (2019) M. Agus, M. Corsini and R. Pintus (Editors)

MLIC-Synthetizer: a synthetic Multi-Light Image Collection

Generator

T. G. Dulecha, A. Dall’Alba, and A. Giachetti

Department of Computer Science, University of Verona, Italy

Abstract

We present MLIC-Synthetizer, a Blender plugin specifically designed for the generation of a syntethic Multi-Light Image Collection using physically-based rendering. This tool makes easy to generate large amount of test data that can be useful for Photometric Stereo algorithms evaluation, validation of Reflectance Transformation Imaging calibration and processing method, relighting methods and more. Multi-pass rendering allows the generation of images with associated shadows and specularity ground truth maps, ground truth normals and material segmentation masks. Furthermore loops on material parameters allows the automatic generation of datasets with pre-defined material parameters ranges that can be used to train robust learning-based algorithms for 3D reconstruction, relight and material segmentation.

CCS Concepts

•Computing methodologies → Computer graphics; • Software and its engineering → Software notations and tools;

Multi-light image collections (MLICs) are an effective mean to gather detailed information on the shape and appearance of objects. For this reason, lots of visualization and analysis methods are built upon this kind of data, e.g. multiple images of the surface of inter-est captured from a fixed point of view, changing the illumination conditions (typically light direction) at each shot [PDC∗19]. In particular Photometric Stereo approaches [AG15] and Reflectance Transformation Imaging/Relightable images [MGW01,GCD∗17] are quite popular, with many applications, especially in the Cultural Heritage domain [Mac15]. MLIC have been used also for material segmentation tasks, e.g. in [WGSD09].

The development and validation of algorithms for image relight-ing (e.g. generation of images of the surface with arbitrary illumi-nation), normal reconstruction and 3D shape estimation, material segmentation, is not easy to perform as the acquisition methods used in the practice have relevant calibration issues. Furthermore, learning based approaches for these tasks are emerging [XSHR18,RDL∗15], requiring lots of annotated images, with known lights, shape and/or materials. For these reasons, an easy generation of custom sets of realistic data with known parameters would be extremely useful in order to create datasets for evaluation of different kinds of tools and training of different kinds of learning-based algorithms.

1. MLIC-Synthetizer

Exploiting the Cycles Physically-Based rendering and the widely used Blender interface, we developed a specific plugin to easily gen-erate MLIC data. The key features of the plugin are the following: • Use of exactly directional lights, with sets of lights loaded from

.lp files (use of point or spot lights could be applied to evaluate calibration issues).

• Loop over material parameters: this feature allows the generation of rendered objects of different materials within specified ranges of material parameters. This will be a fundamental requirement for the generation of training sets for material classification and segmentation

• Export of multiple images exploiting the different Cycles render-ing passes: composite image, Specularity image, shadow image, material indexes. This will be useful in order to develop methods for shadows detection and/or robust fitting of a matte component. Figure1shows the Blender interface with the plugin loaded and en-abled. The plugin is available at the web sitehttps:github.com/ giach68/SyntheticRTI. GUI components allow the creation of the virtual dome with directional lights just loading the file and the automatic creation of the animation with the sequential light activation. Once loaded, lights could be modified (e.g. transforming into spot lights, changing intensity and color). Another button gen-erates the dome camera, that can then be customized to obtain the desired resolution, depth, color mapping, etc. An arbitrary number of loop parameters can then be generated to loop on materials prop-erties. This allows the automatic generation of a large number of images with different, known material parameters. Figure2shows an example with a single loop variable linked to material specularity. Clicking on the "render images" button multiple RTI datasets with different materials are generated. Users can freely configure the camera and rendering options and the images can be exported with linear or nonlinear color correction and different depths. Ground

(2)

T. Dulecha et al. / MLIC-Synthetizer

truth values for normals, specularity maps and material masks can be exported as well.

Figure 1: Blender 2.8 interface with the MLIC-Synthetizer plugin. 2. Example uses

In order to show a first simple use of synthetic data, we present a simple dataset to test relight, Photometric stereo and material segmentation methods. We created a simple near-planar object us-ing the Blender sculptus-ing functions and created a set of 49 light directions simulating a dome configuration. To demonstrate material loops, we rendered the MLIC data assigning to the object material with specularity from 0 to 1 in 5 steps. In this way, we can evaluate the accuracy of normal estimation as a function of the changing parameter. Figure3shows in the top row examples of the surface illuminated from top with different specularity values (0,0.5,1). The accuracy of the normals estimated with Photometric Stereo on our datasets can be compared with the ground truth provided by the rendering engine (bottom left). We can thus estimate normal errors variations with material specularity (bottom right).

We also rendered a MLIC of the same surface splitting the object into 4 quadrants assigned 4 slightly different materials assigned to test material segmentation methods. We applied segmentation methods based on pixel-level feature classification. We followed the protocol of [WGSD09], training a classifier on 5% of the image pixels and evaluating results. Figure4shows the results obtained with the different methods (see caption).

Figure 2: The user interface allows the creation of set of parameters that can be linked to material features on the material node. Here a loop variable creates a set of dataset with different specularity.

These examples show the usefulness of the tool for algorithms validation, however, future relevant use of the tool will be also the creation of large sets of training data for learning based shape and material estimation tasks.

Figure 3: Top: three images from different MLICs with similar lighting and different materials. Bottom: ground truth normals and normal estimation quality (MSE) vs. specularity.

Figure 4: Material segmentation using MLIC. First row: single image, ground truth mask, segmentation based on a single image and min-mean-max of the image stack, respectively. Second row: segmentation using features from [GCD∗17], segmentation based on PTM coefficients, HSH coefficients and rotated HSH coefficients [WGSD09], respectively.

References

[AG15] ACKERMANNJ., GOESELEM.: A survey of photometric stereo techniques. Foundations and Trends in Computer Graphics and Vision 9, 3-4 (2015), 149–254.1

[GCD∗17] GIACHETTIA., CIORTANI. M., DAFFARAC., PINTUSR., GOBBETTIE.: Multispectral RTI analysis of heterogeneous artworks. In Proc. GCH(2017).1,2

[Mac15] MACDONALDL. W.: Realistic visualisation of cultural heritage objects. PhD thesis, UCL (University College London), 2015.1

[MGW01] MALZBENDERT., GELBD., WOLTERSH.: Polynomial tex-ture maps. In Proc. SIGGRAPH (2001), pp. 519–528.1

[PDC∗19] PINTUSR., DULECHAT., CIORTANI., GOBBETTIE., GIA -CHETTIA.: State-of-the-art in multi-light image collections for surface visualization and analysis. In Computer Graphics Forum (2019), vol. 38, Wiley Online Library, pp. 909–934.1

[RDL∗15] RENP., DONGY., LINS., TONGX., GUOB.: Image based relighting using neural networks. ACM TOG 34, 4 (2015), 111:1–12.1

[WGSD09] WANGO., GUNAWARDANEP., SCHERS., DAVISJ.: Mate-rial classification using brdf slices. In 2009 IEEE Conference on Computer Vision and Pattern Recognition(2009), IEEE, pp. 2805–2811.1,2

[XSHR18] XUZ., SUNKAVALLIK., HADAPS., RAMAMOORTHIR.: Deep image-based relighting from optimal sparse samples. ACM TOG 37, 4 (2018), 126:1–126:13.1