A cockpit of multiple measures for assessing ﬁlm restoration quality Pattern Recognition Letters

(1)

ContentslistsavailableatScienceDirect

Pattern Recognition Letters

journalhomepage:www.elsevier.com/locate/patrec

A cockpit of multiple measures for assessing ﬁlm restoration quality ^R

Barbara Rita Barricelli

^a

, Elena Casiraghi

^b^,^∗

, Michela Lecca

^c

, Alice Plutino

^b

, Alessandro Rizzi

^b

a Department of Information Engineering, University of Brescia, Via Branze 38, Brescia 25123, Italy

b Computer Science Department, University of Milan, Via Celoria 18, Milan 20133, Italy

c Fondazione Bruno Kessler (ICT), via Sommarive 18, Trento 38123, Italy

a rt i c l e i nf o

Article history:

Received 18 July 2019 Revised 26 December 2019 Accepted 7 January 2020 Available online 8 January 2020 Keywords:

Image quality assessment Film restoration Features identiﬁcation

a b s t ra c t

Inmachinevision,theideaofexpressingthequalityofafilmsbyasinglevalueisverypopular.Usually thisvalueiscomputedbyprocessingasetofimagefeatureswiththeaimofresemblingasmuchaspos- sibleakindofhumanjudgmentofthefilmquality.Sincehumanqualityassessmentisacomplexmech- anisminvolvingmanydifferentperceptualaspects,webelievethatsuchapproachmayscarcelyprovide acomprehensiveanalysis.Especiallyinthefieldofdigital movierestoration,asinglescorecan hardly providereliableinformationabouttheeffectsofthevariousrestoringoperations.Forthisreasonwein- troducean alternativeapproach, where aset ofmeasures,describing overtimebasicglobaland local visualpropertiesofthefilmframes,iscomputedinanunsupervisedwayanddeliveredtoexpertevalu- atorsforcheckingtherestorationpipelineandresults.Theproposedframeworkcanbeviewedasacar orairplanecockpit,whoseparameters(i.e.thecomputedmeasures)arenecessarytocontrolthemachine statusandperformance.Thiscockpit,whichispubliclyavailableonline,wouldliketosupportthedigital restorationprocessanditsassessment.

1. Introduction

Filmrestorationisacomplexprocessdevotedtorescueandpre- serve the content of degraded films. Such a process should ac- countformanydifferentaspects,suchasthephysicalandchemi- calfeaturesofthefilm,thetypologyofthefilm,theculturalcon- textinwhichthefilmhasbeenproduced,andthetargetaudience.

Therefore,film restorationinvolves manydifferentfieldsofstudy, likephilology,history,chemistry,physics,opticsandcomputersci- ence. The plethora of different aspects that must be taken into account andthe lackof codified rules andshared methodologies [3,10]makefilmrestorationaverychallengingtask[8,9,48].

Digital Restoration, DR, applies two main steps: the “DR” per se, which minimizes or even removes artifacts like ﬂickering, scratches,dustandstabilizesthescenes([14],andthe“Color Cor- rection”,whichhandlescolouralterationandfading.Whilemostly usedrestoringsoftwareemployautomaticorsemi-automaticalgo- rithmsfor DR, the color correction task is still mainly manually performedby expertsvia semi-automaticmethods, thoughunsu- pervisedalgorithmshavebeenrecentlyproposed[5,36,37,46].

R Editor: Prof. G. Sanniti di Baja

∗ Corresponding author.

E-mail addresses: casiraghi@di.unimi.it , elena.casiraghi@unimi.it (E. Casiraghi).

DR presents several advantages with respect to the analog restoration.Firstofall,DRallowstotestandstudydifferent,eventually automatic, restoration procedures, without intervening on theanalogsupportwiththerisktodamagetheoriginaldata.Sec- ond,thedifferentsolutionsprovidedbyDRcanbecompared,giv- ing thepossibility tochoose thebest one orto combinethedif- ferent steps to obtainthe desired result. Third, theuse of image enhancersandcolorgradingalgorithmsworkingindependentlyon thecolorcomponentsenablesacolorcorrectionwhichismoreef- fective than that provided by the procedures generallyemployed inanalogrestoration,wherethecoloristreatedasasingleentity [4,11].Thisisthecaseoftheyellowisheffectduetotheaging,and mainlyaffectingtwoimagecomponentsintheRGBcolorspace.

Regardless ofthe waythe movieisrestored,the qualityofre- stored movieis an importanttask, generallyperformed by cura- torsandexpertsbasedontheirpersonalexperience,expertise,and skills.Theassessmentoftheintermediateresultsproducedbyeach step of the restoration process, allows the expert to assess the restoration pipelinedeciding eachtime ifa processingstepis ef- fective, and must thereforebe kept, orif it must be changed or removed. Therefore,the intermediate assessments,as well asthe assessmentofthefinal result,areimportantforthey allowtode- finetheoptimalrestorationpipeline.Unfortunately,qualityassess- ment of restoredmovies is a not “welldefined” task, for agreed https://doi.org/10.1016/j.patrec.2020.01.009

(2)

Table 1

Title of the ﬁlms, unique ID, cardinality, principal characteristics, and the exploited enhancers with the values of their parameters.

Film ID Cardinality

Main Characteristics Before

Restoration Applied Enhancers (Parameters)

Fiat 508 (1931) 1 888 Black and White movie Flickering

Dust and Scratches Silver Fading Phoenix (Manual) + ACE

( r(x ) = 2 x )

La lunga calza verde (1961) 2 363 Animation Movie Colour Fading STRESS ( M = 200 , N = 20 )

La Ciudad en la Playa (1961) 3 125 Super 8 movie Strong colour

fading Dust and Scratches RSR ( N = 200 , n = 20 ) I funerali delle vittime della

strage di Piazza Loggia (1974)

4 261 Super 8 movie Colour fading Dust

and Scratches Instability Phoenix + DaVinci Resolve ACE

( r(x ) = 5 x ) CLAHE applied on L ^∗ channel ( N = [0 , 01] )

quality standards are still lacking. At the state of the art, film quality assessment is generally performed by analyzing features describing either single frames or inter-frame relationships, and proposingoneorseveralnumericalestimatesoftheirqualityover time. Inthisway,filmqualityassessmentbecomesa problemre- latedto frame(image) quality assessment.Anyway, modelingthe featuresthatconcurtotheImageQuality(IQ)assessmentisadiffi- culttask,stillunderinvestigation[29]andforwhichaglobalstan- dardhasnotbeendefinedyet.

Based on the aforementioned considerations, in this work we presentanalternativeframeworkforsupporting expertusersdur- ing their objective assessmentof restored movies. Thiswork ex- tends that proposed in [32] by including a more comprehensive state ofthe art, explaining in depth the idea atthe basis ofthe cockpit,andprovidinganddiscussingtheWebapplicationMoReCo which allows employing the cockpit and evaluating its performance.Theproposedsetofno-referenceobjectivemeasuresisca- pable of describing the level of readability of the visual content oftheﬁlm andofquantifying theperceptualdifferencesbetween theoriginalandrestoredﬁlms.Thetermreadabilityhererefersto thehumanunderstandingoftheobservedsceneandofitsdetails.

Wemodelthisreadabilitybyﬁvepopularimagefeatures,mostof which alreadyemployed incomputer vision toassessthe perfor- manceofan image enhancer (e.g.,[21],[19]):thebrightness,the contrast, the colordistribution entropy, the local color variations ofeachframe,andperceptualintra-framecolorchanges.

Our framework basically acts as the cockpit of a machine, as a caroran airplane.The cockpitcomputes measures thatcan be viewedastheparameterscontrollingthevideostatusandtheper- formance ofthe restorationpipeline.Asithappensinthecockpit of an airplane, the quality measures are kept separated, andthe understandingoftheirdifferentsemanticmeaning(theconditions of differentcharacteristics of the airplane)is left to expertusers (the pilots), who merge them and form by themselves a global judgement about the system performance (the airplane) by also considering the current needs andtheir experience.Unifying the quality measures ina singleone maycauseinformationloss and would hide the causesof the obtainedassessment number. As a result, if the users wanted to improve performance, they would notknowwhichairplanecharacteristicstomodify.Similarly,inour applicationweletanyexpertuseranalyzetheprovidedmeasures separately, and interpretthem eventually considering other non- visualfeatures,ascontent,historicalperiod,filmemulsion,public, that mayevenchangeovertime. Thebasicideabehindthiswork istoput,atthecentreofthedatainterpretationprocess,theusers andnotthecomputer programsbecausethiswillassuretheirac- tive and engagedparticipation andwill exploit their expertisein the field [1]. To show the usefulness andreliability of the cockpit, and to lay the foundations for a future more complete film- assessmentcockpitexploitingandintegratingmorefilm-description

measures,wetestedtheframeworkondifferentscenesofﬁlmsre- storedthroughoneormoreamongﬁveimageenhancers.

The work is organized as follows: in Section 2 we shortly describe the background and the rationale behind our work;

Section 3 describesthe cockpit; in Section 4 we recall the enhancers employed for the restoration of the video used in the experiments; in Section 5 we describe the analyzed ﬁlms (see Table1)andtheobtainedresults;ﬁnally,inSection6,conclusions andfutureworksarereported.

2. Imagequality

As claimedin Section 1, the useof only one value for movie quality assessmentmay not sufficeto evaluate effectivelythe IQ, since the metric often accounts for few visual cues. This point emerges clearly in the wide survey reported in [27] and [29], where authors compare different assessment metrics, by report- ingthenumberofperformedtestsandthenumberofpersonsin- volvedinthetests.Thisverycarefulandmassiveworkunderlines theinner complexityofIQ assessment,andthe difficulties offit- tingandcompressingitinasinglevalue.

Infilm restoration thequality control isalways madeandsu- pervisedbythecurator,whodefinestherestorationpipelinebased on the consecutiveresults obtainedby theintermediate steps of therestoration process.Theassessment oftherestoringresultsis basedonthecurator’sskillsandexperienceandcontrolsaremade comparingthe restoredfilmswiththe original filmorwithfilms fromthesamehistoricalperiod,whentheoriginal ismissing.The lackoftrustworthyreferenceinfilmrestorationisoneofthemain limitsin the application ofthe so called Full-Reference (FR) and Reduced-Reference(RR)metrics[29].FR methodsassessthequal- ityofrestoration bycomparingtherestoredimagetotheoriginal referenceimage, whileRR methods comparesome features com- putedfromtherestoredimageto thosecomputedontheoriginal image.

Consideringtheclassiﬁcationproposedby[29],someexamples of FR metrics are: mathematically-based assessments, like Mean Square Error and Peak Signal to Noise Ratio; low-level metrics, likeSpatial-CIELAB[51];high-levelmetrics likeStructuralSIMilar- ity[52]andVisualImageFidelity[45].

Unfortunately,sinceevenfilmsstoredinperfectconditionsare subjecttodecayandaging,havingan originalreferenceofa film fromtheageinwhichitwasshotisreallyhard.Asaconsequence, No-Reference(NR) metrics,which donot need anyreference,are the most suited for this purpose. These metrics are mostly de- signed for capturing one or more predefined specific distortions types[26]. Theyare widely used forevaluating the performance ofimagecompression,dynamicmonitoringandadjustmentofIQ, image restoration and enhancement processes, andthus they al- lowoptimizingtheparametersettingsofdenoising,deblurringand

(3)

sharpening algorithms. Some examples are: Blind Image Quality Index[22], Distortion Identiﬁcationâ basedImageVerity andIn- tegrityEvaluation [24], Blind/reference less image spatial quality evaluator[23].

Nowadays, many NR metrics are based on the use of neural networks and, synthetically, computed a quality value based on the“experience” madeonasetoftrainingimages,e.g.,[13].Such methodsaretrainedoncommonlyusedtestimagesnoteffectively representingthe range andvariety ofdifferent scenes a director orartistmayhaverecorded[31].Furthermore,thefilmbasesand technologiesusedduring theyears, such asTechnicolor,Kinema- color,TintedandTonedfilms,presentspecificfeaturesandcolours that differ fundamentally from a natural pristine image. There- fore,inthe currentstage, thesubjectiveassessmentmadeby do- mainexpertscannotbe replacedby asingleIQmeasure, because the restoration process involves different enhancements on spe- cificfilmfeaturesthatcannotbesynthetizedbyonemetric.More- over,sinceIQcriteriamaychangeaccordingtoexternalnon-visual requirements, like aesthetic issues, content, and/or public, when workingwith machine learning methods, the non-informatic ex- pertuser must relabelthe trainingset each time thequality re- quirementschange.

3. Thecockpit

Based on the considerations expressed in Section 2 we pro- posetoreplacethecommonlyuseduniquescoreforIQassessment withamulti-score one.Precisely, ourcockpit consistsofﬁve no- reference, objectivemeasures that are computed frame by frame andthatcapturebasicvisualperceptualfeaturesrelatedtotheim- agereadability.

Speciﬁcally, let Ibe the frame ofa ﬁlm expressed inthe RGB colorspace,L^∗theluminancechannel oftheimageIexpressedin the CIEL^∗a^∗b^∗ colorspace, with a gamma of 2.2 andthe D65 as referenceilluminant.Themeasuresincludedinthecockpitarethe following:

1.meanintensityofall thepixels ineachchannel ofI(mI):this resultsina3DvectormI=[mR,mB,mG].

2. meanluminance(mL):itisthemeanintensityofallthepixels inchannelL^∗.

3. Multi-ResolutioncontrastofL^∗(MRC)[28,35,44].Thelocalcon- trastlc(x)ofL^∗atpixelxisthemeanvalueoftheL¹ distances betweenL^∗(x) andtheintensitiesoftheeightpixelssurround- ingxinthe3× 3windowcenteredonx.Themeanlocalcon- trastofL^∗isthe averageofall lc(x) foreach pixelxinL^∗.The multi-resolutioncontrastMRC ofL^∗istheaverageofthemean local contrasts of a set ofimages L^∗₀,...,L^∗_K K( > 1) obtained by sequentially scaling L^∗, with L^∗₀:=L^∗. In this work, K = 4 and the scaling factor applied to L^∗_i is computed so that the minimum dimension ofthe image L^∗₄ is not lessthan 25pix- els,whilethesizeofthethreeimagesinbetweenL^∗andL^∗₄ is uniformlydistributedbetweenthesizesofL^∗andofL^∗₄. 4. Deviance fromHistogram Flatness(HF):measures the entropy

of the distribution of L^∗ as the L¹ distance between the his- togramH(L^∗)ofL^∗,normalizedtosumupto1,andtheuniform probabilitydensityfunctionovertherangeofI,whichtypically is [0, 255]. Note that, in its original formulation [20], the HF wasappliedtotheR,G,Bchannelsseparately.However,inthis paper,similarto[19],weapplyittotheimagebrightness.

5. CoeﬃcientofLocalVariation(CLV):itisthemeanvalueofthe relative standard deviations

σ

⁽^x⁾ ^computed âtêach ^pixel ^x ôf

L^∗. Inparticular, we rememberthat

σ

⁽^x⁾ ^is ^the ^ratio^between

thestandarddeviationandthemeanofL^∗ ina 9× 9 window centeredatx,wherethemeandiffersfromzero.Inthiscompu- tation,themeanofL^∗isofcourseassumedtodifferfromzero,

andthe neighborhood of pixels adjacent to the image border isassumedhavingthesamevalueofthenearestimageborder.

ThevalueofCLVmeasuresthelocaldispersionoftheprobabil- itydistributionofL^∗andrelatesinformationofbrightnessand contrast.

6. ColorDifference(Ê^):îtîs^theintra-frame normalizedsumof colordifferencesCIEÊ^whichîs^the^L² ^distance^between^two consecutiveframesintheperceptualcolorspaceCIEL^∗a^∗b^∗.

The cockpit measures are very popular and basic perceptual features. They have been chosen among many others since they offerinteresting andintuitiveinformationaboutthequality ofan image and, according to our thesis, they must be considered all together inthequality assessmentprocess. Inparticular, wenote that the valuesof mL,MRC, HF effectivelycorrelate withthe hu- manjudgement, asshownbythesubjectiveexperimentsreported in[18]and[19].Insuch experiments,agroupofvolunteers were askedtolookatasetofpairs(J,E_J)ofimageswhereE_Jisan en- hancedversionofJ,andto choosetheimagethey consideredthe mostreadable.

Theresponses revealedthat inaboutthe60% ofthetests, the enhanced images, having a higherbrightness and contrastand a more uniform colordistributions than the input ones, were pre- ferred. When the imagesJandE_J were similar, in aboutthe 16%

ofthecases,thevolunteersdidnotexpressanypreference.Inthe remaining24%ofthecases,theypreferredtheinputimage,evenif itappeareddarkandwithpoorlyvisibledetails.Thislastoutcome wasmainlyduetoaestheticissuesortothepresenceofnoisethat waserroneouslyemphasizedbytheenhancer.Inthecockpit,noise isestimated bythe measureCLV that accountsforlocalintensity variations.

Thevalue ofÊîs^known^to^be ^related^to^the^human^percep- tion of the colors [50]. The time analysis of Ê ^may ^reveal ^the presence of flickering or abruptchanges ofthe scene color, that could indicate the presence of burned frames. In particular, the moreslowlyÊ^changes,^the^lower^the^flickering.

ItisimportanttonotethatexactvaluesofmI,mL,MRC,HF,CLV,

^E^foridentifyingareadableimage/videodonotexistsincethese measuresdependontheimage/videoitself.Forinstance,veryhigh valuesofmI,mLdonotnecessarilycorrespondtoareadableimage, sincetheycoulddescribeasaturatedimage,wherethedetailsare completelylost.ThevaluesofMRC,HF andCLVarehenceofgreat importance to evaluatethe statusof theIQ, butthey need to be analyzedalsoupon thecontentoftheﬁlm.Here theintervention oftheexpertusersisessentialtodecideifandhowtorestorethe visualmaterial.

Our approach differs greatly from the recently spreading approach,wheresomefeaturesdescribingimagequalitiesaremerged intoauniquemeasureofvisualquality.Anotableexampleofsuch approachesispresentedin[13],whereseventeenIQmeasuresare mergedthroughasupportvectorregressionmodel[7],trainedon asetofimagesautomaticallygeneratedandlabelled.Thoughinter- esting,thisapproachhasonemain weaknessduetothefact that theuserisprovidedwithauniquequalitymeasureandthemoti- vationsforthequalityassessmentresultsarenotprovided.Asare- sult,ifthissinglevaluesaysthattheIQispoorandtheuserswant toimproveit,they willneverbeableto understandwhichframe characteristics have caused the achieved quality measure, unless theyre-computebythemselvestheseventeenmeasures,insearch fortheimagefeatures tomodify.Moreover,asalreadymentioned inSection2,themachinelearningapproachesmustcopewiththe problemoftrainingvideostakingintoaccountalsonon-visualfea- turesthatmaychangeovertime.

(4)

Fig. 1. Plots of mL, mI, MRC (top), HF, CLV and E (bottom)for frames of the video “Fiat508 .

4. Imageenhancers

ThisSectiondescribesthealgorithmsusedforﬁlmrestoration.

Precisely, in Section 4.1 we brieﬂy report three Retinex inspired spatial color algorithms, namely the Automatic Color Equaliza- tionAlgorithm(ACE,[40]),RandomSprayRetinex(RSR,[34]),and Spatio-Temporal Retinex-Inspired Envelope with Stochastic Sam- pling (STRESS,[15]);inSection 4.2we describethe popularCon- trast Limited Adaptive Histogram Equalization (CLAHE, [25,53]) method. We have chosen these algorithms among many others sinceACE,RSR,andSTRESShavebeenalreadyemployedforrestor- ing, while CLAHE is very popular. Anyway, the cockpit might be appliedtoanyvideorestoredbymeansofanyotheralgorithm.

4.1. ACE,RSR,andSTRESS

ACE, RSR, and STRESS belong to the family of Spatial Color Algorithms (SCAs) [42], which recompute wavelength/energy ar- raysintocalculatedcolorappearancearrays,orpreferredcoloren- hancement arrays, according to the spatial distribution of pixel values inthe scene.Importantly, SCAshave theinteresting prop- erty of being unsupervised, meaning that they automatically ad- just colorandcontrast [42]. ACE, RSR, andSTRESSbelong to the Milano-Retinexfamily[33],proposingalternativeimplementations of Land’s Retinex model [16,17] and therefore inherit the prop- erty oflocalﬁltering.Moreover, astheoriginal Retinexalgorithm, when applied on RGB images, ACE, RSR and STRESS work separately onthe three color channels.Foreach target pixelp_t, with intensity I(pt), ACE computesa novel value by ﬁrstly considering the spatialcolor distribution of pixels p_neigh located in a circular neighborhood,ofradiusR,centeredonthetargetpixelpt.Thislo- cal processing, exploits a function r( · ), whoseinput is obtained by reworkingthe pairwisedifferences

|

^I(^pt)− I(^pneigh)

|

,weighted by the scaled Euclidean distance between the two pixels. After the local processing, a global rescaling maximizesthe image dy- namic. ACE has been successfullyapplied for ﬁne artprint [43], wallpainting[12]recovery,interfacesassessment[39],anddigital ﬁlmrestoration[6,38,41].

RSR computes the new pixel intensity of the target pixel pt

by considering a 2D-shaped circular neighbourhood of radius R around p_t. The intensityofp_t is rescaled withthe maximum in- tensityvaluechosenbysamplingtheneighborhoodofptbyNran- dom sprays, i.e.N sets of Mpixels randomly sampledaround pt

withradialdensity,andaveragingthereciprocalsofthemaximum intensities ofthe sprays.RSR hasproven tobe effectiveforcolor movierestoration[36].

STRESSinherits fromRSR the idea of a local whitereference, butimplementsaverydifferentpixelvaluestretching.Ratherthan consideringonlyalocalreferencewhiteforcomputinga“compen- sated” valuefor eachtarget pixel pt,it alsoconsiders a localref- erenceblack,sothattheintensityatptisstretchedbetweenthese references.Foreachtargetpixelp_t,itusesthesamespraysgener- ationofRSR,whereitstochasticallysamplesMpixelsforNtimes.

The newvalue forpt isthen stretched betweentwo values E_min andE_max computedbyre-working respectivelytheminimumand maximumintensities oftheN randomsprays sampledaroundpt. STRESShasbeensuccessfullyappliedforspatio-temporalcolorcor- rectionofmovies[47].

4.2.CLAHE

DevelopedasanadvancementoftheAdaptiveHistogramEqual- ization[30], CLAHEisa spatialdomain method.CLAHEsplits the imageintoNxNnotoverlapping regions,alsocalledtiles,wherea mappingfunction is derived as the cumulative distribution func- tionoftheclippedhistogramofthe tile.Clippingallows toavoid over-ampliﬁcationsin near-constantregions wherethe histogram ismostlyconcentrated. Sincethemappingisappropriate onlyfor thepixelatthecenterofthetile,themappingforeachotherpixel pisobtainedbyinterpolatingthemappingsofthetileswithcenter pixelsclosesttopitself.

5. Results

Tomeasurethereliabilityandusefulnessoftheproposedcock- pit,weanalyzedfourframesetsextractedfromfourdifferentﬁlms.

The cardinalityof theframe sets, their visual characteristics, and thetechniquesusedforrestoration,aredifferent.Precisely,theem- ployedenhancersare:ACE,RSRandSTRESS,CLAHE,andtwocom- merciallyavailable softwareproviding techniquesformanualﬁlm restoration,whicharethePhoenixsoftware[49],andtheDaVinci Resolve[2].ACE, RSR,STRESS,Phoenix, andDaVinciResolvewere appliedon theimage framesexpressed inRGBcolor coordinates, while CLAHE was applied only on the L^∗ channel of the image transformedintotheCIEL^∗A^∗B^∗colorspace.

The ﬁlm restorations have been performed during the past yearsattheMIPSLaboftheComputerScienceDepartmentatthe UniversityofMilan,Italy,forresearch studies.Table1reportsthe title of the ﬁlms fromwhich the frame sets were extracted, the frame-setcardinality,thecharacteristicsofdegradedframesbefore restoration,andthelistofappliedenhancersandtheirparameters.

(5)

Fig. 2. Plots of mL, mI, MRC (top), HF, CLV and E (bottom) for frames of the video “La Lunga Calza Verde”.

Fig. 3. Top panel: all the plots produced by the cockpit for ﬁlm “La ciudad en la playa”. Bottom panel: all plots produced on the restored ﬁlm: “I funerali delle vittime della strage di Piazza Loggia”, by ACE, CLAHE, and manual restoration.

NotethatACE andSTRESSuseasneighborhood thewholeimage.

Figs.1,2and3showsomeexamples¹

The ﬁlm“Fiat508

^(see^Fig.¹^),îs^blackând^whiteând^has^the

435th frame completely burnt and white,which causes a strong ﬂickering. The per-frame mI=[mR,mG,mB] and mL plots show thatDRandACEincrease theframebrightness;moreoverthemR andmGvalueshavebeenmadesimilarbyACE;nevertheless,both theenhancers still leave a bluedominantin thewhole scene.At thesametime,theper frameMRCplotindicatesthatcontrasthas

1 Figures with higher dimension are shown in the Supplementary material.

beenincreased. The valueof HFis increasedintherestoredﬁlm, whichisprobablyduetothecontrastincreaseresultinginthein- creaseofthehistogramentropy.

In “La Lunga Calza Verde” (see Fig. 2), the analyzed scene presentsananimation inwhichthesceneisuniformlycoloredin red,aroundthe80thand90thframes.Thischaracteristicisvisible inthemIplot,wherethemBandmGcurvesstronglydecreasein correspondence to those frames. STRESS increases the intensities ofthe threeRGB channels,remarkablyfadedin theoriginal ﬁlm, andresultsinacomparabledifferencebetweentheG,BandtheR channel.Moreoveritcreatesbrighterandmorecontrastedframes, asshownbytheplotsofmL,CLVandMRC.Simultaneously,thehis-

(6)

togram flatness is decreased,and the Ê ^values âre comparable, suggestingthatnoflickeringhasbeenintroduced.

The film “LaCiudad enLa Playa” (seethe toppanel of Fig.3) presentsaverystrongcolourfadingandtheapplicationofRSRwas afirstattempttorestoretheoriginalcoloursofthefilm.Although RSR increasedthe frameluminance mL,itincreasedboththe dif- ferencebetweentheR,GandBchannelsandtheinter-framevari- abilityoftheRchannel.ThevaluesofMRC,HF,andCLVhavebeen increased,indicatingthattherestorationhasrevealedmoredetails.

The increase of Ê ând ^the êmphasized ^difference ^between ^the mean of R, G, and B values, suggeststhat some color unbalance andflickeringmighthavebeenintroduced. Inthiscase,thecock- pitallowstooutlinetheprosandconsoftherestoration,andlets the userdecide whethertoproceed withtherestoration athand orstartagainusingothertechniques.

Thelast applicationofourcockpitismadeon “Ifuneralidelle vittime dellastragediPiazzadellaLoggia” (see thebottom panel ofFig.3).ThisSuper8documentarywasrestoredthroughthreedif- ferentenhancement methods:manual restorationby Phoenixand DaVinci,ACEandCLAHE.Manualrestorationdecreasedthelumi- nancemL andincreasedthecontrast MRC.ACE produced mLval- uesclosetotheoriginalvideo,whileCLAHEgreatlyincreasedthe mL values.Consideringthehighercontrast MRC,the performance ofManualrestorationandCLAHEaresimilar, whileACE produces the best results.This example showsthat different enhancement methodsproducedifferentresults.Inthisrespect,thecockpitmay beavaluabletooltoguideandsupporttheworkoftherestorerin identifyingthemethodthat providesthebestresultsfortheﬁnal purpose.

To let anyuser experience the proposed approach,we imple- mented thecockpitasaWebapplicationcalledMoReCo,fromthe keywordsMovieRestorationCockpit.MoReCoisavailableinanon- line version at thisWeb address http://159.149.129.182/moreco ² The application allows the user to upload up to two videos and then computes six charts: mI,mL, MRC, HF, CLV, andÊ^. Îfônly one video is uploaded, the generated charts show only the data relative to thatvideo; when two videosare uploaded,the gener- atedchartsshowacomparisonoftheresultsusinglinesindiffer- entcolorsandpatterns.Allgeneratedchartscanbedownloadedas PNGfigures.MoReCohasbeenimplementedusingHTML5,Chart.js, PHPandMatlabR2018a,andisdownloadableatthesameWebad- dressofitsonlineversion³

6. Conclusionsandfutureworks

Inthispaperwehaveproposedacockpitcomputingbasicmea- surestosupportexpertsdecisionsduringrestoredﬁlmassessment.

The ideaisto useacockpitasan alternativetoa singlevalue IQ metric,thatestablishesalevelofthegeneralstatusofthefilmun- derexambutthatdoesnotprovidespecificinformationaboutthe manyaspectsthatconcurtoformthatstatus.Anymethodprovid- ing a singleIQ measure, is considered by film editors and cura- tors as a black-box, which hidesthe real causesof the final out- put. Indeed, such a measure releases global information hard to beinterpreted.As aresult,userswishingtomodifytheprocessed videowouldnotbeabletounderstandwhichframecharacteristics shouldbechanged.

The application of thecockpit on restoration examples shows its characteristicandusefulness.TheMoReCocockpitisaprelimi- naryimplementation.TheWebapplicationallowstodownloadthe

2 Access credentials will be provided upon request to the corresponding author.

3 The current implementation of MoReCo can be extended with other features.

Suggestions, comments, requests, and proposals can be sent via the MoReCo application itself.

generatedchartsasPNGﬁgures;however,infutureversions,itwill beextendedtoallowuserstostoretheresultsforlateruses.

SincetheMoReCowebapplicationsallowsexperttosendtheir own proposals, our future works will be devoted to expand the cockpit by inserting more quality related measures suggestedby experts themselves, eventually clustering those with a similar meaning.Maybethiscouldfosternewopendatabasestotestnew setsofsharedqualityassessmentmeasures.

DeclarationofCompetingInterest

Theauthorsdeclarenoconﬂictofinterest.

Supplementarymaterial

Supplementary material associated with this article can be found,intheonlineversion,atdoi:10.1016/j.patrec.2020.01.009. References

[1] B.R. Barricelli, G. Fischer, D. Fogli, A. Morch, A. Piccinno, S. Valtolina, Cultures of participation in the digital age: From have to to want to participate, in: Proc.

of NordiCHI ’16, 2016. Pp. 128:1–128:3. doi: 10.1145/29714 85.298766 8 . [2] B. M. Design, Da Vinci Resolve 16, 2019. https://www.blackmagicdesign.com/it/

products/davinciresolve/ .

[3] V. Boarini , V. Opela , Charter of ﬁlm restoration, J. Film Preserv. 83 (2010) 37–39 .

[4] R. Catanese , Lacune binarie. il restauro dei ﬁlm e le tecnologie digitali, Bulzoni, 2013 .

[5] M. Chambah , B. Besserer , P. Courtellemont , Recent progress in automatic digital restoration of color motion pictures, in: Proceedings of SPIE - The Interna- tional Society for Optical Engineering, 2002, pp. 98–109 . doi:10.1117/12.452979 [6] M. Chambah , A. Rizzi , C. Gatta , B. Besserer , D. Marini , Perceptual approach for unsupervised digital color restoration of cinematographic archives, in: Color Imaging VIII: Processing, Hardcopy, and Applications, Santa Clara, CA , USA , Jan- uary 20, 2003, 2003, pp. 138–149 . doi:10.1117/12.472019

[7] C.C. Chang , C.J. Lin , LIBSVM: A library for support vector machines, ACM Trans.

Intell. Syst. Technol. 2 (2011) . 27:1–27:27

[8] A. Cornwell-Clyne , Colour Cinematography, Chapman and Hall, 1936 . [9] L. Enticknap , The culture and science of audiovisual heritage, Palgrave MacMil-

lan, 2013 .

[10] N. Fairbairn , M. Pimpinelli , T. Ross , The FIAF Moving Image Cataloguing Man- ual, Edited by Linda Tadic for the FIAF Cataloguing and Documentation Com- mission, 2016 .

[11] G. Fossati , From Grain to Pixel, Amsterdam University Press, 2009 .

[12] D. Gadia, C. Bonanomi, M. Marzullo, A. Rizzi, Perceptual enhancement of degraded etruscan wall paintings, J. Cult. Herit. 21 (2016), doi: 10.1016/j.culher.

2016.04.009 .

[13] K. Gu , D. Tao , J. Qiao , W. Lin , Learning a no-reference quality assessment model of enhanced images with big data, IEEE Trans. Neural Netw. Learn. Syst. 29 (2018) 1301–1313 .

[14] A. Kokaram , Motion Picture Restoration - Digital Algorithms for Artefact Sup- pression in Degraded Motion Picture Film and Video, Springer, 1998 . [15] O. Kolas, I. Farup, A. Rizzi, Spatio-temporal retinex-inspired envelope with

stochastic sampling: a framework for spatial color algorithms, J. Imaging Sci.

Technol. 55 (2011), doi: 10.2352/J.ImagingSci.Technol.2011.55.4.040503 . [16] E.H. Land, Color vision and the natural image. part i, Proc. Natl. Acad. Sci. 45

(1959) 115–129, doi: 10.1073/pnas.45.1.115 .

[17] E.H. Land , Experiments in color vision, Sci. Am. 200 (1959) 84–99 .

[18] M. Lecca, STAR: A segmentation-based approximation of point-based sampling milano retinex for color image enhancement, IEEE Trans. Image Process. 27 (2018) 5802–5812, doi: 10.1109/TIP.2018.2858541 .

[19] M. Lecca, S. Messelodi, SuPeR: Milano Retinex implementation exploiting a regular image grid, J. Opt. Soc. Am. A 36 (2019) 1423–1432, doi: 10.1364/JOSAA.

36.001423 .

[20] M. Lecca, A. Rizzi, Tuning the locality of ﬁltering with a spatially weighted implementation of random spray retinex, J. Opt. Soc. Am. A 32 (2015) 1876–

1887, doi: 10.1364/JOSAA.32.001876 .

[21] M. Lecca, G. Simone, C. Bonanomi, A. Rizzi, Point-based spatial colour sampling in milano-retinex: a survey, IET Image Proc. 12 (2018) 833–849, doi: 10.1049/

iet-ipr.2017.1224 .

[22] C. Li, A.C. Bovik, X. Wu, Blind image quality assessment using a general regres- sion neural network, IEEE Trans. Neural Netw. 22 (2011) 793–799, doi: 10.1109/

TNN.2011.2120620 .

[23] A . Mittal , A .K. Moorthy , A .C. Bovik , No-reference image quality assessment in the spatial domain, IEEE Trans. Image Process. 21 (2012) 4695–4708 . [24] A .K. Moorthy , A .C. Bovik , Blind image quality assessment: from natural

scene statistics to perceptual quality, IEEE Trans. Image Process. 20 (2011) 3350–3364 .

[25] N. Nahvi , D.P. Mittal , Medical image fusion using discrete wavelet transform, Int. J. Eng Res. Appl. 4 (2014) 165–170 .

(7)

[26] S. Ouni , H.M.Z.E. Chambah M. and , Are existing procedures enough? image and video quality assessment: review of subjective and objective metrics, Im- age Qual. Syst. Perform. V, edited by Susan P. Farnand, Frans Gaykema 6808, 68080Q (2008) .

[27] M. Pedersen , Image quality metrics for the evaluation of printings workﬂows, Doctoral Dissertation submitted at the Faculty of Mathematics and Natural Sci- ences at the University of Oslo (2011) .

[28] M. Pedersen , A. Rizzi , J.Y. Hardeberg , G. Simone , Evaluation of contrast measures in relation to observers perceived contrast, in: CGIV/MCS, 2008, pp. 253–258 .

[29] M. Pedersen , J. Yngve Hardeberg , Full-reference image quality metrics: classi- ﬁcation and evaluation, Found. Trends(R) in Comput. Graphics Vision 7 (2012) 1–80 .

[30] S.M. Pizer , E.P. Amburn , J.D. Austin , R. Cromartie , A. Geselowitz , T. Greer , B.T.H. Romeny , J.B. Zimmerman , Adaptive histogram equalization and its variations, Comput. Vision Graph. Image Process. 39 (1987) 355–368 .

[31] A. Plutino, M.P. Lanaro, S. Liberini, A. Rizzi, Work memories in super 8: search- ing a frame quality metric for movie restoration assessment, J. Cult. Herit.

(2019), doi: 10.1016/j.culher.2019.06.008 .

[32] A. Plutino , M. Lecca , A. Rizzi , A cockpit of measures for image quality assessment in digital ﬁlm restoration, in: M. Cristani, A. Prati, O. Lanz, S. Messelodi, N. Sebe (Eds.), New Trends in Image Analysis and Processing – ICIAP 2019, Springer International Publishing, Cham, 2019, pp. 159–169 .

[33] E. Provenzi, L.D. Carli, A. Rizzi, D. Marini, Mathematical deﬁnition and analysis of the retinex algorithm, J. Opt. Soc. Am. A 22 (2005) 2613–2621, doi: 10.1364/

JOSAA.22.002613 .

[34] E. Provenzi , M. Fierro , A. Rizzi , L. De Carli , D. Gadia , D. Marini ,Random spray retinex: a new retinex implementation to investigate the local properties of the model, IEEE Trans. Image Process. 16 (2007) 162–171 .

[35] A. Rizzi , T. Algeri , G. Medeghini , D. Marini , A proposal for contrast measure in digital images, Conference on Colour in Graphics, Imaging, and Vision 2004 (2004) 187–192 .

[36] A . Rizzi , A .J. Berolo , C. Bonanomi , D. Gadia , Unsupervised digital movie restoration with spatial models of color, Multimed. Tools Appl. 75 (2016) 3747–3765 . [37] A. Rizzi , M. Chambah , Perceptual color ﬁlm restoration, SMPTE Motion Imaging

J. 119 (2010) 33–41 .

[38] A. Rizzi , M. Chambah , D. Lenza , B. Besserer , D. Marini , Tuning of perceptual technique for digital movie color restoration, in: Visual Communications and Image Processing 20 04, VCIP 20 04, San Jose, California, USA, 18-22 January 20 04, 20 04, pp. 1286–1294 . doi: 0.1117/12.525789

[39] A. Rizzi, D. Fogli, B.R. Barricelli, A new approach to perceptual assessment of human-computer interfaces, Multimed. Tools Appl. 76 (2017) 7381–7399, doi: 10.1007/s11042- 016- 3400- 8 .

[40] A. Rizzi , C. Gatta , D. Marini , A new algorithm for unsupervised global and local color correction, Pattern Recognit. Lett. 24 (2003) 1663–1677 .

[41] A. Rizzi , C. Gatta , C. Slanzi , G. Ciocca , R. Schettini , Unsupervised color ﬁlm restoration using adaptive color equalization, in: Visual Information and In- formation Systems, 8th International Conference, VISUAL 2005, Amsterdam, The Netherlands, July 5, 2005, Revised Selected Papers, 2005, pp. 1–12 . doi:10.1007/11590064_1

[42] A. Rizzi, J. McCann, On the behavior of spatial models of color, Proc SPIE 6493 (2007), doi: 10.1117/12.708905 .

[43] A. Rizzi, C. Parraman, Developments in the recovery of colour in ﬁne art prints using spatial image processing, J. Phys. Conf. Ser. 231 (2010) 27–28, doi: 10.

1088/1742-6596/231/1/012003 .

[44] A. Rizzi , G. Simone , R. Cordone , A modiﬁed algorithm for perceived contrast measure in digital images, in: CGIV 2008 and MCS’08 Final Program and Pro- ceedings, 2008, pp. 249–252 .

[45] H.R. Sheikh, A.C. Bovik, Image information and visual quality, IEEE Trans. Image Process. 15 (2006) 430–4 4 4, doi: 10.1109/TIP.2005.859378 .

[46] A.B.M. Tariqul Islam , I. Farup , Spatio-temporal colour correction of strongly degraded movies, in: Proceedings of SPIE, Color Imaging XVI: Displaying, Process- ing, Hardcopy, and Applications„ 2011, p. 1 . doi:10.1117/12.872105

[47] A.B.M. Tariqul Islam , I. Farup , Spatio-temporal colour correction of strongly degraded movies, in: Color Imaging XVI: Displaying, Processing, Hardcopy, and Applications, 2011, p. 78660Z . doi:10.1117/12.872105

[48] P. Uccello , Cinema. tecnica e Linguaggio, Edited by San Paolo, 1997 . [49] D. Vision, Phoenix ﬁlm restoration software, 2019. http://www.digitalvision.tv/

products/phoenix _ ﬁlm _ restoration/ .

[50] R.F. Witzel, R.W. Burnham, J.W. Onley, Threshold and suprathreshold perceptual color differences, J. Opt. Soc. Am. 63 (1973) 615–625, doi: 10.1364/JOSA.63.

0 0 0615 .

[51] X. Zhang, B.A. Wandell, B.A. W, A spatial extension of cielab for digital color image reproduction, 1996.

[52] Z. Wang , A.C. Bovik , H.R. Sheikh , E.P. Simoncelli , Image quality assessment:

from error visibility to structural similarity, IEEE Trans. Image Process. 13 (20 04) 60 0–612 .

[53] K. Zuiderveld , Contrast limited adaptive histogram equalization, in: Graphics gems IV, Academic Press Professional, Inc., 1994, pp. 474–485 .