Appendix A
Fourier Ring Correlation Analysis
A.1 The spatial resolution in laser scanning microscopy
The spatial resolution stands as a critical parameter of any give microscopy image: its precise knowledge is paramount to extract correct biological conclusions from the experiment, whereas its over- or sub-estimation may lead to misinterpretations of the observed processes. In the context of laser scanning microscopy techniques, the resolution always depends on the optical/technical characteristics of the system (objective lens, working spectral region, system’s misalignments and aberrations, photon collection efficiency, etc.) and on the optical characteristics of the sample (thickness, refractive indices, scattering properties, etc.). Additionally, fluorophore tags contribute to the resolution of conventional microscopy techniques mainly in terms of brightness. Notably, when considering super-resolution techniques, many other photo-physical properties drive on the resolution. For example, considering stimulated emission depletion (STED) microscopy (Hell and Wichmann (1994); Vicidomini et al. (2018)), the transit absorption spectra (Hotta et al. (2010)), the photo-stability (Oracz et al. (2017)) and the lifetime (Vicidomini et al. (2012)) of the fluorophore are key parameters to achieve effective sub-diffraction resolution. Because of all these dependencies, the knowledge of the effective resolution of a nanoscopy experiment, although essential to enable sound biological conclusions, is uncertain (Li and Betzig (2016); Sahl et al. (2016)).
A purely theoretical resolution estimation - based on prior information about the system, the fluorophore and the specimen - can at best provide a rough estimate. Alternatively, resolution can be evaluated leveraging the usage of calibration samples, such as nanometer-scale fluorescence structures (e.g., beads), line-patterns or nanorulers (Schmied et al. (2012)). However, the resolution obtained via any calibration sample may not be in agreement with the
A.2 The Fourier ring correlation analysis 95 resolution achieved for the sample-of-interest, since sample characteristics, such as scattering, refractive index and fluorophore concentration are different. Furthermore, also in the case of identical fluorophores, the brightness, the photo-stability and the photo-physical properties may change depending on the environment of the sample. Thus, there is the need for a robust metric for estimating the effective resolution directly from the image of the sample-of-interest (image resolution) that would be sensitive to all the above-mentioned factors. Notably, the popular full-width at half-maximum (FWHM) criterion - i.e., the FWHM of the intensity line profile over isolated linear or punctuated nanometer sized structures within the image (such as tubulin filaments, vesicles, ribosomes, etc.) - is an attempt in this direction. However, the FWHM criterion may be considered incomplete, laborious and often overestimates the effective resolution: it is indeed easy to select line cuts narrowed by noise.
To address this need, we will introduce in the following Section the Fourier ring correlation (FRC) analysis, a complete and straightforward method which is sensitive to all factors that can affect the image resolution. The FRC analysis was introduced in the field of cryo-electron microscopy (Saxton and Baumeister (1982); Unser et al. (1987); van Heel and Schatz (2005)), and recently it has been successfully applied to the field of single-molecule-localization (SML) microscopy (Banterle et al. (2013); Nieuwenhuizen et al. (2013)) and point scanning microscopy (Tortarolo et al. (2018)).
A.2 The Fourier ring correlation analysis
It is possible to describe any given microscope - in the context of the image formation process - as a filter that attenuates the high spatial frequencies associated with increasingly fine features. Therefore, every raw microscopy image is a low-pass filtered representation of the specimen, that emphasizes coarse structures while blurring fine ones. Due to diffraction, a conventional microscope can be considered as a short-pass filter with a fixed cut-off frequency: the sample’s frequencies beyond the diffraction limit are not transmitted to the image. Conversely, a super-resolved microscope can theoretically transmit all the sample’s spatial frequencies to the image, since no cut-off-frequency is imposed by diffraction. In practice, however, different noise sources impose an effective cut-off-frequency, that is the highest sample’s frequency that emerges from the noise itself. The effective resolution can be defined as the inverse of such effective cut-off frequency. Since all factors that influence the resolution correspond to changes in the transmission of the frequencies and/or in the noise level, the FRC analysis is sensitive to all these factors.
A.2 The Fourier ring correlation analysis 96 noise realisations, the FRC analysis allows to retrieve the effective cut-off frequency of the images with no prior knowledge or calibration. In a nutshell, the FRC measures the degree of correlation of the two images at different spatial frequencies. The resulting curve is close to unity at low spatial frequencies; for spatial frequencies higher than the effective cut-off frequency, non-correlated (independent) noise realisations dominate and the curve approaches zero. The effective cut-off frequency is the frequency at which the correlation curve drops below a given threshold (Fig. A.1a).
In a laser point scanning microscope, the two independent images can be obtained by registering two sequential frames. However, any sample drift between the two frames would influence the FRC curve and lead to an underestimated resolution (Fig. A.2). We compensated for this problem by using a simple drift correction algorithm based on phase correlation, the same that we leveraged for assessing the shift values in the context of the Image Scanning Microscopy (see Chapter 3). We additionally explored more advanced acquisition approaches in order to retrieve the two independent images in an increasingly parallel fashion, e.g., line-by-line, pixel-by-pixel or pulse-by-pulse (in case of pulsed laser implementations). Notably, in this modality, the drift correction is no longer necessary (Fig. A.3).
In the following Sections we will analytically describe the FRC analysis, and present various results to demonstrate its robustness in the context of STED microscopy in particular and laser point scanning microscopy in general.
A.2 The Fourier ring correlation analysis 97
Figure A.1 The principle of Fourier ring correlation.(a) The correlation between the Fourier transforms of the two independent images over the perimeter of the circle of radius q is calculated, resulting in a FRC curve indicating the decay of the correlation with spatial frequency. The image resolution is the inverse of the spatial frequency for which the FRC curve drops below the threshold 1/7 ⇠ 0.143, so a threshold value at q = 9.2 µm 1 is
equivalent to 108 nm resolution (excitation beam power Pexc= 420 nW and STED beam
power PST ED= 60 mW). (b) FRC curves and (c) corresponding effective resolution values
for two fixed Pexc and increasing PST ED. Insets show magnified views (renormalized in
signal intensity) of the marked area (a) of the analyzed images for different Pexcand PST ED
combinations. Pixel-dwell time: 50µs. Pixel-size: 30 nm. Image format: 500 ⇥ 500 pixels. Scale bars: 1µm.
A.2 The Fourier ring correlation analysis 98
Figure A.2 Sample drift correction for the FRC analysis. To demonstrate the ability of the procedure for correcting the sample drift in the FRC analysis we collected pairs of consecutive frames (F1 and F2) with increasing drift. We mimicked the sample drift by
moving the sample between the collection of the two frames. We moved the sample along the x-direction by means of the 3D piezo-stage. In addition we registered the two frames in the so called pixel modality: we divided the pixel-dwell time in two identical temporal windows and we registered the fluorescence photons in synchronization with the two windows. This acquisition modality allows to obtain two independent iamges from each frame (F1,1, F1,2
form the first frame and F2,1, F2,2from the second frame). Since the images are collected
"simultaneously", the drift of the sample can be neglected. Very important, since bleaching from the two frames is negligible the four images have similar SNR. By applying the FRC analysis on specific pairs of images (F1,1, F1,2) and (F1,1, F2,1) it is possible to calculate the
FRC curves in the case of negligible sample drift and (imposed) sample drift, respectively. We also performed the FRC analysis of the pair of drifted images (F1,1, F2,1) but including
the procedure for drift correction. The comparison of the FRC curves for increasing sample drift values (24 nm, 34 nm, 53 nm, and 61 nm respectively for a,b,c, and d) clearly shows the artifacts introduced by the drift, but also the ability of the drifting correction procedure to recover free-drift FRC curves. Since the SNR of the four images are similar the FRC curve on pixel-based images and on the frame-based images corrected for the sample drift are identical. Right-side images show details of the two images extracted by the two frames (F1,1, F2,1) before the drift correction. By overlapping the two details with two different
color-maps it is possible to highlight the increasing drift. Excitation beam: 635 nm at 80 MHz. STED beam: 775 nm at 80 MHz. PST ED= 52 mW. Pexc= 300 nW. Pixel-dwell time
A.2 The Fourier ring correlation analysis 100 Figure A.3Strategies to obtain simultaneously a pair of independent images. The most straightforward and general approach to obtain two independent images of the sample-of-interest is to collect sequentially two frames (frame-by-frame modality in short frame modality). Clearly, this strategy works only when the sample drift between the two frames is negligible. Whilst a solution to this problem can be obtained by using correction drifting pro-cedures, a more elegant solution consist in the parallelization of the image registration, i.e. in the implementation of acquisition modalities which collects the two image "simultaneously" (a). These strategies can also compensate for slow motion of the sample, which can not be recovered by a drift correction procedure. Here, we demonstrates three different strategies with increasing parallelism, namely, line-based (line-by-line), pixel-based (pixel-by-pixel), and histogram-based (bin-by-bin) acquisitions (b). The line-based modality (b, left), in which every line of a single frame is scanned twice; the pixel-based modality (b, center), in which every pixel of the frame (pixel-dwell time) is divided in two identical time windows, and fluorescent photons are sorted accordingly; the histogram-based modality (b, right), in which a time-correlated single-photon-counting (TCSPC) card is used to obtain the photons arrival-time histogram and the two images are obtained from photons related to even and odd temporal bins, respectively. To validate these approaches and to compare with the general frame-based approach, we acquired two consecutive frames (F1 and F2) using the
acquisition approaches described above. This procedure allows to obtain two "simultaneous" images for each-frame (F1,1, F1,2 from the first frame and F2,1, F2,2 from the second frame).
We applied the FRC analysis on different combinations of images and we compared the proposed "parallel" modalities with the frame-based modality properly corrected for drift. The first line of the FRC curves shows the results obtained for confocal experiments. In this case the sample drift of all the microscopes used is negligible, at least with respect to the diffraction-limited resolution. The second line of the FRC curves shows the results obtained from the same microscope architecture but for STED microscopy experiments. In this case the drift is not anymore negligible, i.e. the FRC curves obtained from the frame modality need to be corrected for the sample drift to match with the FRC curves obtained from the proposed parallel registration modalities. Line-based modality experiments have been obtained using the Leica TCS CW-STED microscope (Excitation beam: 488 nm at 80 MHz. STED beam: 595 nm CW. Time-gated delay: 1.2 ns. Pixel-size: 30 nm. Format: 512 ⇥ 512 pixels). Pixel-based modality experiments have been obtained using the custom-made pulsed STED microscope (Excitation beam: 635 nm at 80 MHz. STED beam: 775 nm at 80 MHz. PST ED= 50 mW. Pexc= 560 nW. Pixel-dwell time for the frame: 50µs. Pixel-size: 25
nm. Format: 600 ⇥ 600 pixels). Histogram-based modality experiments have been obtained using the custom-made gated CW-STED microscope (Excitation beam: 488 nm at 80 MHz. STED beam: 575 nm CW. PST ED= 50 mW. Pexc= 560 nW. Pixel-dwell time for the frame:
50 µs. Pixel-size: 25 nm. Format: 600 ⇥ 600 pixels.). Insets show details of the original frames. Scale bars: 1µm.
A.3 Theoretical description 101
A.3 Theoretical description
The Fourier ring correlation (FRC) measures the normalized cross-correlation coefficient between two bi-dimensional images over corresponding rings (qi) in the Fourier space, i.e.,
the FRC is a function of the spatial frequency (qi) and is calculated as follows:
FRC(qi) = Âfx,fy2 qi
G1(fx,fy)· G⇤2(fx,fy)
q
Âfx,fy2 qi|G1(fx,fy)|2Âfx,fy2 qi|G2(fx,fy)|2
, (A.1)
where G1 and G2 are the (discrete) Fourier transforms of the two images g1 and g2, i.e.
F (g) = G. It is important to remark that the FRC(qi)is always real.
The termÂfx,fy2 qiG1(fx,fy)G⇤2(fx,fy)is real, if g1and g2are real, because then it holds that
G( fx,fy) =G⇤( fx, fy)and in each term of the sum on the circle the complex parts cancel
out. Since the definition of discrete Fourier transform (DFT) assumes that the (finite) signal is infinitely repeated in the space, false edges can be created. For this reason, a window function is applied to the two raw-images before calculating their DFT avoiding the creation of false high-frequencies that may result in spurious correlation. In this work, a Hann window was used. Hann window is a particular form among the generalized Hamming windows, defined as follows:
w(n) =a b˙cos 2pn
N 1 , (A.2)
wherea = b = 0.5 and N is the number of element on the mask. The mask was applied in the x and the y directions of the images. The FRC curve is often quite noisy, for this reason it was smoothed with a simple moving average filter (with a half-width of the average window equal to 3 frequency bins.).
A specific consideration is needed to define the discretization of the spatial frequencies of the resulting FRC curve, in particular to calculate the (discrete) values of the frequencies the cross-correlation values belongs to. It is well known that the maximum frequency ( fmax) is
the half of the inverse of the pixel size (ps):
fmax= 2p1
s (A.3)
In the case of a squared image (Nx = Ny = N), the FRC will be composed by N/2 values,
thus:
D f = fmax N/2 =
1
A.4 FRC on STED microscopy: a test case 102 If the images are not squared, they have to be zero-padded to perform the calculation. Given the FRC curve, it is necessary to define a criterion to find the effective frequency cut-off. The most used criterion choses as effective cut-off frequency the frequency for which the FRC curve drops below a given threshold. Determining a threshold for the FRC remains a controversial issue: fixed-value thresholds were argued to be based on incorrect statistical assumptions. Among the variable-threshold criteria, the 3-s and the 5-s were defined. Their goal is to indicate where the FRC systematically emerges above the expected random correlations of the background noise. The so-called sigma-factor curves are defined as follows:
si=psf actor
Ni/2
, (A.5)
wheres Nithe number of pixel contained in the ring of radius qi; the extra factor of “2” is
required since the FRC summations include all Hermitian pairs in Fourier space. The most usedsf actor are 3 and 5 and the obtained curves are called respectively 3- and 5-s. Among
the fixed threshold methods, the correlation value of 1/7 is widely used and reported in the literature. We found that, for what concerns this context of application, the final resolution obtained with whatever criterion is approximately the same. For this reason and in analogy with what done in the work on single molecule localization microscopy (Nieuwenhuizen et al. (2013)), we used a threshold fixed to 1/7. Furthermore, this threshold methods provides resolution values compatible with the visual inspection of nanorulers imaging. After defining the criterion, the resolution can be defined as the inverse of the first frequency value for which the correlation is under the threshold. In the frequencies above this value, the noise dominates over the signal.
A.4 FRC on STED microscopy: a test case
In this Section I will present various results we obtained to validate the FRC analysis in the context of STED microscopy. We started by imaging microtubules with a pulsed STED microscope: the FRC metric clearly reveals the diffraction-unlimited but noise-limited nature of STED microscopy (Fig. A.1b): for a fixed excitation beam power Pexcand for increasing
STED beam power PST EDthe image resolution initially improves until a value well below the
diffraction-limited value of the confocal counterpart; the noise at high-frequencies starts to dominate and the image resolution degrades with increasing PST ED. In essence, for increasing
STED beam power, the stimulated emission probability increases, but the signal-to-noise ratio (SNR) decreases, due to, for example, non-perfect "zero" intensity point of the doughnut
A.4 FRC on STED microscopy: a test case 103
Figure A.4Comparison between different FRC threshold methods. We compared the FRC-based effective resolution using the most common threshold methods. The 1/7 threshold and the 3-s threshold methods show almost identical results. As expected the 5-s is the most conservative threshold. Clearly, the differences between the threshold criteria increase for reducing image formats, here we used 500⇥500, which is a typical format value. or background signal induced by the STED beam. As expected, it is possible to recover the SNR and increase the effective image resolution by increasing the intensity of the excitation beam. In this work, we used the 1/7 fixed threshold to obtain the effective cut-off frequencies, but we achieved similar results also using the 3- and the 5-s thresholds (Fig. A.4).
A.4 FRC on STED microscopy: a test case 104 We continued and compared the FRC metric with the most common resolution metric used in STED microscopy, i.e., the FWHM of the Gaussian (or Lorentzian) fit of the image of punctuated and isolated nanometer-sized structures (sometimes referred as the FWHM of the point-spread-function (PSF)). Since not every sample contains punctuated structures, the method is usually applied on the image of a sparse distribution of fluorescent beads. However, fluorescent beads are usually characterized by superior brightness and photo-stability, thus the estimated resolution may not be accurate for the sample-of-interest. We compared the resolution obtained via FRC and FWHM analysis by imaging samples of 60 nm sized Crimson fluorescent beads (Fig. A.5 and Fig. A.6).
The resolution estimates obtained with fixed STED beam power and increasing excitation power reveal the higher noise sensitivity of the FRC compared to the FWHM analysis; the resolution estimated via the FWHM analysis is almost constant for every SNR condition (power of the excitation beam). This can be explained considering the absence of any prior information in the FRC analysis, information that is, on the other hand, imposed when considering a Gaussian model (or Lorentzian) for fitting. The higher noise sensitivity of the FRC metrics with respect to the FWHM metrics is also demonstrated by plotting the image resolution as a function of increasing STED beam power (Fig. A.5b): the improvement of resolution according to the FRC metric is slower compared to the FWHM metric, that is due to the simultaneous SNR reduction. The differences between the FWHM and FRC metrics for lower STED beam power (higher SNR conditions) can be attributed to the choice of the statistical model, Gaussian or Lorentzian, and its parameters fitting, and/or to the choice of the threshold criterion used for the FRC analysis. We then continued the validation of the FRC technique by imaging nanorulers (or DNA origami), emerging tools for characterizing the resolution in STED microscopy and other nanoscopy techniques (Schmied et al. (2012)). A nanoruler consists on a pair of clusters of fluorophores located at a well known and precise distance. If the STED image is able to reveal the two clusters, the image resolution is higher than the nanoruler nominal length. We applied the FRC analysis on STED images of nanorulers with different spacing and increasing STED beam power. The FRC metric is in tune with the visual estimation, as it is only possible to discern two adjacent spots that are at a distance above the retrieved resolution (Fig. A.7). Lastly, we used nanorulers also to demonstrate the sensitivity of the FRC metrics to the SNR reduction induced by photo-bleaching (Fig. A.8).
The high sensitivity to the SNR and the low-computational cost (mainly two fast Fourier transforms) of the FRC analysis make this metric a perfect candidate for implementing real-time and auto-alignment tools for STED microscopy. The performance of any STED
A.4 FRC on STED microscopy: a test case 105
Figure A.5FRC analysis on fluorescent beads. FRC curves for imaging of 60 nm crimson fluorescent beads at a fixed STED beam power PST ED= 12 mW and increasing excitation
beam powers Pexc(a) and for a fixed Pexc= 560 nW and increasing PST ED(b). The comparison
between the FRC- and FWHM-based resolution as function of Pexc(a) and PST ED(b). Insets
show portions of the analyzed images for different combinations of Pexcand PST ED.
Pixel-dwell time: 50µs. Pixel-size: 40 nm. Format: 1000 ⇥ 1000 pixels. Scale bars: 1 µm. microscopy architecture strictly depends on its ability to match to different spatial, temporal and spectral conditions (Vicidomini et al. (2018)). For example, the Gaussian excitation and the doughnut-shaped depletion intensity distributions at the focus should overlap to obtain the best SNR (Gould et al. (2013)) and thus the best resolution (Figure A.9a). Furthermore, in the case of pulsed STED microscopy implementation, the depletion pulses need to follow immediately the excitation pulses to obtain the smallest fluorescent confinement (Vicidomini et al. (2013)), thus the best resolution (Figure A.9c). The FRC analysis reveals both benefits and limits of the time-gated detection: the registration of the signal only after the STED
A.4 FRC on STED microscopy: a test case 106
Figure A.6Application of the FRC analysis on imaging of 60 nm fluorescent beads with confocal microscopy. (a) The FRC curves as function of the excitation beam power Pexc.
show the diffraction-limited resolution of the confocal microscope. In particular, the effective resolution measured with the FRC metric (b) is limited to 240 nm also for increasing Pexc,
thus high SNR. The comparison of the FRC-based and the FWHM-based effective resolution shows confirms the higher noise sensitivity of the FRC metric respect to the FWHM metric. Insets show magnified details (renormalized in signal intensity) of the analyzed images for different Pexc. Excitation beam: 635 nm at 80 MHz. Pixel-dwell time: 50µs. Pixel-size: 40
nm. Format: 1000 ⇥ 1000 pixels. Scale bars: 1 µm.
beam action allows to compensate for temporal misalignment and to use sub-nanosecond pulses (Castello et al. (2016), Fig. A.9d), but the longer is the delay between excitation and depletion, the lower is the SNR and thus the resolution (Figure A.9e).
A.4 FRC on STED microscopy: a test case 107
Figure A.7FRC analysis for nanoruler imaging. FRC analysis on imaging of two different nanoruler lengths (90 nm and 160 nm) for increasing PST EDand Pexc= 750, 1800, 3000 nW.
Portions of the analyses images representing an isolated nanoruler are reported together with the FRC-based resolution. Pixel-dwell time: 100µs. Pixel-size: 20 nm. Format:500 ⇥ 500 pixels. Scale bars: 100 nm.
A.4 FRC on STED microscopy: a test case 108
Figure A.8 Influence of photobleaching on the FRC analysis. We collected seven con-secutive STED microscopy frames (Fi : i = 1,..,7) of 90 nm length nanorulers (a). The
dwell-time of each pixel is spitted in two identical windows in order to generate from each frame (Fi) two independent images (Fi,1,Fi,2), i.e. we acquired the frames using the so-called
pixel modality. The insets show a magnified view of the marked areas, renormalized in signal intensity. (b) FRC curves obtained analyzing each consecutive frame using the pixel-modality to obtained the two independent images ((F1,1,F1,2), .., (F7,1,F7,2)). In this case difference
due to bleaching from the two images is negligible. All the FRC curves start from one, clearly indicating the same average intensity for the two images. On the other hand, because of photo-bleaching the SNR for increasing frames reduces and as a consequence also the effec-tive resolution reduces (c). It is also interesting to calculate the FRC curves and estimate the effective resolution using as first image the one obtained from the first frame and as second image the one obtained from the increasing frames (((F1,1,F2,1), .., (F1,1,F7,1)). In this case,
the FRC curves does not start anymore from one, indicating that the photo-bleaching between the two image can not be neglected, but the effective resolution values are comparable with the value obtained using the pixel modality. This behavior is expected: nevertheless the first image has high resolution (since higher SNR) the second image reduce in SNR due to bleaching, thereby the high frequencies of the higher resolution image does not correlate with the high frequencies of the lower resolution image. This experiment demonstrate the conservative nature of the FRC-metric in estimating the effective resolution. Excitation beam: 635 nm at 80 MHz. Pexc = 4.2 µW. STED beam: 775 nm at 80 MHz. PST ED = 60 mW.
Pixel-dwell time: 100µs. Pixel-size: 20 nm. Format: 500 ⇥ 500 pixels. Scale bars: 1 µm, 100 nm (insets).
A.4 FRC on STED microscopy: a test case 109
Figure A.9 The FRC curves (a), the crimson beads (60nm) images, and the focal intensity distribution for the STED (red) and excitation (green) beams for an aligned (left) and a mis-aligned (right) configuration (b). (c) Details of STED images obtained with increasing delay (Dt) between the excitation pulses and the depletion pulses for conventional (top) and time-gated detection (bottom), and the related FRC curves (d). Temporal scheme of the pulses and the detection is reported. (e) The FRC-based resolution analysis as function ofDt both for the conventional and time-gated detection. PST ED= 38 mW (a,b) and 24 mW (c,d,e).
Pexc= 330 nW (a,b) and 560 nW (c,d,e). Pixel-dwell time: 50µs. Pixel-size: 20 nm (a,b)
