To enhance the resolution of a confocal laser scanning microscope the additional information of a pinhole plane image taken at every excitation scan position can be used (Sheppard 1988). This photon reassignment principle is based on the fact that the most probable position of an emitter is at half way between the nominal focus of the excitation laser and the position corresponding to the (off centre) detection position. Therefore, by reassigning the detected photons to this place, an image with enhanced detection efficiency and resolution is obtained. Here we present optical photon reassignment microscopy (OPRA) which realizes this concept in an all-optical way obviating the need for image-processing. With the help of an additional intermediate optical beam expansion between descanning and a further rescanning of the detected light, an image with the advantages of photon reassignment can be acquired. However, just as in computational photon reassignment, a loss in confocal sectioning performance is caused by working with relatively open pinholes. The OPRA system shares properties such as flexibility and ease of use with a confocal laser scanning microscope, and is therefore expected to be of use for future biomedical routine research.
Keywords:Photon reassignment; Image scanning microscopy; Confocal laser scanning microscopy
Confocal laser scanning microscopy (CLSM) is an established tool in fluorescence microscopy and well-known for its optical sectioning ability and high contrast (Pawley 2006; Minsky 1961). These characteristics are achieved by using detectors with a high dynamic range and collecting the emitted light through a pinhole, which is usually aligned to the position of the excitation focus (thus the name “confocal”). The resulting image is constructed by assigning the detected intensity to the corresponding excitation scan position. In 1982 it was shown that it is possible to achieve enhanced resolution by applying an off-axis pinhole (Cox et al. 1982). In 1988 pinhole plane image detection and computational reassignment to a position half way between nominal excitation and detection position was proposed (Sheppard 1988), to improve detection efficiency and resolution. Note that for identical excitation and emission point spread functions (PSF) this reassigned position corresponds to the most probable position of an emitter in the sample. Recent work applied this principle in single (Müller & Enderlein 2010) and multispot excitation (York et al. 2012) to the imaging of biological samples.
Here we present optical photon reassignment microscopy (OPRA). It is an optical realization of these computer based methods which avoids the need for data processing. Furthermore at a different scaling ratio, our method is applicable to the direct visualisation of high-resolution imaging methods like STED.
In normal CLSM the detected intensity values of every scanning position are recorded with an integrating detector such as a photo-multiplier-tube (PMT) or an avalanche photo-diode. If a whole image of the pinhole plane is recorded at each scan position the acquired data-set of a single focal slice can be viewed as a 4 dimensional set of data (intensity values in dependency of xy scan and xy pinhole-plane image-position). Such data can be processed in several ways, for example allowing for a retroactive choice of the pinhole diameter and/or applying multi-view deconvolution (Brakenhoff & Visscher 1992 Heintzmann et al. 2003). Photon reassignment microscopy (Sheppard 1988; Müller & Enderlein 2010) is based on the insight that the most probable origin of the detected photons is at maximum of the joint probability function (i.e. the product of the individual probability functions) of excitation and (off-centre) detection. This is contrary to a CLSM where all detected photons are assigned to the nominal excitation position s.
In OPRA a similar reassignment to the optimal emission location is achieved optically. For example by an intermediate beam expansion between descanning and a subsequent rescanning. This is illustrated in Figure 1 at 3 successive time points. The upper row (1a) depicts the situation in the sample, where a scanned excitation beam (blue) together with a fixed emitter (green) at the origin is shown. The lower row (1b) refers to the final image plane. Due to the intermediate beam expansion, the emission PSF is reduced in size. As a further consequence the image of the emitter is now found at s(1-m), with s being the nominal image position of the centre of the excitation focus and m being the intermediate magnification. At non-uniform intermediate magnification (m≠1) the image of the emitter now performs a small scan on the final image plane, changing its brightness (not shown in Figure 1) under the influence of the excitation spot.
Figure 1. The principle of OPRA. The figure shows the imaging process of one point source at different times. The fluorophore is placed at the point of origin in the sample plane (a). The emitted photons are imaged to different positions in the image plane (b) according to the excitation positions s. If the general magnification of the microscope is neglected and the intermediate magnification is m=0.5 the photons are reassigned to half the distance between the nominal excitation position s and the position of the detected photon without intermediate magnification. In normal scanning microscopy the photons are always assigned to position s. Note that the brightness changes of the green emitter caused by the variation in excitation are not shown in this scheme.
To aid understanding, a movie of a simulated scan process depicting the sample plane, the pinhole plane and the camera- (or display screen-) plane is given in Additional file 1, for the confocal case and OPRA at m=1 and m=0.5.
Additional file 1. Simulation of the OPRA principle. Top row (left to right): object and illumination intensity; pinhole plane; rescanned (m=1); rescanned sum (m=1). Bottom row (left to right): sum confocal (0.3 AU pinhole); pinhole plane (m=0.5); rescanned (m=0.5); rescanned sum (m=0.5).
Format: AVI Size: 578KB Download file
We now aim to find an analytical description of the PSF of the overall system. The emission PSF has also undergone the intermediate beam expansion m and can therefore be written as h(x/m), positioned at s(1-m). Thus, the current image generated by the point source at the point of origin is , where s is the nominal scan position and x the image coordinate (measured in sample coordinates).
The total image of our point emitter is formed by integrating over all scan positions s:
This can be written using the convolution operator ⊗ as
where and the symmetrical excitation PSF are used.
If we assume a Gaussian shaped excitation and emission PSF (standard deviation σex for the excitation and corresponding σem for the emission function) the integral can be solved analytically. The final PSF is found to have the standard deviation.
The minimal total extent is found at:
Thus the additional expansion m should be adjusted to the different width of the excitation and emission PSFs. This difference can be induced by the Stokes shift of the used fluorophores or is a feature of the microscopy technique itself, to which OPRA is applied (e.g. STED microscopy, where σex is significantly smaller than σem).
If we assume a beam expansion value of m=0.5 (which makes a wider beam and therefore produces a smaller spot when focussed) we get a rough estimate for the resolution ability of OPRA
If no Stokes shift is considered (σex = σem) we obtain a resolution improvement of over what we would expect for confocal detection with a closed pinhole. This shows that reassignment microscopy realizes high resolution at the theoretical overall detection efficiency of a widefield microscope. OPRA attains the same characteristics as computational reassignment without the need for high-speed pinhole cameras and without the increased read-noise of multiple fast readouts. This raises the acquisition speed as the whole image is acquired in only one exposure frame. An additional pinhole can also be integrated in OPRA (before rescanning) to achieve confocal sectioning. Note that all emitted light of a scan from a planar fluorescent sample would otherwise reach the detector and thus prevent optical sectioning. For a detailed discussion of the sectioning ability see (Sheppard et al. 2013).
Figure 2 shows the experimental setup. The illumination part is a normal laser scanning setup, which creates a moving illumination spot in the sample plane. The beam of the excitation laser (Coherent, Sapphire LP 488 nm) is sent through a beam expander (L1 and L2) to a dichromatic beam splitter (BS1, AHF Analysetechnik Tübingen, ZT488RDC) where it is reflected towards the scanning unit. Here, two scanning mirrors SMY1 and SMY2 (Cambridge Technologies, CT6800HPL with CTI CB6580 driver) achieve the scan along the y-axis while keeping the pupil plane stable at the position of the resonant x-scan mirror SMX (EOPC, SC-30, resonant optical scanner, 15 kHz, USA). Another beam expander consisting of the tube lens (fTL=400 mm) and an achromatic doublet (f3=60 mm) provides a slight over-illumination of the back focal plane of the objective (Carl Zeiss, Plan-Apochromat 63×/0.7 Oil). On the detection side, the returning light is descanned using scan mirrors SMX, SMY1 and SMY2. Fluorescent and back-scattered illumination light are separated by the dichromatic beam splitter (BS1) and the fluorescent light is expanded by lenses L4 and L5 – this is the additional intermediate beam expansion. To achieve confocal sectioning a pinhole can be placed between these lenses, as this is a conjugate plane of the sample plane. Since the ideal intermediate beam expansion depends on the Stokes shift of the imaged fluorophores, the magnification can be adjusted by choosing the focal lengths of the lenses L4 and L5. After this intermediate magnifying step, the emission light is guided to the same scanning unit to be rescanned. The concept of using the same mirror(s) for de- and rescanning is similar to the description in (Brakenhoff & Visscher 1992) and was part of a commercial system as it was sold by the company Meridian. Lens L6 finally directs the emission light to a camera (Andor Technology Inc., Neo sCMOS, Belfast) where a super-resolved image is captured by integrating (5s for the widefield case and 10s for the rescanned case in Figure 3) over a full scan process.
Figure 2. OPRA Setup. The laser emitting at 488nm passes a clean-up (lens L1 and L2 with appending focal length f1=50mm; f2=100mm) and is directed to a dichromatic beam splitter (BS1, detection wavelength bigger than 505nm). The scan unit in detail is shown in the inset. Here two y-scanning-mirrors (SMY) are used to project the spot to the rotation axes of the x-scanning-mirror (SMX, 15 kHz). After the scanning unit the beam passes a second beam expander and is directed to the objective. The returning fluorescent light is descanned and separated from the excitation light using the dichromatic beam splitter (BS1). After descanning the fluorescent beam is expanded by a factor of two (f4=200mm; f5=400mm). The adjustable detection pinhole between the lenses L4 and L5 can be used to achieve confocal sectioning (not in measurements). After the expansion the beam is rescanned using the same scanning system and projected via the lens L6 (f6=200mm) to the camera. To compare the images with a widefield setup an excitation light source, the optional (opt.) lenses L7, and a dichromatic beam splitter (BS2) were added in this configuration, while the scanner does not move and the detection pinhole is removed.
Figure 3. Comparison of fluorescent beads imaged in OPRA (a) and widefield (b) mode (with an arrested scan system). For determination of the FWHM, 8 beads were individually fitted with a 2D Gaussian function. The data shown in (c) corresponds to the average of 3 consecutive centered lines of the summed 8 bead images. Solid lines in (c) correspond to the average FWHM results. The FWHM of the scanned bead images is visibly reduced. The average FWHMs of the 8 beads in the respective images were determined to amount to 473 ± 19 nm (widefield) and 327 ± 4 nm (OPRA). Scale bar 5μm.
To compare theory with measurements, point spread functions were calculated using vectorial theory (here assuming random polarisation) for the respective experimental wavelengths. The excitation PSF was then convolved with the emission PSF and the scale changed by a factor of 2 according to eqn. (2) with m=0.5. To finally account for the size of the beads, this resulting PSF was then convolved in 3D with a three-dimensional spherical volume of 200 nm diameter and the width of the resulting function was fitted with a Gaussian and measured.
To demonstrate the OPRA principle, fluorescent coated beads (FluoSpheres® Carboxylate-Modified Microspheres, 0.2 μm, Yellow-Green Fluorescent (505/515)) with a diameter of 200 nm were imaged. For comparison a widefield excitation lamp (EXFO photonics solutions Inc., X-Cite series 120 Q) was coupled into the setup with an optional dichromatic beam splitter (Carl Zeiss Microscopy GmbH, FT 510), such that the same sample position could be imaged with both methods as shown in Figure 3. For comparison 8 bead-images were analysed and the full-widths at half maxima (FWHM) of fitted 2D Gaussian functions were determined. The mean FWHM of measured 200 nm beads in the widefield image (Figure 3b) is determined to be (473± 19)nm and in the OPRA-image (327± 4)nm (Figure 3a), without using a pinhole. Accounting for the 200nm diameter of the beads, equation (2) predicts a FWHM of 297 nm (256 nm as Gaussian fit without accounting for the bead size) for the bead images acquired with OPRA, and 428nm for the widefield case, with the PSFs generated for the parameters as given above using vectorial diffraction theory. Thus theory and experiment agree to within about 10%. For the OPRA mode and 11% for the widefield detection. As the OPRA image theoretically contains the same number of photons as the widefield image but distributed to a sharper image, it is expected that the OPRA image also looks significantly brighter that the corresponding widefield (or confocal) image. However, since we used a separate illumination source to generate the widefield image, we could not do an appropriate comparison in this study.
In the presented paper a new method in fluorescence microscopy was introduced - OPRA microscopy. It realizes super-resolved images with high detection efficiency. Similar to computational photon reassignment (Cox et al. 1982; Müller & Enderlein 2010; York et al. 2012), the lateral resolution enhancement goes beyond the performance of a confocal microscope, even when compared to the limiting case of a confocal microscope with a completely closed pinhole. The PSF of a confocal microscope with closed pinhole is given by a product of excitation and emission PSF, whereas photon reassignment (eqn. 2) is governed by a convolution of the excitation with the emission PSF combined with a scaling of the coordinate system. These two are compared in Figure 4 along with their Fourier transforms, the optical transfer functions (OTF). It can be seen, that the OPRA PSF is slightly smaller in full width at half maximum (FWHM) than the confocal PSF even with a (in practice impossible) fully closed pinhole. Note that approximating the PSF as a Gaussian and neglecting the Stokes shift, would predict identical PSFs for the fully closed pinhole confocal and the OPRA case. Also its optical transfer function has a significantly enhanced transfer strength at frequencies higher than 10% beyond the detection Abbe limit. Note also that this calculation was done with a demagnification of m=0.5 which was not optimized to account for the slight change in wavelength (Sheppard et al. 2013).
Figure 4. PSF and OTF comparison. a) Comparison of the expected point spread functions of the widefield, confocal (pinhole 0.3 and 0 AU) and OPRA system (m=0.5). Note that all these curves are normalized to one, whereas the OPRA point spread function is significantly brighter at the peak compared to the widefield system. b) Comparison of the respective optical transfer functions. The frequency axis is normalized to the Abbe limit and the transfer strength to a maximum of one.
Compared to computational photon reassignment no post-processing is required as the summation and photon reassignment is a system inherent property of OPRA. This prevents artefacts (e.g. pixilation artefacts, additional read noise) and is even insensitive to small variations in the scanning process, as those will mostly lead to small brightness changes in the resulting image. This makes it useful for very fast imaging with the scanning-speed and the camera frame-rate as the only limiting factors. We showed that the principle improves the resolution in comparison to classical widefield microscopy and we derived the basic theory for OPRA performance. It should be noted that the required OPRA properties are also achievable with realization methods other than intermediate magnification, such as the use of separate scan-units for illumination and detection light running at different speeds. OPRA can be adapted to various ratios of the sizes of excitation and emission PSF. Therefore the OPRA principle can also be used to optically realize versions of super-resolution modes such as STED, GSD and RESOLFT (Hell 2003). At large transition saturation factors, these methods would profit only marginally from an additional resolution gain but such a setup would, however, enable these modes for the first time to directly generate a highly resolved optical image without even the need for any data acquisition. Even a multi-spot STED, GSD or RESOLFT microscope should be feasible with the OPRA approach, where the super-resolved image is built up during the integration time of a single frame. When using a periodic multi-spot excitation array, a single scan mirror can suffice. This was demonstrated in (York et al. 2013) which appeared after our manuscript was accepted for publication. Rescanning can also be performed with an electronically synchronised second scan mirror (or system of mirrors) or especially in the case of multi-spot illumination the rear side of the scan mirror can be used for rescanning. The camera can also be replaced by the human eye, realizing direct-view versions of STED or RESOLFT microscopy. Due to the simplicity and flexibility of the realizations, OPRA can enhance the performance of nearly every laser scanning microscope.
Currently our system does not use an appropriate tube lens to be free of chromatic aberrations. Future refinements of the imaging and scan optics are expected to push the performance of the system in both modes closer to the theoretical limit, especially for larger numerical apertures and low magnification.
Especially noteworthy is that OPRA achieves a theoretical image brightness superseding the performance of a widefield microscope under the same illumination dose. More photons are concentrated onto the same image pixel area. In this respect it differs from many alternative high-resolution methods which often only “shave” the PSF.
Even though OPRA improves photon reassignment in its all-optical realization, it should be noted that a full pinhole plane scan dataset of images with full dependency of scan and image coordinates is richer, and allows for better ways of image processing. These range from the ability retroactively to select the pinhole size, to optimization strategies such as weighted averaging in Fourier space and combined deconvolution (Heintzmann et al. 2003). As pinhole plane array data does not require a physical pinhole it can avoid the compromise between lateral resolution and optical sectioning performance of OPRA. Nevertheless, OPRA avoids generating large amount of data, along with the additional associated readout noise, with the additional benefit (even over a classical confocal microscope) of an inherent stability against scan imprecisions even when caused by mechanical vibrations influencing the scanners.
CLSM: Confocal laser scanning microscope; FWHM: Full width at half maximum; GSD: Ground state depletion; ISM: Images scanning microscopy; LSM: Laser scanning microscope; OPRA: Optical photon reassignment; OTF: Optical transfer function; PSF: Point spread function; RESOLFT: Reversible saturable optical fluorescence transitions; sCMOS: Scientific complementary metal–oxide–semiconductor; STED: Stimulated emission depletion.
The authors have jointly submitted a patent application.
SR designed and performed the experiments, analysed the data and contributed to the concept. CJRS contributed to the concept and the theoretical understanding. KW contributed to the concept and helped with the data analysis. RH conceptualized OPRA and contributed to the experiment and data analysis. All authors read and approved the final manuscript.
Brakenhoff GJ, Visscher K (1992) Confocal imaging with bilateral scanning and array detectors. J Microsc 165:139-146 Publisher Full Text
Heintzmann R, Sarafis V, Munroe P, Nailon J, Hanley QS, Jovin TM (2003) Resolution enhancement by subtraction of confocal signals taken at different pinhole sizes. Micron 34(6–7):293-300 PubMed Abstract | Publisher Full Text
Sheppard CJR, Mehta S, Heintzmann R (2013) Superresolution by image scanning microscopy using pixel reassignment. Optics Lett 38:2889-2892 Publisher Full Text
York AG, Parekh SH, Nogare DD, Fischer RS, Temprine K, Mione M, Chitnis AB, Combs CA, Shroff H (2012) Resolution doubling in live, multicellular organisms via multifocal structured illumination microscopy. Nat Methods 9(7):749-754 PubMed Abstract | Publisher Full Text | PubMed Central Full Text