A Machine-Vision Approach to Transmission Electron Microscopy Workflows, Results Analysis and Data Management

The interpretation of TEM experimental results is often contingent on many inter-connected experimental parameters, such as microscope settings, imaging conditions, and in the case of operando or in situ experiments, changes to the environment or stimuli^1,23. Accurate analysis of large TEM datasets, over which these parameters may be continuously modified, requires significant attention from the operator to accurately record each condition and setting for each image in a lab journal or other external documentation source. As TEM datasets grow in size and complexity, manual recordkeeping becomes unmanageable, and key information may be missed or inaccurately recorded. The MVS software described here consolidates the metadata generated during an experiment from the microscope, the detector/camera, and other systems (such as in situ sample holders) and aligns them with their respective images.

In addition to metadata consolidation, the software applies machine-vision algorithms to track and stabilize the field of view through a combination of spatial, beam, and digital corrections using its Drift Correct and Focus Assist functions. When the Drift Correct function is engaged, a cross-correlation 'template' image is generated using the first image pulled into the MVS software. The template is then compared to incoming images to calculate the direction and magnitude of the sample drift or movement. With this information, the MVS software automatically applies the necessary corrections to keep the image features in the same place by adjusting at least one of three parameters: stage location, beam or image shift, and digital image correction. The Focus Assist function utilizes a combination of algorithms to assign a focus value, called the focus score to each image, and those scores are compared to determine the magnitude and direction of defocus adjustment to apply to keep the sample in focus. In STEM imaging mode, the MVS software attempts to maximize contrast through a proprietary version of normalized variance to assign the focus score. In TEM mode, a radial sum of intensity is calculated in the FFT and used to calculate the focus score. Limitations to the MVS software's ability to optimize the focus occur when it cannot accurately calculate the correct focus score for an image. This typically occurs when the microscope is misaligned or the sample is significantly out of focus during calibration, preventing the software from correctly calculating the correct starting focus score value. The MVS software can have difficulty calculating the focus score for samples with well-defined lattice fringes, as the lattice fringes in the FFT can 'overwhelm' the focus scoring algorithm; thus, if a sample moves out of focus, the focus score may not accurately reflect the change in focus. Conversely, working at low magnifications or with a sample that has a low FFT signal can also make it challenging to calculate a good focus score. To mitigate these difficulties, the MVS software contains a number of additional algorithms that can be selected by the user for calculating the focus score if the default settings are unsuitable for the sample. These must be tested and applied on a case-by-case basis to determine the best algorithms for a given experiment.

Morphological changes in the sample structure over time are accounted for using a template morphing factor. This filter is tunable by the operator, so that registration algorithms account for morphological changes over time. Additionally, the software monitors the continuous image, microscope settings, and camera or detector settings to automatically update the template when triggered by changes in sample structure and after any operator-induced changes to the microscope, camera, or detector parameters. As shown in Figure 4, Figure 5, Supplementary File 7, and Supplementary File 8, the MVS software provides effective, immediate stabilization, allowing high resolution imaging of dynamically moving or changing samples. Although the software is capable of controlling very high rates of drift or sample movement, such as those that occur when applying a heating ramp during an in situ experiment, there are limitations to the maximum stage corrections or beam shifts that the software can control if the sample is moving or drifting very rapidly. This limit is a function of the image update rate, field of view size, and drift rate. For a given field of view and image update rate, there is a maximum drift rate that can be corrected, and if the physical movements cannot keep up, then the process may end or become unstable. From the registration templates generated when features such as Drift Correct are applied, additional calculated metadata can be generated. For instance, Match Correlation is a numerical record of the extent of change between templates in a series and is used to identify points in an experimental timeline in which the sample changed. A high match correlation value corresponds to a sample that has undergone changes to its morphology, and a low match correlation value corresponds to a sample whose structure remains relatively static. Match correlation is particularly valuable for in situ studies as it can be plotted graphically, enabling the user to quickly pinpoint images in the series corresponding to significant sample change. It is important, however, to understand that high match correlations values can also correspond to changes in imaging conditions, such as moving the stage or changing the magnification, if these actions are performed while the Drift Correction function remains active.

The calibration workflow presented here utilizes a unique calibration holder and a semi-automated calibration routine to accurately calibrate the beam under a variety of lens conditions with minimal operator intervention. The dose calibration routine is accessed through the MVS software installed on the TEM. The MVS software automatically reads the relevant microscope settings to save all measurements to reference for later experiments. On some TEMs, it is not possible to read the aperture or monochromator settings, and these must be entered into the MVS software settings by the operator during calibrations and during use. There are reminders built into the software to help keep these operator input settings updated by following the program prompts. The development of a holder with a built-in current collector, rather than relying on one integrated elsewhere in the microscope column, is a deliberate design choice. This enables the current collector to be positioned at the same plane as a sample, eliminating errors in the current measurement caused by beam deflection or differences in the absorption of electrons by apertures at different beam positions. The MVS software follows an automated routine to measure the beam current and area for any combination of lens conditions. The software can then correlate these measured calibrations with the camera or screen current and extrapolate any changes in magnification etc. to the beam area during the experiment. Once generated, these calibration files can be used immediately and are automatically saved for later use if the software detects the same settings being used during a future session. Although the longevity of the calibration file varies from microscope to microscope, the authors have found that they are able to use the same calibration files for several months without observing substantial changes to the current values. There are built-in routines monitoring the emission profile of guns to help keep these calibrations relevant, especially on cold FEG emission guns.

Normalization of dose measurements between microscopes and automated tracking of a sample's beam exposure are critical functions of the MVS software, as they allow quantitative comparisons of dose conditions between experiments to be performed on different microscope systems. Dose induced degradation of a zeolite sample (ZSM-5), obtained during identical experiments using different microscopes, results in complete disappearance of the FFT spots after a maximum cumulative or threshold electron dose (~60.000 e^–/Ų when applying a dose rate of ~500 e^–/Ų·s) for both setups. These comparative results demonstrate that the dose software facilitates reproducible, quantitative dose measurements. The small difference in the cumulative dose at which full FFT spot disappearance is observed for each experiment is likely a result of the different acceleration voltages employed by the two microscopes, with lower acceleration voltages resulting in more radiation damage pathways, and higher acceleration voltages typically resulting in more knock-on damage²⁴. Literature results for the critical dose of ZSM-5 nanoparticles range from 9,000-14,000 e^–/Ų using the first FFT spot disappearances, rather than the complete disappearance of all the FFT spots^25,26. In our results, the first FFT spot disappearance corresponds to a cumulative dose of around 25,000 e^–/Ų. Previous studies relied on current measurements obtained using a phosphor screen, which is well documented to underestimate beam current measurements when compared to a Faraday cup¹⁵. The determined critical dose can vary by a factor of two or more, depending on which FFT peak is used to track the dose. This indicates that the higher spatial frequencies degrade first, and can result in different values depending on the zone access used during the measurements (our results focused on FFT spots from the entire zeolite crystal, rather than specific structural features)^25,26. These differences in techniques and current calibration account for the difference in values between the two experiments reported in our results and previous literature studies.

Although the electron dose interactions are a significant factor in many TEM experiments, in situ and specifically liquid-EM studies are particularly sensitive to its effects. Radiolysis of liquids by the electron beam results in a cascade of chemically reactive species which can interact with the sample, complicating the analysis. Both the dose rate or fluence used during a liquid-EM experiment and the cumulative dose can have an influence on the concentration of radical species generated due to liquid radiolysis^27,28. Thus, collecting and recording both cumulative dose and dose rate metadata throughout an experiment allows direct correlation between images and a sample's dose history, and is a more accurate way to elucidate and control the impact of the electron beam in these experiments. Although not covered in this protocol, an example of the utility of the dose management features for liquid-EM is shown in Figure 6.

Figure 6: Beam-induced growth of gold nanoparticles during an in situ liquid-EM experiment. (A) Low-magnification STEM overview of the resulting particle growth with a color overlay of the cumulative dose map across the region. Red areas in the overlay indicate regions of high cumulative dose exposure and yellow areas indicate regions of lower exposure. Highlighting an individual pixel with the cursor or drawing a box over an area using the included drawing tools indicates the cumulative dose for that pixel or area. The scale bar is 2 µm. (B,C) Higher magnification STEM images of the areas indicated by the orange boxes (b,c) in A. Area b, exposed to a higher cumulative dose (10.811 e^–/Ų) contains larger particles than those found in area c, which was exposed to a lower cumulative dose (0.032 e^–/Ų). Please click here to view a larger version of this figure.

The enriched dose rate and cumulative dose metadata simplifies the analysis of dose-dependent nanomaterial growth and degradation pathways. Figure 6 shows the beam-induced reduction of a solution of gold auric chloride (HAuCl₃) ions in water during liquid-EM experiments. From the color dose map overlay in Figure 6A, it is easy to visualize that the cumulative electron dose influences the resulting size and shape of the nanoparticles^29,30,31,32. The low magnification STEM overview shows regions exposed to a high (red) and low (yellow) cumulative dose. The particles in the region exposed to higher doses are larger than those in the regions exposed to lower cumulative doses. Because the dose metadata is directly embedded in each image at the pixel level, the complex effects of electron dose in liquid-EM experiments can now be systematically analyzed in a way that was never-before achievable.

In this protocol, we have demonstrated that MVS software provides a comprehensive solution for calibrating, monitoring, and tracking both the electron dose and the total dose delivered to a sample on a pixel-by-pixel basis. This ability unlocks a new paradigm for imaging dose-sensitive samples and understanding the electron beam interactions. It is particularly exciting for liquid-EM experiments, as it will allow for a more effective interrogation into the role that electron dose plays and improve experimental reproducibility. It is our hope that this new framework will allow the accurate collection of dose rate and accumulated dose information, facilitate sharing this data with the community for a more accurate interpretation of TEM results, and advance scientific collaboration and data sharing by enabling FAIR principal reporting and analysis.