电子断层扫描中“缺失楔形效应”：无监督机器学习

电子断层扫描因其高分辨率在纳米材料三维表征方面受到青睐，但受限于“缺失楔形效应”，导致重建图像出现畸变。目前的算法，特别是机器学习中的神经网络技术，虽在纠正这些畸变方面取得了进展，但因训练数据与实际情况存在差异，仍面临着准确性的挑战。

由美国伊利诺伊大学香槟分校材料科学与工程系的Qian Chen教授领导的团队提出了UsiNet，这是一个无监督投影图修复方法，旨在解决电子断层扫描中常见的缺失楔形效应问题。

UsiNet的无监督训练机制免除了对基准真值、人工标注或倾斜图像模拟的依赖，显著提升了其在真实电子断层扫描数据集处理中的实用性，特别是在无法获取全角度倾斜序列的场合。

该方法在训练时只需极少量的数据集（甚至低至20个纳米颗粒）和较小的倾斜范围（±45°），使其在处理那些对束流敏感的聚合物和生物材料时变得尤为宝贵，因为这些材料的倾斜范围可能由于束流累积伤害而受到限制。

UsiNet对窄倾斜范围的容忍性在原位电子断层扫描的研究中至关重要，比如在研究电化学循环、催化作用或腐蚀过程中纳米颗粒形态变化时，由于需要保持时间分辨率，仅能收集有限的倾斜序列。

Fig. 5 Orientation-dependent missing wedge artifact and comparison between different reconstruction algorithms.

此外，UsiNet无需进行样品平均化处理，因此可广泛适用于诸如用于可充电离子电池的电极纳米颗粒、催化剂纳米颗粒和纳米塑料等多种异质纳米颗粒系统。在这些系统中，缺失楔形效应尤其棘手，因为它会引起明显的各向异性失真。尽管此次展示主要聚焦于胶体纳米颗粒，但UsiNet无监督修复的基本原理也同样适用于其他包含3D纳米尺度形态细节的样本，如合金的微观结构域和聚酰胺分离膜的皱折。UsiNet的出现极大地扩展了电子断层扫描技术在解析材料的形态、合成及性能之间关联性方面的潜力。

Fig. 6 Comparison of 3D reconstructions of experimentally synthesized NPs with and without inpainting.

UsiNet的应用前景极为广阔，它不仅能够揭示电池或催化纳米材料的退化机理，还能帮助理解自然形成的纳米塑料的形态与聚集行为，并能优化不同组成的纳米颗粒的合成流程。该文近期发表于npj Computational Materials 10: 28 (2024).

Fig. 7 Visualizing the heterogeneity of experimentally synthesized NPs.

Editorial Summary

Electron tomography is favored for its high-resolution in 3D characterization of nanomaterials but is limited by the “missing wedge effect”, leading to distortions in the reconstructed images. Modern algorithms, especially neural network technologies in machine learning, have made advances in correcting these distortions, yet they still face challenges in accuracy due to differences between training data and actual conditions. 

A team led by Prof. Qian Chen from Department of Materials Science and Engineering, University of Illinois at Urbana-Champaign, USA, purposed UsiNet, an unsupervised sinogram inpainting method to correct the missing wedge effect in electron tomography. The unsupervised training in UsiNet does not require ground truth, manual annotation, or tilt image simulation, and thus is practically applicable to real electron tomography datasets where full angle tilt series are not obtainable. The authors demonstrate that UsiNet works with a small number of training dataset (down to 20 NPs) and narrow tilt range (±45°), which can be immediately useful for beam sensitive polymeric and biological materials where the tilt range can be limited by accumulated beam damage. The tolerance with a narrow tilt range could be critical for studies involving in-situ electron tomography—for example, on the evolution of the 3D shapes of NPs during chemical reactions such as electrochemical cycling, catalysis, and corrosion—where only scarce tilt series can be collected to ensure temporal resolution. Moreover, UsiNet does not require sample averaging and can thus apply to a broad range of heterogeneous NP systems such as electrode NPs used in rechargeable ion batteries, catalytical NPs, and nanoplastics. The missing wedge effect is otherwise particularly problematic for heterogeneous systems by generating anisotropic distortion. Although the demonstration focuses on colloidal NPs, the principle of unsupervised inpainting is expected to work for other samples containing 3D nanoscale morphology details, such as microstructural domains in alloys and crumples in polyamide separation membranes. UsiNet brings the full potential of electron tomography in charting the relationships of morphology with synthesis and performance of materials. A wide scope of applications can be enabled by UsiNet, such as uncovering degradation mechanisms of battery or catalytical nanomaterials, understanding morphologies and aggregation behaviors of naturally formed nanoplastics, and optimizing synthetic protocols of NPs with varying compositions. This article was recently published in npj Computational Materials 10: 28 (2024).

原文Abstract及其翻译

No ground truth needed: unsupervised sinogram inpainting for nanoparticle electron tomography (UsiNet) to correct missing wedges (无需基准真值：用于校正缺失楔形区的无监督纳米颗粒电子断层扫描图像重建的投影图修复)

Lehan Yao, Zhiheng Lyu, Jiahui Li & Qian Chen 

Abstract Complex natural and synthetic materials, such as subcellular organelles, device architectures in integrated circuits, and alloys with microstructural domains, require characterization methods that can investigate the morphology and physical properties of these materials in three dimensions (3D). Electron tomography has unparalleled (sub-)nm resolution in imaging 3D morphology of a material, critical for charting a relationship among synthesis, morphology, and performance. However, electron tomography has long suffered from an experimentally unavoidable missing wedge effect, which leads to undesirable and sometimes extensive distortion in the final reconstruction. Here we develop and demonstrate Unsupervised Sinogram Inpainting for Nanoparticle Electron Tomography (UsiNet) to correct missing wedges. UsiNet is the first sinogram inpainting method that can be realistically used for experimental electron tomography by circumventing the need for ground truth. We quantify its high performance using simulated electron tomography of nanoparticles (NPs). We then apply UsiNet to experimental tomographs, where >100 decahedral NPs and vastly different byproduct NPs are simultaneously reconstructed without missing wedge distortion. The reconstructed NPs are sorted based on their 3D shapes to understand the growth mechanism. Our work presents UsiNet as a potent tool to advance electron tomography, especially for heterogeneous samples and tomography datasets with large missing wedges, e.g. collected for beam sensitive materials or during temporally-resolved in-situ imaging.

摘要 复杂的自然和合成材料，如细胞亚结构器官、集成电路中的器件架构，以及具有微观结构领域的合金，需要能够在三维（3D）中研究这些材料的形态和物理性质的表征方法。电子断层扫描在成像材料的3D形态方面具有无与伦比的（亚）纳米分辨率，这对于描绘合成、形态和性能之间的关系至关重要。然而，电子断层扫描长期以来一直受到实验上不可避免的缺失楔形效应的困扰，这导致最终重建中出现了不希望的、有时甚至是大量的失真。在这里，我们开发并展示了用于纳米颗粒电子断层扫描的无监督投影图修复（UsiNet）来校正缺失的楔形区。UsiNet是第一个可以在现实实验电子断层扫描中使用的投影图修复方法，它避开了对基准真值的需求。我们使用模拟的纳米颗粒电子断层扫描来量化其高性能。然后我们将UsiNet应用于实验层析图，其中同时重建了100多个十面体纳米颗粒和极为不同的副产物纳米颗粒，且没有缺失楔形失真。根据它们的3D形态对重建的纳米颗粒进行分类，以理解生长机制。我们的工作将UsiNet作为推进电子断层扫描的强大工具，特别适用于异质样品和具有大量缺失楔形区的断层数据集，例如为了敏感材料而收集的数据或在时间解析的原位成像过程中收集的数据。