光学仪器  2023, Vol. 45 Issue (1): 18-24 PDF

Virtual staining techniques for cellular microscopic imaging
ZHANG Hao, DAI Bo, ZHANG Dawei
School of Optical-Electrical and Computer Engineering, University of Shanghai for Science and Technology, Shanghai 200093, China
Abstract: Cell microscopic imaging is an important tool for cell phenotype detection. Traditional fluorescence imaging techniques are widely used in the cell imaging. However, the fluorescence imaging instruments have complex structure and high cost. Besides, staining could cause damage to cells. To address this problem, this paper proposes a virtual staining technique that performs strict alignment of bright field and fluorescence image datasets using a multimodal alignment algorithm, and improves network architecture, loss function, post-processing, and hardware adaptation for training optimization. The staining conversion bias is calculated by the evaluation criteria of the virtual staining. The method presented in this paper could simplify the fluorescence imaging equipment and eliminates the need for various staining operations, which could reduce the burden of research and diagnostic processes for biologists and pathologists.
Key words: deep learning    cellular imaging    virtual staining

1 虚拟染色技术的显微成像

 图 1 图像配准流程 Figure 1 Image registration process

2 样品制备及标定

 图 2 使用未染色组织的自发荧光进行基于深度学习的虚拟组织学染色 Figure 2 Deep-learning-based virtual histology staining using autofluorescence of unstained tissue
3 虚拟染色算法

 图 3 cGAN网络结构图 Figure 3 cGAN network structure
 $\begin{split} {\mathcal{L}}_{{\rm{cGAN}}}\left(G,D\right)=& {{\rm{E}}}_{x,y}\left[{\rm{log}}\;D\left(x,y\right)\right]+\\ & {{\rm{E}}}_{x}\;\left[{\rm{log}}\;\left(1-D\left(x,G\left(x\right)\right)\right)\right] \end{split}$

 ${\mathcal{L}}_{L1}\left(G\right)={{\rm{E}}}_{x,y}\left[{||y-G\left(x\right)||}_{1}\right]$
 ${\mathcal{L}}_{{\rm{PCC}}}\left(G\right)={{\rm{E}}}_{x,y}\left[{PCC}\left(y,G\left(x\right)\right)\right]$

 ${G}^{*}=\mathit{{\rm{arg}}}\;\underset{D}{{\rm{min}}}\;{\underset{G}{{\rm{max}}}\;\mathcal{L}_{_{{\rm{cGAN}}}}}\left(G,D\right)+\lambda {\mathcal{L}}_{L1}\left(G\right)+\gamma {\mathcal{L}}_{{\rm{PCC}}}\left(G\right)$

 图 4 网络结构图 Figure 4 Network structure
 $\begin{split} {\mathcal{L}}_{{\rm{GAN}}}\left(G\right)=& {{\rm{E}}}_{b～{p}_{{\rm{data}}}\left(b\right)}\left[{\rm{log}}\;{D}_{B}\left(b\right)\right] +\\ &{{\rm{E}}}_{a～{p}_{{\rm{data}}}\left(a\right)}\left[{\rm{log}}\;\left(1-{D}_{B}\left(G\left(a\right)\right)\right)\right] \end{split}$
 $\begin{split} {\mathcal{L}}_{{\rm{GAN}}}\left(F\right)=&{{\rm{E}}}_{a～{p}_{{\rm{data}}}\left(a\right)}\left[{\rm{log}}\;{D}_{A}\left(a\right)\right] +\\ & {{\rm{E}}}_{b～{p}_{{\rm{data}}}\left(b\right)}\left[{\rm{log}}\;\left(1-{D}_{A}\left(G\left(b\right)\right)\right)\right] \end{split}$

 $\begin{split} {\mathcal{L}}_{{\rm{cycle}}}\left(G,F\right)=& {{\rm{E}}}_{a～{p}_{{\rm{data}}}\left(a\right)}\left[{||F\left(G\left(a\right)\right)-a||}_{1}\right] +\\ & {{\rm{E}}}_{b～{p}_{{\rm{data}}}\left(b\right)}\left[{||F\left(G\left(b\right)\right)-b||}_{1}\right] \end{split}$

 $\begin{split} {\mathcal{L}}_{{\rm{sc}}}\left(G,F\right)=& {{\rm{E}}}_{a～{p}_{{\rm{data}}}\left(a\right)}\left[{||{T}_{\alpha }\left(a\right)-{T}_{\beta }\left(G\left(a\right)\right)||}_{1}\right] +\\ & {{\rm{E}}}_{b～{p}_{{\rm{data}}}\left(b\right)}\left[{||{T}_{\beta }\left(b\right)-{T}_{\alpha }\left(G\left(b\right)\right)||}_{1}\right] \end{split}$

 ${T}_{\alpha }\left(x\right)=Sigmoid\left[100\left(x-\alpha \right)\right]$
 ${T}_{\beta }\left(x\right)=1-Sigmoid\left[100\left(x-\beta \right)\right]$

 $\begin{split} {\mathcal{L}}_{{\rm{CycleGAN}}}=& {\mathcal{L}}_{{\rm{GAN}}}\left(G\right)+{\mathcal{L}}_{{\rm{GAN}}}\left(F\right)+\\ & \sigma \left[{\mathcal{L}}_{{\rm{cycle}}}\left(G,F\right)+\rho {\mathcal{L}}_{{\rm{sc}}}\left(G,F\right)\right] \end{split}$

4 虚拟染色评价标准

 $S_{\rm{SSIM}}\left(x,y\right)=\frac{\left(2{\mu }_{x}{\mu }_{y}+{a}_{1}\right)\left(2{\delta }_{xy}+{a}_{2}\right)}{\left({\mu }_{x}^{2}+{\mu }_{y}^{2}+{a}_{1}\right)\left({\delta }_{x}^{2}+{\delta }_{y}^{2}+{a}_{2}\right)}$

SSIM适用于测量高层结构误差，而PSNR对像素级的绝对误差更为敏感。PSNR反映的是图像质量信息的失真度。其值PPSNR越高，图像受噪声影响越小，失真越少。对于大小为 m×n 的两张彩色图像 IKI为虚拟染色图像，K为真实染色图像），均方误差MMSE定义为

 $M_{\rm{MSN}}=\frac{1}{3mn}\sum _{R,G,B}\sum _{i=0}^{m-1}\sum _{j=0}^{n-1}{\left[{I}_{{\rm{color}}}\left(i,j\right)-{K}_{{\rm{color}}}\left(i,j\right)\right]}^{2}$

PPSNR（以dB为单位）定义为

 $P_{\rm{PSNR}}=20{{\rm{lg}}}\;\frac{{P}_{{\rm{max}}}}{\sqrt{M_{\rm{MSE}}}}$

5 结　论