Molecular Image Fusion Concept

Figure 1 Concept of image fusion of imaging mass spectrometry (IMS) and microscopy. Image fusion generates a single image from two or more source images, combining the advantages of the different sensor types. The integration of IMS and optical microscopy is given as an example. The IMS-microscopy fusion image is a predictive modality that delivers both the chemical specificity of IMS and the spatial resolution of microscopy in one integrated whole. Each source image measures a different aspect of the content of a tissue sample. The fused image predicts the tissue content as if all aspects were observed concurrently.

Example Application: Sharpening of ion images

One of the predictive applications that the Molecular Image Fusion framework enables is the sharpening of ion distributions in tissue. This application uses microscopy measurements to predict ion distributions at a spatial resolution that exceeds that of measured ion images by ten times or more. Figure 2 gives an example of combining coarse resolution IMS measurements with fine resolution microscopy measurements to predict ion distributions to a fine spatial resolution, and compares the prediction result to a genuine measurement obtained at that same spatial resolution.

Figure 2 Prediction of the ion distribution of m/z 762.5 in mouse brain at 10 μm resolution from 100 μm IMS and 10 μm microscopy measurements (sharpening). This example in mouse brain fuses a measured ion image for m/z 762.5 (identified as lipid PE(16:0/22:6)) at 100 μm spatial resolution (a) with a measured H&E-stained microscopy image at 10 μm resolution (b), predicting the ion distribution of m/z 762.5 at 10 μm resolution (reconstruction score 82%) (c). For comparison, (d) shows a measured ion image for m/z 762.5 at 10 μm spatial resolution, acquired from a neighboring tissue section.

Example Application: Prediction of ion distributions in tissue areas not measured by mass spectrometry

Prediction of ion distributions in tissue areas that were not measured by IMS is a second application enabled by the Molecular Image Fusion framework. 

Figure 3 Prediction of m/z 10,516 distribution in mouse brain areas not measured by IMS (out-of-sample prediction). An IMS-microscopy model is built on a tissue sub-area for which IMS is available at 100 μm resolution (a) and H&E-stained microscopy is available at 5 μm resolution (b). The model is then used to predict the distribution of m/z 10,516 in areas where no IMS was acquired and only microscopy is available (reconstruction score 88%) (d).

Method

Figure 4 Method overview. The fusion process consists of two phases. (a) Phase I builds a cross-modality model from the two measurement sources and evaluates for which ions good prediction is possible. (b) For those ions, phase II uses the model and the high-resolution microscopy measurements to predict the ion distribution at higher-than-IMS resolutions.