Multimodal Medical Image Fusion: Techniques, Databases, Evaluation Metrics, and Clinical Applications -A Comprehensive Review

1.1. Imaging Modalities in Medical Image Fusion

Through the fusion of several medical imaging techniques, MMIF manages to gather a variety of information concerning structural and functional features of human bodies. An overview of structural, functional, and multimodal imaging modalities is given in Fig. (3).

X-rays are a fundamental imaging modality and are still highly used for imaging bones, for identifying fractures, infections, tumors, etc., in a two-dimensional image. Computed Tomography (CT) is a method that utilizes ionizing radiation in the form of X-rays for producing cross-sectional three-dimensional images of the body in great detail. It is superior in visualizing bones, revealing internal bleeding, and detecting tumors. CT scans offer rapid and cost-effective imaging, making them a routine choice in trauma evaluations; however, they involve exposure to ionizing radiation [6]. While CT excels in imaging bones compared to MRI, it cannot differentiate soft tissues as effectively.

Magnetic Resonance Imaging (MRI) is a non-invasive technology that uses magnetic fields and radio waves to create high-quality images of soft tissues in the brain, muscles, and other internal organs without the use of ionizing radiation. Owing to its superior image quality and the absence of radiation exposure, MRI is the modality of choice for evaluating brain tumors, spinal cord abnormalities, and ligament injuries [7].

Positron Emission Tomography (PET) is an invasive imaging modality utilizing ionizing radiation to assess metabolic and functional processes within the body. The technique involves administering a radiotracer that emits positrons during radioactive decay. PET is beneficial for oncology, neurology, and cardiology because it detects biochemical changes before anatomical changes are observable. The PET’s weak spatial resolution is complemented by its combination with MRI or CT techniques to enhance anatomical mapping. SPECT and PET use radioactive tracers to measure blood flow and metabolic processes. However, SPECT has less sensitivity and poorer imaging resolution than PET. Despite its limitations, SPECT offers important data for diagnosing cardiac perfusion problems, epilepsy, and bone disorders. When SPECT is combined with CT or MRI, its diagnostic reliability increases significantly, minimizing the effect of its poorer spatial resolution. fMRI uses the measurement of changes in blood oxygenation and flow that correspond to neural activity as an extension of a standard MRI. It is especially evident in brain mapping applications in neuroscience research and surgery planning. Researchers at MMIF often combine fMRI with structural MRI to impart brain function to particular anatomical structures [8]. Table 2 presents the categorization of imaging modalities, such as ultrasound and endoscopy, according to their degree of invasiveness and the extent of patient exposure to ionizing radiation.

2.1. The Cancer Imaging Archive (TCIA)

TCIA appears to be one of the top sources of comprehensive and exhaustively curated cancer imaging data repositories. Under the direction of the National Cancer Institute (NCI), the collection encompasses more than 50,000 richly varied imaging cases CT, MRI, PET, and histopathology. TCIA is a benchmark for assessing fusion algorithms because of its ability to support multimodal imaging research. For example, the Lung-PET-CT-Dx dataset contains synchronized PET and CT scans for the diagnosis of lung cancer, and BraTS contains multimodal MRI (T1, T1-Gd, T2, FLAIR) images annotated for brain tumor research. The platform allows for direct visualization and annotation support through integration with 3D Slicer, ITK-SNAP [9].

2.2. Open Access Series of Imaging Studies (OASIS)

OASIS provides an extensive collection of neuroimaging data, specifically, destined for studies in aging, Alzheimer’s disease, and cognitive deficits. OASIS consists of cross-sectional MRI and PET imaging datasets. OASIS-3, the latest version, contains over 2,000 subjects that had a series of imaging sessions, genetic testing, and clinical testing [10]. The variety of modalities within OASIS datasets makes them ideal for fusion studies. One example is the accuracy of diagnosis of Alzheimer's disease at its earliest stages by using the amyloid PET along with the T1-weighted MRI. Each dataset is represented in the NIfTI format, a reputable neuroimaging standard, and is accompanied by metadata that includes the results of cognitive evaluations.

2.3. Alzheimer’s Disease Neuroimaging Initiative (ADNI)

ADNI is a collaborative investigation for the study of the progress of Alzheimer’s disease through imaging and clinical data. It provides rich longitudinal data drawn from several imaging and biomarker modalities such as MRI, FDG-PET, amyloid PET, and CSF biomarkers. The initiative tracks more than 1,700 participants through various stages of cognitive decline. ADNI research subjects are characterized at three cognitive levels such as cognitively normal, MCI, and Alzheimer’s disease. The large temporal resolution of ADNI allows the study of temporal dynamics of brain change represented by other imaging technologies. In many ADNI data-based investigations, researchers use fusion methods to predict progression from MCI to Alzheimer’s by combining structural MRI with metabolic PET information [11]. Researchers can obtain data from the ADNI through an application process, in the form of DICOM and NIfTI, via the Laboratory of Neuro Imaging (LONI) platform.

2.4. Medical Image Data Archive System (MIDAS)

MIDAS, an adaptable data management system, was developed by Kitware, which serves the area of imaging datasets, including the realms of radiology, pathology, and ultrasound. The system’s flexible architecture enables the integration and storage of 2D and 3D imaging data together with corresponding clinical and demographic records. The uniqueness of MIDAS is its capability to integrate custom plugins and tools, which is useful for researchers who work with end-to-end pipelines for image fusion and segmentation. Some of the commonly used datasets in MMIF research include Head-Neck Cancer CT-MRI and Cardiac MR + Ultrasound [12]. MIDAS can handle DICOM, RAW, and MetaImage (.mha).

2.5. Digital Database for Screening Mammography (DDSM)

The DDSM repository exists with the objective of large-scale breast detection, predominantly facilitated by X-ray mammograms. Although the repository originally concentrated on one modality, the recent additions, such as INbreast and CBIS-DDSM, have included histopathology and ultrasound data for supporting multimodal analysis. The DDSM collection contains more than 2,500 studies, several of which are annotated with information about mass boundaries. The DDSM’s usefulness for MMIF lies in its synthesis of mammographic features with pathological confirmation, supporting the integration of image and diagnostic data [13].

2.6. Annotated Alzheimer Neuroimaging Library (AANLIB)

AANLIB was created as a dataset specific to assisting multimodal fusion and classification research for Alzheimer’s disease. It includes thousands of cases with T1-MRI, FLAIR, PET, and neurocognitive records. Unlike large datasets such as OASIS or ADNI, AANLIB provides images that are prepared for fusion (preprocessing corrects for skull stripping and registration). These standardized images significantly streamline the work in the MMIF workflows. The AANLIB’s corresponding images are archived in NIfTI format, and its accompanying ground truth labels are provided for employing it in supervised learning procedures. The open access for academic users makes it suitable for evaluating deep learning fusion algorithms [14]. The source images of the brain from AANLIB in axial, sagittal, and coronal sections are presented in Fig. (5).

2.7. Dataset Summary and Usage Patterns

The most widely used fusion modalities across these datasets include MRI+PET, MRI+CT, and PET+CT, reflecting the clinical demand for combining anatomical and functional insights. For example, ADNI and OASIS predominantly use MRI+PET, while TCIA and MIDAS offer CT+PET or MRI+CT combinations. Table 3 provides an overview of the available databases with respect to imaging modalities, anatomical regions, and file formats. Regarding disease focus, the datasets are primarily oriented toward the following pathological conditions:

• Neurodegenerative diseases: OASIS, ADNI, AANLIB
• Cancer imaging: TCIA, DDSM, MIDAS
• Cardiac and head-neck imaging: MIDAS
• Neurofunctional tasks: AANLIB, OASIS

3.1. Preprocessing

Preprocessing serves as the foundation of MMIF and involves preparing images for further stages by enhancing quality and ensuring uniformity across modalities. This stage includes noise reduction, intensity normalization, contrast enhancement, resolution adjustment, and format conversion. Since different imaging modalities have different spatial resolutions, acquisition angles, and noise characteristics, preprocessing plays a vital role in aligning these disparities. For instance, Magnetic Resonance Imaging (MRI) often exhibits inhomogeneous intensity, while Computed Tomography (CT) is prone to beam-hardening artifacts. Normalizing these differences using histogram equalization or z-score normalization improves the effectiveness of subsequent fusion steps. Additionally, many public datasets like AANLIB provide preprocessed images, including skull stripping and spatial standardization, reducing the preprocessing burden on researchers [15]. Advanced preprocessing techniques such as total variation filtering, anisotropic diffusion, and rolling guidance filters have been adopted in recent MMIF research to preserve structural details while eliminating noise. These techniques are especially beneficial in fusion scenarios involving low-quality or low-contrast images. The process of image fusion is shown in Fig. (6).

3.2. Image Registration

Registration is the process of spatially aligning two or more images of the same anatomical region, but from different modalities. It compensates for differences in image scale, orientation, and position, ensuring that corresponding anatomical structures overlap accurately. The accuracy of registration directly affects the fidelity of the final fused image, especially in pixel- and feature-level fusion.

Image registration methods are typically categorized into rigid, affine, and non-rigid (deformable) techniques. Rigid registration handles only translation and rotation, while affine registration includes scaling and shearing. Non-rigid registration addresses complex deformations and is often necessary when fusing modalities like PET and MRI due to organ motion or different acquisition geometries. Techniques such as mutual information maximization, normalized cross-correlation, and landmark-based mapping are commonly employed. Deep learning-based registration models, especially U-Net architectures trained on spatial transformer networks, are gaining popularity due to their speed and robustness. These methods can perform unsupervised, real-time registration even in challenging clinical settings.

3.3. Image Fusion Techniques

The core of MMIF is the fusion process itself, which combines information from registered images into a single image. Fusion techniques are classified based on the level at which integration occurs, as described in Fig. (7).

3.3.2. Feature-level Fusion

Feature-level fusion focuses on processing and integrating visual features that are more relevant than individual pixels. This strategy involves independently extracting high-level characteristics such as edges, textures, and shapes from the source images and then using a fusion method on them. This level offers better robustness against misregistration and provides more semantically meaningful outputs, leading to enhanced visual perception, decision-making precision. Techniques like SIFT (Scale-Invariant Feature Transform), Gabor filters, and gradient domain methods are widely used for this purpose.

Table 4.

Fusion Level	Description	Techniques	Advantages	Limitations	Typical Applications
Pixel-Level	Combines corresponding pixels from source images using arithmetic or transform-based methods	Averaging, Maximum Selection, DWT, NSCT, Shearlet	Preserves fine structural details; computationally simple	Highly sensitive to noise and registration errors	MRI-CT fusion in brain imaging; PET-CT for tumor mapping
Feature-Level	Extracts and fuses intermediate features such as edges, texture, and gradients before reconstruction	SIFT, CNN features, sparse representation, PCA	Robust to alignment errors; preserves semantic content	High computational complexity; depends on effective feature extraction	Alzheimer’s analysis using MRI-PET; tumor segmentation from hybrid MRI
Decision-Level	Fusion occurs after independent modality analysis at the classification or prediction stage.	Majority voting, Bayesian inference, ensemble fusion	Allows modality-specific models; lower dependence on pixel accuracy	Does not generate a fused image; interpretability is reduced	CAD systems for tumor detection, AI-based diagnosis integration

Feature-level fusion can be categorized according to the nature of the methodologies used and the combined features. These techniques can be broadly categorized into sparse representation methods and clustering-based methods. Sparse techniques express images as sparse vectors in a dictionary. These methods typically divide source images into patches, organize them into vectors, and then perform the fusion process. Clustering algorithms, including Quantum Particle Swarm Optimization and Fuzzy C-means, can split feature spaces and provide weighting factors for fusion.

3.4. Validation and Evaluation

Validation assesses the quality and clinical utility of the fused image using objective metrics and, where possible, expert evaluation. The most commonly used metrics include:

• Structural Similarity Index (SSIM): Measures perceived image quality and structural preservation.
• Peak Signal-to-Noise Ratio (PSNR): Evaluates image fidelity based on pixel intensity.
• Mutual Information (MI): Quantifies the amount of shared information between fused and source images.
• Entropy (EN): Reflects information richness in the fused image.
• Edge Preservation Index (EPI) and Spatial Frequency (SF): Measure edge clarity and texture detail [18].

Visual comparison remains essential in clinical validation. Radiologists or clinical experts often review Fusion outputs to assess diagnostic relevance and interpretability. Increasingly, evaluation also includes testing downstream tasks such as segmentation, classification, and localization to assess the practical benefits of fusion. MMIF is a multi-stage process where each phase requires tailored algorithms and quality checks to ensure robust performance, from preprocessing to validation. Fusion can be applied at different levels depending on the application, with each level offering unique advantages. Recent advances, especially in deep learning and transformer models, have improved registration accuracy and fusion robustness, allowing real-time, scalable implementations in clinical environments. The modular nature of the MMIF makes it adaptable, allowing hybrid strategies that combine pixel, feature, and decision-level fusion to maximize clinical efficacy [19]. The various steps in the entire workflow are depicted in Fig. (8).

4.1. Spatial Domain

Image fusion in the spatial domain is distinguished by simple computation and sound handling, which can be easily used and processed. In this process, image fusion is performed using methods that manipulate pixel values without first transforming them into frequency domain formats. Conventional spatial fusion methods include Principal Component Analysis (PCA), Intensity-Hue-Saturation (IHS) mapping, Brovey transformation, and high-pass filtering. Such methods have the advantage of simplicity, faster execution times, and more vivid color expression, making them suitable for real-time cases and applications where computational power is limited. For instance, SPECT-MRI fusion has opportune over the IHS-based methods, which enhance the blending of anatomical and functional views without involving complex computational routines, as shown in Fig. (10).

Spatial domain methods tend to cause edge softening, spectral distortion, and weak noise tolerance. Since these techniques lack frequency composition, they cannot capture subtle textural and high-frequency details, and thus their role in complex applications such as brain tumor localization or microvascular imaging cannot be fully realized. Initial Studies by Baraiya and Gagnani [20] and Parekh et al. [21] highlighted traditional techniques like Principal Component Analysis (PCA) and Brovey Transform, which are utilized in fields such as remote sensing and preliminary diagnostic systems. These methods highlighted fundamental spatial integration while exposing significant problems, including spatial distortions and inadequate spectral preservation. Morris and Rajesh [22] recognized the constraints of static fusion rules, advocating for image-adaptive approaches. Bhuvaneswari and Dhanasekaran [23] identified that conventional spatial domain techniques diminish image contrast, prompting the advancement of transform-domain solutions.

Li et al. [24] and Du et al. [25] made significant advancements by incorporating multi-scale transforms and edge-preserving filtering, thereby enhancing both structural integrity and contrast preservation. Their efforts established the foundation for hybrid spatial-transform techniques. Zhan et al. [26] tackled brain image fusion by implementing guided filtering and spatial gradient-based enhancements, which demonstrated enhanced performance in high-detail areas like cerebral tissue and lesions. Kotian et al. [27] conducted a comparative analysis, suggesting that a spatial/wavelet hybrid approach, which combined both spatial and spectral attributes, is a recommended method for general Multi-Image (MI) fusion. From 2018–2023, the field underwent substantial maturation when researchers such as Liu [28], Na [29], Saboori [30], and Pei [31] employed wavelet transforms, multi-resolution decomposition, and guided filtering to preserve feature layers and minimize distortion. They attained significant success in applications including PET-MRI, CT-MRI fusion, and brain pathology analysis. Tan [32], Chen [33], and Deepali [34] focused on enhancing low-complexity algorithms for rapid implementation in real-time systems, whereas Kong [35], Li [36], Feng [37], and Zhang [38] introduced sophisticated spatial techniques such as Framelet Transform, quasi-bilateral filtering, and structural dissimilarity metrics. These recent methodologies show enhanced performance in terms of structural retention, contrast preservation, and computational efficiency relative to State-of-the-Art (SOTA) fusion techniques. The progression of spatial domain techniques indicated a trend towards hybridization, multi-scale modeling, and adaptive spatial decision-making, thereby enhancing robustness in clinical applications such as lesion localization, radiotherapy planning, and neuroimaging diagnostics. Table 5 depicts the fusion strategy and area of application in the spatial domain.

4.2. Transform Domain

The transform or frequency domain has emerged as a powerful approach to mitigate several challenges associated with secure image fusion. Leading up to these transformations is the conversion of images using conversion algorithms such as the Fourier Transform, Discrete Wavelet Transform (DWT), Non-Subsampled Contourlet Transform (NSCT), or Laplacian Pyramid method. These changes separate images into separate spatial-frequency components, thus allowing for a more accurate separation of structural vs. textural aspects. Thereafter, when fusion is carried out in the transform domain, the reconstructed final image features an inverse transformation. Since frequency-based approaches excel at maintaining edge sharpness and texture information, they are suitable for clinical endeavors involving precise structure separation, such as identifying tumor edges. The basic process of decomposition and application of the inverse transform is shown in Fig. (11).

Preliminary research, including that of Singh et al. [39], tackled the shortcomings of real-valued wavelet transforms, specifically shift sensitivity and inadequate directionality by utilizing the Dual-Tree Complex Wavelet Transform (DCxWT). This method markedly enhanced fusion results by maintaining phase and directional specifics. In the same year, Ganasala et al. [40] introduced an image fusion technique for CT and MR images with the Nonsubsampled Contourlet Transform (NSCT), resulting in improved visualization of soft tissue and osseous structures. Bhateja et al. [41] advanced this research by developing a two-stage architecture that included Stationary Wavelet Transform (SWT) and NSCT, utilizing PCA for redundancy minimization and contrast enhancement.

To more effectively capture edge information, Srivastava et al. [42] utilized the Curvelet Transform, which showed enhanced efficacy in maintaining anisotropic features and improving visual perception via a local energy-based fusion rule. Xu et al. [43] presented the Discrete Fractional Wavelet Transform (DFRWT), which, due to its fractional order parameters, facilitated adaptive decomposition and enhanced multimodal image fusion efficacy. Gomathi et al. [44] demonstrated that the NSCT method provides improved frequency decomposition while efficiently preserving high-frequency image components. In contrast, Liu et al. [45] showed that the NSST is highly effective in maintaining texture and detail. Numerous scholars investigated shearlet-based and nonsubsampled techniques for improved frequency decomposition. Gambhir et al. [46] also demonstrated that the fused images obtained from the proposed method offer better clarity and enhanced information, making them more useful for quick diagnosis and improved treatment of diseases.

Li et al. [47] improved fusion efficacy by utilizing NSST with a novel fusion rule that reduced blocking and blurring artifacts through local coefficient energy and mean-based methodologies, while Ganasala and Prasad [48] implemented SWT with Transformation Error Minimization (TEM) to improve image fusion quality while decreasing computational load. Goyal et al. [49], Khare et al. [50], and Kong et al. [51] enhanced image fusion by preservation of edges, textures, and structural boundaries, minimizing artifacts with RGF/DTF, median-based NSST, and Framelet Transform, respectively. The results of the work by Diwakar et al. [52] demonstrated that non-conventional transform domains yield improved outcomes when integrated with various spatial domain architectures. An overview of various transform techniques is presented in Fig. (12). Although distinguished by their quality enhancements, frequency domain approaches are accompanied by increased computational costs and high requirements for exact image registration, which may be quite challenging to implement in practice. The key contributions are presented in Table 6.

4.3. Sparse Representation

The sparse representation-based fusion method provides a more compacted, higher information-to-data ratio. Sparse methods examine images by projecting them onto a dictionary of basis functions, which can be trained or predetermined. The focus is on a small set of coefficients that correspond to significant aspects of the image. By combining coefficients from different modalities, as shown in Fig. (13), the method constructs a fused image with enhanced predominant structure and contrast.

It has gained significant interest because of its ability to maintain prominent image elements such as edges and textures while minimizing redundancy. A typical SR-based image fusion process comprises the following steps:

1. Extraction of patches from source images (such as CT and MRI).

2. Encoding of each patch with a trained dictionary.

3. Integration of sparse coefficients by a criterion (e.g., max-selection or averaging).

4. Reconstruction of amalgamated patches to achieve the final image [53].

Zhang et al. [54] described how the SR model forms a dictionary through a sparse linear combination of prototype signal models. According to Joint Sparse Representation (JSR), several signals from several sensors of the same scene constitute an ensemble. While each signal possesses an innovative sparse component, they all share a common sparse component. As compared to SR, the JSR presents reduced complexity. Zong et al. [55] proposed a sparse method using categorized image patches based on their geometric orientation. Liu et al. [56] made a significant early contribution by introducing Convolutional Sparse Representation (CSR), an alternate representation of SR using the convolutional form, aiming to achieve SR of a complete image rather than a localized image patch.

Liu et al. [57] presented Convolutional Sparsity-based Morphological Component Analysis (CS-MCA). In contrast to the conventional SR model, which relies on a single image component and overlapping patches, the CS-MCA model can concurrently accomplish multicomponent and SRs of the source images by amalgamating MCA and CSR within a cohesive optimization framework. Shabanzade and Ghassemian [58] employed sparse representation in the NSCT (Nonsubsampled Contourlet Transform) domain in a hybrid context. Their approach integrated low-frequency coefficients through sparse coding and high-frequency components using max-selection, resulting in both multiscale decomposition and sparse adaptability. Alternative hybrid methodologies have integrated SR with PCNN (Pulse Coupled Neural Networks), and clustering-based multi-dictionary learning, which allocates region-specific dictionaries (e.g., for edges versus smooth regions) to enhance localization and context-aware fusion. The various categories of sparse methods are also described in Table 7.

SR continues to serve as a robust intermediary solution between conventional and deep learning-based fusion approaches. Its unsupervised characteristics and adaptability render it especially advantageous in contexts with limited labeled data. As highlighted in the paper by Hermessi et al. [59], SR remains fundamental to numerous advanced approaches, particularly in hybrid fusion architectures where it enhances contrast, clarity, and feature retention [60]. When sparse representation is utilized alongside multi-scale techniques, such as NSCT, Laplacian Pyramid, Dual Tree Complex Wavelet, or Curvelet, it effectively improves fine details, boosts contrast, and minimizes noise. According to a review by Zhang et al. [61], sparse representation proved superior to traditional multi-scale approaches in terms of retaining image structures and defining edges. It learns an overcomplete dictionary from a set of training images, resulting in more stable and significant results. A smart blending approach that combines SR with SCNN to overcome flaws such as edge blurriness, diminished visibility, and blocking artifacts was proposed by Yousif et al. [62]. The results have demonstrated that the proposed method is superior to previous techniques, particularly in suppressing the artifacts produced by traditional SR and SCNN methods. Table 8 summaries the advantages and challenges of this domain. SR-based approaches encounter constraints, including:

• The high computational cost results from sparse optimization procedures.
• Block artifacts arising from independent patch-based judgements.
• Sensitivity to misregistration, as the majority of approaches presume aligned inputs.

4.4. Deep Learning Fusion Methods

The last few years have seen a paradigm shift in MMIF research, with deep learning shaping into an enabler of automated multimodal fusion through end-to-end learning architectures. Although Deep Neural Networks (DNNs) have yielded exceptional results in learning multi-level feature representations from raw image data, the architects of the networks are finding it difficult to give them meaningful names. Notable compositions of DNNs have been CNNs, U-Nets, and Generative Adversarial Networks (GANs). The training objective of these models incorporates optimal structural alignment, maintaining semantic information, and enhancing disease identification.

The initial implementation of CNNs in medical image fusion was presented by Liu et al. [56], exhibiting superior performance as compared to spatial and transform domain approaches. CNNs' design effectively extracts spatial and textural information, yet they necessitate extensive annotated datasets and intricate tuning processes. Yu Liu et al. [63] emphasized the dual roles of fusion rules and activity level estimation, using local filters and clarity maps to guide the amalgamation of high-frequency features. Gibson et al. [64] insisted upon the importance of deep networks for neurological diagnosis through pixel-level fusion.

Xia et al. [65] accomplished a significant advancement by integrating multiscale decomposition with CNNs, facilitating high/low-frequency discrimination and enhanced multi-resolution fusion. Wang et al. [66] employed Siamese CNNs to create activity-guided weight maps, resulting in structurally intricate fused images. To overcome batch processing restrictions, Li et al. [67] created CNN-based frameworks that facilitated real-time, multimodal fusion, therefore improving efficiency and detail retention. These approaches provided practical utility in clinical environments requiring simultaneous processing of many images. The fusion approach using CNN and autoencoders is depicted in Fig. (14).

Chuang et al. [68] introduced a fusion framework that integrated U-Net and Autoencoder architectures (FW-Net), where the encoder-decoder configurations exhibited U-Net’s skip connections. The design, initially limited to CT-MRI, has the potential for future PET-MRI and SPECT fusion. Kumar et al. [69] examined the utilization of CNNs for thermal-visual fusion in remote sensing, showing cross-domain relevance. Shafai et al. [70] introduced a hybrid fusion technique in which CNNs integrated three images using traditional methods, leading to less redundancy and enhanced semantics.

To address the issues of semantic degradation in fused outputs, Ghosh [71] introduced a dual U-Net FW-Network that prioritized semantic-level fusion. This approach markedly enhanced clinical utility in disease localization and segmentation. Deep learning-based image fusion is a powerful paradigm that provides excellent visual fidelity, task adaptability, and end-to-end automation. However, accessible data sets, interpretable models, and the creation of computationally efficient architectures that work across modalities are necessary for the advancement.

A CNN-based multimodal fusion approach designed for oncology not only succeeded in improving diagnostic accuracy but also outperformed current methods in both subjective and objective metrics. Due to their generative properties, GANs can create high-resolution fused images that preserve both anatomical structure and functional intensity. The dependence of such models on large sets of labeled data and their tendency to overfit limited data variability are serious issues. The lack of interpretability is also a concern for these models in clinical settings. Fig. (15) demonstrates the domain-wise publication trends across Web of Science, in multimodal image fusion from 2000 to 2024. It indicates the rapid proliferation of deep learning and hybrid domains in this period. Driven by the endorsement of CNNs, U-Nets, and GANs, the adoption of deep learning methodologies has shown remarkable expansion. The explosion in growth indicates a shift in the sphere towards using data-driven fusion techniques, characterized by a strong understanding and fusion of intricate anatomical and functional relationships.

At the same time, the development of hybrid approaches using such spatial, frequency, and deep learning approaches is relatively stable, demonstrating their ability to handle the dissimilar nature of medical data. Due to their modular structure, these techniques provide superior flexibility and precision, which improves overall fusion performance. Collectively formed by these advances, these fields have outperformed traditional approaches and indicate a shift towards smarter, more holistic fusion methods for medical applications. MMIF's future research appears to be majorly focused on DL-based and hybrid techniques, especially where complex cases of oncology and neuroimaging are involved. The key contributions from DL are presented in Table 9.

4.5. Hybrid Domain

To benefit from diverse domains and tend towards their minimal negative effects, hybrid fusion approaches have attracted much attention. Such approaches utilize specific domain expertise, such as NSCT for frequency analysis and CNNs for feature extraction, or the use of neural networks. Simultaneously managing the spatial, frequency, and contextual information, hybrid systems are capable of creating informative fused images.

Chen Y [72] established a foundational framework by combining Nonsubsampled Contourlet Transform (NSCT) and Dual-Tree Complex Wavelet Transform (DTCWT), which was supplemented by Pulse-Coupled Neural Networks (PCNN) for the creation of composite images. This represented a crucial advancement in integrating handcrafted and adaptive transformations to tackle challenges such as image noise and resolution discrepancies. Wang et al. [73] improved diagnostic precision by employing NSST and adaptive decomposition frameworks that consider high/low frequency layers and dynamic textural features, respectively. These works acknowledged that conventional static decomposition inadequately reflects contextual differences across modalities. Bhateja et al. [74] advanced the hybrid approach by introducing shearlet and NSCT-based fusion for PET/SPECT, MRI, and CT images, including contrast enhancement and weighted PCA to preserve multispectral integrity. Du et al. [75] and Zhao et al. [76] tackled modality-specific inconsistencies by developing distinct methodologies for MRI and PET, thereby enhancing fusion accuracy through edge-based weighting.

Maqsood and Javed [77] employed two-scale decomposition with spatial gradients to enhance edges. Wang Z et al. [78] and Liu Y et al. [79] employed adaptive sparse coding and total variation transformations, enhancing detail retention while diminishing high-frequency noise. Yadav [80] utilized independent and principal component analysis using a wavelet framework but observed residual noise and artifacts. In his study, Nath [81] showed that the Stationary Wavelet Transform (SWT) surpasses the standard Discrete Wavelet Transform (DWT) in terms of entropy preservation. Huang et al. [82] emphasized that while hybrid models enhance established frameworks, problems persist, especially in feature extraction and color distortion. These constraints stimulated investigation into hybrid deep learning strategies. Harmeet et al. [83] integrated ANFIS with cross-bilateral filtering, resulting in enhanced entropy (2.92), and suggested volumetric fusion for future neuroimaging applications. Harpreet et al. [84] conducted a systematic review emphasizing the necessity for precise, reliable, and interpretable fusion techniques.

Polinati et al. [85] and Li et al. in 2021 [86] introduced innovative frameworks employing Variational Mode Decomposition (VMD), local energy gradients, and bilateral filtering, all designed to improve clarity and reduce luminance deterioration. Zhu et al. [87] and Alseelawi et al. [88] concentrated on enhancing NSST-DTCWT-based fusion techniques to achieve equilibrium between texture and structural integrity. Alseelawi’s work prominently highlighted velocity and visual excellence, using PCNN as a guiding framework. Kalamkar and A Mary [89] utilized transfer learning with DWT, attaining enhanced structural similarity index values. Goyal and Dogra [90] developed a cross-bilateral, edge-preserving fusion filter that reduces artifacts while improving image quality. Likewise, Shafai et al. [91] amalgamated CNN with three traditionally fused image sets to improve overall image quality and diagnostic efficacy. Zhou et al. [92] tackled issues of brightness degradation and detail retrieval by the application of NSST and Improved Structure Tensor (IST) decomposition, resulting in enhanced contrast while differentiating smoothing, edge, and corner layers.

Kittusamy et al. [93] utilized Joint Sparse Representation (JSR) and NSCT to attain enhanced high perceptual clarity in MRI-CT fusion. Dinh [94, 95] formulated two complementary models to address inadequate contrast and edge degradation through a three-scale decomposition and local energy-based fusion rules. Balakrishna et al. [96] evaluated nine DWT-based combinations, validating the method’s efficacy in the detection of abnormalities and clinical planning. Moghtaderi et al. [97] suggested a Multilevel Guided Edge-Preserving Filtering (MLGEPF) technique that combined computational expense with structural accuracy. Zhao et al. [98] proposed a novel method based on three-scale frequency decomposition along with SSIM-optimized feature blending, which significantly facilitates the process of fusing MRI and PET images for brain tumor examination. Various hybrid combinations, along with their advantages and disadvantages, are described in Table 10.

Table 11 signifies the evolving emphasis of research within Multimodal Medical Image Fusion (MMIF) over three distinct phases. Spatial domain methods prevailed in the initial period but have since been replaced by more sophisticated techniques. Frequency domain techniques have shown significant output growth and have sustained stability. Sparse representation techniques have transitioned from minimal utilization in their first stages to considerable significance in recent years.

Deep learning was nearly non-existent before 2016, but has become the dominant methodology. Hybrid approaches, which amalgamate multiple techniques, have experienced significant expansion, signifying a trend towards integrative and adaptive fusion approaches. A comprehensive analysis of Multimodal Medical Image Fusion (MMIF) domains emphasizing their strengths, weaknesses, and principal application areas is highlighted in Table 12.

5.1. Surgical Planning Using Synthetic CT from MRI

A recent study has explored the feasibility of generating synthetic Computed Tomography (CT) images of the lumbar spine from Magnetic Resonance Imaging (MRI) using a patch-based convolutional neural network. This approach aimed to support pre-operative planning without exposing patients to ionizing radiation using deep learning. The study involved three cases and demonstrated that deep learning-enabled MRI-to-CT conversion offers high-quality structural visualization, reducing radiation exposure compared to conventional CT scans (typically 3.5–19.5 mSv for spine imaging) [99].

5.2. Orthopedic Implant Design and Additive Manufacturing

The combination of CT and MRI has also been pivotal in the design of patient-specific orthopedic implants. These modalities are employed to capture comprehensive anatomical data, including size, shape, texture, and bone density. This facilitates the development of custom implants and allows for the reconstruction of traumatic bone defects through additive manufacturing technologies [100].

5.3. Management of Complex Fractures Using Rapid Prototyping

Fusion imaging has enabled advances in Rapid Prototyping (RP) and 3D reconstruction, particularly for complex fractures in anatomical regions such as the joints, acetabulum, and spine. RP assists surgeons in visualizing fracture geometry preoperatively, which improves the accuracy of anatomical reduction and reduces surgical time, anesthesia duration, and intraoperative blood loss [101].

5.4. Surgical Effects of Resecting Skull Base Tumors Using Preoperative Multimodal Image Fusion

A retrospective study was conducted on 47 patients with skull base tumors. Preoperative CT and MRI data acquisition were performed using GE AW workstation software for co-registration, fusion, and three-dimensional reconstruction of the brain. The surgical plan was designed based on multimodal images. The application of the fusion technique provided essential visual guidance in skull base tumor surgery, assisting neurosurgeons in accurately planning the surgical incision and precisely resecting the lesion [102].

5.5. Trigeminal Neuralgia Treatment with Integrated Neuro-Navigation

In 13 patients undergoing percutaneous radiofrequency trigeminal rhizotomy, the fusion of MRI and intraoperative CT (iCT) images provided improved anatomical delineation of the trigeminal cistern. This integration supports more accurate targeting, especially in recurrent cases of trigeminal neuralgia. The study emphasized the benefits of fusion-guided neuro-navigation and suggested that longer follow-ups are needed to assess long-term therapeutic efficacy [103].

5.6. Frameless Stereotactic Radiosurgery with Gamma Knife Icon

In a clinical series of 100 patients, MRI-CBCT (Cone Beam CT) fusion was utilized for frameless stereotactic radiosurgery using the Gamma Knife Icon. MRI provided superior soft tissue contrast, while CBCT served as the baseline for stereotactic registration. The adaptive dose distribution was computed based on the patient's real-time geometry after fusion, optimizing treatment accuracy. This method overcomes the limitations of traditional CT-guided radiotherapy by ensuring better alignment of tumor and anatomical landmarks [104, 105].

5.7. Image Fusion in Precision Medicine for Oncology

In the context of precision oncology, fusion imaging technologies are increasingly used to enhance diagnostic accuracy and individualized treatment planning. The integration of CT, MRI, and PET enables better visualization of tumor morphology and metabolic activity, facilitating more accurate localization and classification of malignancies. Such multimodal fusion approaches are pivotal in improving therapeutic outcomes and reducing harm to healthy tissue [106].

5.8. Image Fusion in the Diagnosis and Treatment of Liver Cancer

The rapid advancement of medical imaging has facilitated the effective application of image fusion technology in diagnosis, biopsy, and radiofrequency ablation, particularly for liver tumors. Employing image fusion technology enables the acquisition of real-time anatomical images overlayed with functional images of the same plane, thereby enhancing the diagnosis and treatment of liver cancers. This study provides a comprehensive examination of the fundamental concepts of image fusion technology, its application in tumor therapies, specifically for liver cancers. It finishes with an analysis of the limitations and future prospects of this technology [107].

6.1. Subjective Evaluation

Subjective analysis entails expert evaluators who assess the images according to their visual characteristics, such as object clarity, spatial detail, geometric consistency, and color equilibrium. This method, although indicative of human perception, is compromised by observer bias, environmental reliance, and lack of reproducibility, rendering it less reliable for quantitative comparison.

6.2. Objective Evaluation

It utilizes the mathematical and statistical metrics to measure image quality quantitatively. These metrics yield reliable, quantifiable, and algorithm-independent outcomes, making them crucial for the consistent and automated validation of fusion approaches. They are further categorized into two categories such as those employing a reference image and those without a reference image, as depicted in Table 13, with each parameter demonstrating distinct characteristics. The objective image fusion performance characterization utilizing the gradient information is also taken into account. This provides an in-depth analysis by assessing total fusion performance, fusion loss, and fusion artifacts as represented in Table 14. It is noted that the total fusion performance is depicted by the sum of these three, and the result is unity, as shown in the formula [110-113].

7.1. Research Trends in Modality Integration

The growing interest in MMIF studies from 2015 to 2024, analyzed according to the three most commonly used modality groups, MRI–PET, MRI–CT, and MRI–SPECT, is illustrated in Fig. (18). MRI–PET fusions report the greatest number of publications, with a significant boost in 2021, representing high influence in oncology and neuroimaging. The constant growth of MRI–CT literature (although its volume is less compared to MRI–PET) indicates growing use for surgical planning, and tasks where both soft and hard tissue information is needed.

The MRI–SPECT fusion approach is the least discussed, possibly due to the lower spatial resolution and the comparatively narrower use of SPECT compared to PET. The possible correlation of periodic changes in MRI–PET and MRI–CT domains with the emergence of new advances, such as convolutional neural networks or transformer models, indicates that these improvements have likely reinforced the fusion and registration processes. The data shows an increasing interest in hybrid imaging as the main research topic, highlighting the clinical necessity for accurate diagnostics based on the integration of complementary modalities.

8.1. Data Scarcity and Imbalance in Public Datasets

The foundation of any robust MMIF model is a sufficiently large and diverse dataset. However, available multimodal image datasets such as AANLIB, ADNI, TCIA, and MIDAS lack balance in organ representation, modality pairing, and demographic diversity. While these repositories provide aligned pairs (e.g., MRI–PET), they often suffer from incomplete labeling and variation in acquisition protocols. For example, Venkatesan et al. [15] emphasized that most fusion research focuses on brain datasets (MRI-CT), leaving thoracic, abdominal, and musculoskeletal fusions underrepresented. The imbalance leads to biased models that cannot generalize across anatomical regions. Moreover, annotating multimodal images, particularly PET and SPECT, requires domain expertise and is cost-intensive, limiting supervised learning approaches. Recent works, such as those by Tirupal et al. [8], propose the use of fuzzy sets and unsupervised learning to overcome the lack of labels, while Dinh [94, 95] explores decomposition techniques to create proxy supervision.

8.2. Registration and Inter-modality Inconsistency

Accurate registration is pivotal in MMIF, as misalignment between modalities can propagate errors into every subsequent fusion stage. Unlike unimodal registration, where intensity similarities guide optimization, multimodal images exhibit diverse characteristics (e.g., CT for density, MRI for soft tissues, PET for metabolism), making intensity-based alignment ineffective. Errors in registration introduce artifacts, structural shifts, and contrast mismatches, especially at organ boundaries. Goyal et al. [90] and Ibrahim et al. [109] report that deformable and affine transformations often underperform when the inter-modality gap is large, such as in PET–MRI or SPECT–CT fusion. Furthermore, hybrid methods that employ wavelet or Laplacian transforms are highly sensitive to minor misregistrations, causing blur or duplication of anatomical features. Although deep-learning-based spatial transformers, such as the Swin Transformer by Ghosh [71], offer improved alignment, these solutions are computationally expensive and poorly validated across multiple organs. Thus, the modality inconsistency and lack of robust, generalized registration algorithms significantly hinder the clinical reliability of MMIF systems.

8.3. Computational Overhead in Deep Learning-based MMIF

While Deep Learning (DL) methods, particularly CNNs, GANs, and U-Nets, have revolutionized image fusion, they bring substantial computational burdens. Training these models requires massive datasets, high-end GPUs, and time-intensive tuning. Li et al. and Zhang et al. [60, 61] showed that deep CNNs produce sharper and semantically richer fused images, but the cost of training (over 20 million parameters) and risk of overfitting persist. These limitations restrict the practical deployment of MMIF in real-time or edge-based medical devices like portable ultrasound or emergency CT units. Efforts to develop lightweight architectures, such as Mobile Net or efficient transformer hybrids, have been partially successful. For instance, Kalamkar & Geetha [89] propose transfer learning-based MMIF using pretrained models to reduce training time. However, performance degradation and sensitivity to modality shifts are still major issues. Hybrid DL models, such as PCNN + NSCT or GANs over NSST coefficients, further increase architectural complexity, which limits scalability and increases latency during clinical inference.

8.4. Ethical and Privacy Concerns in Multimodal Data Use

The use of medical images, especially in a multimodal and longitudinal context, raises significant privacy and ethical issues. Most MMIF datasets are derived from patients with chronic or terminal conditions, making de-identification difficult due to unique anatomical signatures (e.g., tumors or prostheses). Compliance with GDPR, HIPAA, and institutional IRB guidelines restricts dataset sharing. El-Shafai et al. [70] observed that fewer than 20% of fusion studies in recent years could access external datasets, limiting generalizability and leading to model bias. Federated learning, as explored in Goyal et al. [90], is a promising avenue to train models across silos without data exchange. However, challenges in harmonizing modalities, synchronizing update cycles, and addressing vulnerability to gradient attacks remain. Ethically, there's a lack of transparency in fusion models. Black-box CNNs may produce composite images that suppress or misrepresent subtle pathologies. There are also legal uncertainties in attributing diagnostic responsibility when AI-generated fusion images are used clinically. To address this, future MMIF research must include explainable AI (XAI) methods, probabilistic uncertainty maps, and formal ethical frameworks that define accountability for AI-assisted diagnostics [115]. While the field of MMIF continues to evolve rapidly, these four key challenges, data imbalance, registration inconsistency, computational overhead, and ethical constraints, remain persistent obstacles. Addressing these will require interdisciplinary collaboration between data scientists, radiologists, ethicists, and software engineers. The integration of robust registration models, edge-efficient architectures, federated training schemes, and ethically compliant data pipelines will be vital to unlocking MMIF’s full potential.

9.1. Explainable AI (XAI) in MMIF

Explainable artificial intelligence is a technique for AI-powered diagnosis and analysis that ensures features such as ethics, transparency, and accountability in the traditional approach to AI. This will lead to outcome tracing and model improvements in health care. EXAI relies on feature extraction to make the model more explainable and interpretable. EXAI proposed a self-explanatory framework based on design principles for understanding and predicting the behavior of ML/DL models. Although deep models such as CNNs and Transformers outperform traditional algorithms, their “black-box” nature hinders clinical trust. Recent work integrates attention maps, Layer-wise Relevance Propagation (LRP), and Grad-CAMs into MMIF pipelines, allowing radiologists to verify the influence of fused features during diagnostics [115, 116].

9.2. Quantum Image Fusion

Quantum computation in medical image processing has improved edge detection, segmentation, watermarking, encryption, and classification. A quantum edge detection technique using superposition and fuzzy entropy can better detect strong and weak edges. Automatic hippocampal segmentation using a Quantum-Inspired Evolutionary Algorithm (QIEA) yielded good correlation between segmented and microscopic images. A hybrid technique using QPSO and fuzzy k-nearest neighbours enhances cervical cancer cell classification on the Herlev dataset for feature selection and classification. The hybrid method decreased features from 17 to 7, outperforming Naive Bayes and SVM with a 2% to 11% increase in accuracy. These inventions show how quantum computing might improve medical image analysis and diagnosis precision [117].

9.3. Federated and Privacy-preserving MMIF

Machine learning algorithms learn to solve problems autonomously based on the data provided to them, but they require extensive training to do so. Personal data in training datasets for AI systems, especially for health care applications, must be considered. Data from special categories (including health data) requires more safeguards than common data under the General Data Protection Regulation (GDPR). AI developers and users in health care are especially affected by this issue. With increasing data privacy regulations (e.g., GDPR, HIPAA), centralized MMIF training on patient data faces ethical and legal barriers. Federated learning frameworks (e.g., FL-MedFuseNet) enable MMIF model training across distributed hospital nodes without sharing raw data. This decentralized approach preserves privacy and supports continual learning from real-world clinical inputs [118].

9.4. Robotic Surgery and Smart Operating Rooms

The integration of MMIF into AI-assisted robotic surgeries is a fast-growing field. As the population ages, the prevalence of spinal degenerative illnesses such as lumbar disc herniation and spinal stenosis increases the need for accurate and safe spinal operations. The spine intricacy makes surgery difficult, making robotic help invaluable. Pedicle screw implantation is performed using surgical robots like TiRobot, Mazor, Da Vinci, ROSA, Excelsius GPS, and Orthbot, which enhance precision, reduce operative time, decrease the chances of hemorrhage, and minimize radiation exposure. These approaches use unimodal CT scans, which cannot detect nerves and intervertebral discs. The incorporation of a multimodal image fusion (CT/MR)- based software system for intraoperative navigation expands the horizons of robotic spinal surgery. Intuitive Surgical’s Da Vinci system already shows early adoption of MMIF-enhanced visual pipelines, although this is mostly experimental. Future research could focus on ultra-low-latency hardware-driven MMIF solutions embedded into surgical robots. NVIDIA’s Clara, a GPU-accelerated platform, is used to segment and diagnose cardiac images in real time. To improve processing speed and accuracy, the system takes advantage of powerful DL algorithms explicitly designed for cardiac imaging. Using rigorous data processing and a deep learning model, it can perform exact partitioning of cardiac components and identify positioning defects, resulting in the diagnosis of rapid heart arrhythmia [119, 120].

9.5. MMIF-based Watermarking for Tele-health Applications

Watermarking in medical image fusion involves embedding a hidden signal (such as patient ID, diagnosis detail, time stamps, or institutional information) into a fused image generated from multiple modalities. This ensures date integrity, ownership authentication, and secure communication in telemedicine or cloud environments. Embedding of watermark is done by techniques like Redundant Discrete Wavelet Transform (RDWT), Singular Value Decomposition (SVD), or NSCT. The integration of deep learning needs further exploration to create adaptive and intelligent watermarking methods [121].