Data and AI Research

My research in AI has spanned the various phases of AI from statistical machine learning, through deep learning to currently generative and agentic AI. It dates back to the 1980s where I first explored the concept of attentional selection in object recognition leading to early work in color, texture and saliency-based object detection and recognition. It was also during this time, I explored problems in knowledge representation proving the decidability of monotone inference relations. A lark project done during those days under the guidance of Prof. Rod Brooks, produced the design of the first vacuum cleaner robot (Roomba by iRobot).

My current work in AI covers the following topics:

• Multimodal RAG Search
• Compression in vector databases
• Rethinking relational and vector databases
• Bioinspired episodic and semantic memory models (VLM, SemCLIP)
• Cross-modal Hopfield networks
• Fact-checking of generative AI

Topics explored in earlier eras include:

• OCR and Document analysis and recognition
• Content-based image and video retrieval

Fact-checking of generative AI

Despite the promise of generative AI in report generation, retaining factual correctness with minimal hallucinations has been challenging. Current methods of hallucination reduction such as DPO and PPO are applied at training or fine-tuning. Currently, there aren't many methods available at inference time to fact-check generative AI in a manner independent of the underlying generative AI. We have been developing fact-checking models for generated reports from imaging data. We are also developing methods to correct such reports with the help of language models. The key idea here is to generate a synthetic data designed to elicit the broad class of errors made by generative AI models and use this train a fact-checking model. We are also working on an automatic report correction method that uses both image and textual information in automated reports to spot identity and location errors of concepts mentioned through the use of fact-checking models. Promising results are already being seen in X-ray datasets.

• R. Mahmood, P. Yan, Tanveer Syeda-Mahmood, “Automatic Correction of AI Reports using Fact-Checking Model-guided LLMs,” in Proc. NeurIPS Workshop on GenAI for Health: Potential, Trust, and Policy Compliance, Dec. 6, 2025.
• R. Mahmood, Diego Machado-Reyes, Joy Wu, Parisa Kaviani, Ken C.L. Wong, Niharika D'Souza, Mannudeep Kalra, Ge Wang, Pingkun Yan, Tanveer Syeda-Mahmood, “Phrase-grounded Fact-checking for Automatically Generated Chest X-ray Reports,” in Proc. Medical Imaging and Computer-Assisted Intervention (MICCAI), Sept. 2025.
• R. Mahmood, D. Machado-Reyes, J. Wu, P. Kaviani, Ken C.L. Wong, N. D'Souza, M. Kalra, Ge Wang, P. Yan, Tanveer Syeda-Mahmood, “Evaluating automated radiology report quality through fine-grained phrasal ground of clinical findings ” in IEEE ISBI, Apr. 2025.

Compression in vector databases

Vector databases have become ubiquitous in the era of generative AI for enabling semantic search for downstream use in general search, LLM-based chatbots, and retrieval augmented generation (RAG) of context-sensitive responses. When such databases are used to house all of an organization's vectorized data, this leads to excess storage increase, particularly for large size embeddings. Applying compression methods, however, is known to lower the quality of search by as much as 50%. Developing techniques for vector compression that preserve the accuracy of an uncompressed data search while still reducing the memory and storage footprint are desirable. In our research, we are developing such methods both within the product quantization (PQ) formulation as well as using binary quantization frameworks. We show that among all permutation transformations, the sorting transformation minimizes L2 distance and maximizes similarity measures such as cosine similarity and Pearson correlation for vector data. Applying sorting transformation with vector quantization can substantially reduce compression errors, and offer search performance comparable to uncompressed search. The small overhead in storage for the sorting permutation is balanced by the reduction in the number of clusters due to the sorting transformation. Our papers on this topic:

• H. Wang and T. Syeda-Mahmood, “Vector data search with sorting transformation,” in ICML Workshop on Vector Databases, Vancouver, July 18, 2025.
• H. Wang, and T. Syeda-Mahmood, “Vector quantization with sorting transformation,” in IEEE International Conference on Big Data, Washington, D.C., Dec. 2024. Also in NeurIPS Workshop on Compression, Vancouver, Dec. 2024.

Bioinspired episodic and semantic memory models

Vision-language models (VLM) bring image and textual representations close together in a joint embedding space, which is useful for tagging and retrieval from content stores. However such associations are not very stable in that a synonymous textual query does not retrieve the same set of images or with a high degree of overlap. This is due to the absence of linkages between semantically related concepts in vision-language models. In contrast, the episodic memory store in the brain has linkages to the semantic conceptual memory subsystem which helps in both the formation and recall of memories. We exploited this paradigm to link a VLM to a semantic memory thereby producing a new semantic vision-language model called SemCLIP. Specifically, we developed a semantic memory model for the language of object-naming nouns reflecting their semantic similarity. We then linked a vision language model to the semantic memory model through a semantic alignment transform. This led to a richer and more stable understanding of the concepts by bringing synonymous visual concepts and their associated images closer. Both the semantic memory model and the alignment transform can be learned from word knowledge sources thus avoiding large-scale retraining of VLMs from real-world image-text pairs. The resulting model is shown to outperform existing

Content-addressable memories such as Modern Hopfield Networks (MHN) have been studied as mathematical models of auto-association and storage/retrieval in the human declarative memory, yet their practical use for large-scale content storage faces challenges. Chief among them is the occurrence of meta-stable states, particularly when handling large amounts of high dimensional content. We introduced Hopfield Encoding Networks (HEN), a framework that integrates encoded neural representations into MHNs to improve pattern separability and reduce meta-stable states. We show that HEN can also be used for retrieval in the context of hetero association of images with natural language queries, thus removing the limitation of requiring access to partial content in the same domain.

• T. Syeda-Mahmood, K.C.L. Wong, S. Kashyap, N Dsouza, L. Shi, R. Mahmood, H. Wang, “SemCLIP: A semantic memory-aligned vision language model,” in Proc. Machine Learning Research (PMLR) Journal, Also in NeurIPS, UniReps Workshop on Unified Representations, Dec. 6 2025.
• S. Kashyap, N Dsouza, L. Shi, K.C.L. Wong, H. Wang, T. Syeda-Mahmood, “Modern Hopfield Networks meet Encoded Neural Representations – Addressing Practical Considerations,” in TMLR and NeuIPS 2024 UniReps Workshop, 2024.

Content-based Image and Video Retrieval

Database systems of the 80s primarily catered to structured data (numeric, ascii). However, with the growth of digital multimedia data (images, videos, audio), there was a need to develop multimedia data management systems that were based on automated content extraction and search. Our early work in content-based image and video retrieval addressed challenging issues of feature extraction, indexing structure creation and search using color, texture, and line groupings in images, and audio-video topics in video. Several efficient indexing data structures such as the interval hash tree, which allowed complex query transformations to search and localize objects under changes in pose and appearance. We also developed an early theory of action recognition modeling actions as objects and methods for combining audio and visual information to search for topics which paved the way for many commercial video search engines and TREC video retrieval evaluation benchmark at NIST later. The work in the CueVideo project laid the foundations for modern day video browsing and playing over the web by Yahoo and later Youtube. QBIC itself was sold by IBM as an early product in content management and the paper in IEEE Computer alone was one of the highest (6000 citations) before deep learning era.

Healthcare AI Research

Over the last 25 years we have worked in several areas of medical imaging focusing on end-to-end solutions to many problems in modality interpretation ranging from mode and viewpoint recognition, to organ segmentation and anomaly detection. These were incorporated into the development of diagnostic decision support systems covering cardiology, neuro-radiology, chest radiology and cancer. The work in clinical decision support can be grouped the work done over the years into a few research focus areas, namely:

• Cardiovascular imaging
• Chest X-rays
• Brain imaging
• Multimodal fusion for cancers and other diseases
• Precision AI for interventional imaging
• Healthcare informatics
• Digital twins

Cardiovascular image analysis

The work in cardiovascular image analysis started during an exploratory project called AALIM at IBM in 2004. The field of automated clinical decision support had seen little growth from the original rule-based expert systems since the 70s until in 2005. Anticipating the roll out of electronic health record systems in hospitals, I came up with the idea of patient similarity-guided decision support. Here, the knowledge from past diagnosed patients was leveraged to drive diagnosis recommendations for current patients by searching for clinically similar patients. To demonstrate the idea, we chose the domain of cardiology and explored virtually all modalities and modes used in cardiology ranging from auscultation data, EKG, Echocardiograms, coronary angiograms, cardiac MRI, cardiac CT, and nuclear cardiology. To take a holistic approach, we also considered information from textual reports (discharge summaries, modality reports), labs data and clinical information including problem lists and billable diagnosis. This was possible due to a collaboration we set up with Kaiser Permanente where we also pilot deployed our first multimodal cardiac clinical decision support system at the San Francisco Medical Center covering over 3000 patients. The AALIM system was also later deployed for patient-similarity-based selection of patients for clinical trials at the Cedars-Sinai Medical Center and ran operationally for over 6 years covering 1 million patients.

In analyzing each cardiac modality, we developed a thorough approach building new machine learning models for registration, shape segmentation, analysis and detection of anomalies in modalities as well as for content-based image and multimodal data search techniques to find clinically similar patients. Among the anomalies addressed were valvular diseases (CW Doppler, Echo, Auscultation), cardiac aneurysms, dilated cardiomyopathy (Echo), and hypertrophic cardiomyopathy. Over 146 measurements from cardiac echo were automated including ejection fraction estimation.

The AALIM project produced a stream of early papers on finding similarity in cardiovascular modalities such as heart sounds, electrocardiograms, echocardiograms, coronary angiograms and cardiac MRI data which were published in conferences such as MICCAI, CVPR, ISBI, etc. An awareness of this field of research was also introduced through successive workshops at MICCAI on multimodal learning for clinical decision support (ML-CDS) running for the 14th year in a row at MICCAI 2025. Patient similarity as a paradigm was adopted by at least 50 other researchers world-wide in their publications and startups such as patientslikeme.com.

To deploy AALIM, we also developed scalable data integration technology based on HL7 messages to assemble longitudinal patient records for this purpose. The cardiologists in the Cath Lab at Kaiser used our system to get quick summary overviews of their patient’s data and recommendations for diagnosis, treatment and outcome, while the clinicians at Cedars Sinai used the system to recruit similar patients for clinical trials. The AALIM project illustrated how novel medical AI and data science methods could be developed, and clinically translated by automatic integration into IT systems of hospitals to ensure relevance and impact to healthcare.

A few representative publications across cardiac modalities addressed:

Echocardiography

• K. C. L. Wong, E. S. Sinkovskaya, M.D., A. Z. Abuhamad, M.D., T. Syeda-Mahmood, “Multiview and Multiclass Image Segmentation using Deep Learning in Fetal Echocardiography,” in Proc. SPIE Medical Imaging 2021.
• A. Lu, E. Dehghan, M. Moradi, T. Syeda-Mahmood, “Detecting Anomalies from Echocardiography using Multi-View Regression of Clinical Measurements”, IEEE ISBI 2018.
• G. Veni, M. Moradi, H. Bulu, G. Narayan, T. Syeda-Mahmood, “Echocardiagraphy segmentation based on a shape-guided deformable model driven by a fully convolutional network prior”, IEEE ISBI 2018.
• T. Baldwin, Y. Guo, V. Mukherjee, T. Syeda-Mahmood, “Generalized Extraction and Classification of Span-Level Clinical Phrases,” in Proc. AMIA Annual Symposium, November 2018.
• A. Lu, E. Dehghan, M. Moradi, T. Syeda-Mahmood, “Detecting Anomalies from Echocardiography using Multi-View Regression of Clinical Measurements”, IEEE ISBI 2018.
• T. F. Syeda-Mahmood, Y. Guo, M. Moradi, D. Beymer, D. Rajan, Yu Cao, Y. Gur, M. Negahdar, “Identifying Patients at Risk for Aortic Stenosis Through Learning from Multimodal Data,” in Proc. MICCAI (3) 2016: 238-245
• T. Syeda-Mahmood, Q. Wang, P. McNeillie, C. Compas, D. Beymer, “Discriminating normal and abnormal left ventricular shapes in four-chamber view 2D echocardiography,” in Proc. IEEE International Symposium on Biomedical Imaging, ISBI 2014.
• R. Mahmood, Q. Wang, T. Syeda-Mahmood, “Automatic detection of dilated cardiomyopathy in cardiac ultrasound videos,” in Proc. AMIA 2014.
• Chen Ting, R. Kumar, G. A. Troianowski, T. Syeda-Mahmood, D. Beymer, and K. Brannon,”PSAR: Predictive Space Aggregated Regression and Its Application in Valvular Heart Disease Classification”, in Proc. IEEE International Symposium on Biomedical Imaging, (ISBI) pp. 1122--1125, 2013.
• R. Kumar, T. Syeda-Mahmood, D. Beymer, C. Compas, K. Brannon, "Mining Echocardiography Workflows for Disease Discriminative Patterns," AMIA 2013- Annual Symposium of the American Medical Informatics Association, Washington, DC, Nov 16 - 20, 2013.
• T. F. Syeda-Mahmood, P. Turaga, D. Beymer, F. Wang, A. Amir, H. Greenspan, K. Pohl, “Shape-similarity-based retrieval of Doppler images for clinical decision support,” accepted for IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2010.
• R. Kumar, F. Wang, D. Beymer and T. F. Syeda-Mahmood, "Echocardiogram View Classification using Edge Filtered Scale-invariant Motion Features," in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), R. Mahmood, Q. Wang, T. Syeda-Mahmood, “Automatic detection of dilated cardiomyopathy in cardiac ultrasound videos,” in Proc. AMIA 2014 June 2009.
• D. Beymer, T. Syeda-Mahmood, A. Amir, F. Wang, and Scott Adelman, "Automatic Estimation of Left Ventricular Dysfunction from Echocardiogram Videos," in Proc. IEEE Computer Society Workshop on Mathematical Methods in Biomedical Image Analysis (MMBIA), June 2009
• T. Syeda-Mahmood, F. Wang, D. Beymer, M. London, and R. Reddy, "Characterizing spatio-temporal patterns for disease discrimination in cardiac echo videos", in Proc. International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI2007).

Cardiac CT

• M.Negahdar, D. Beymer, T. Syeda-Mahmood, “Automated volumetric lung segmentation of thoracic CT images using fully convolutional neural network”, SPIE 2018.
• H. Tang, M. Moradi, P. Prasanna, H. Wang, T. Syeda-Mahmood, “An algorithm for fully automatic detection of calcium in chest CT imaging,” ISBI 2017: 265-269, 2017.
• E. Dehghan, H. Wang, T. Syeda-Mahmood, “Automatic detection of aortic dissection in contrast-enhanced CT,” ISBI 2017: 557-560, 2017

Cardiac MRI

• A. Harouni, A. Karargyris, M. Negahdar, D. Beymer, T. Syeda-Mahmood, "Universal multimodal deep network for classification and segmentation of medical images", IEEE ISBI 2018.
• L. Xie, S. Sedai, X. Liang, C.B. Compas, H. Wang, P.A. Yushkevich, T. Syeda-Mahmood. “Multi-atlas Label Fusion with Augmented Atlases for Fast and Accurate Segmentation of Cardiac MR Images,” International Symposium on Biomedical Imaging, 2015.

Coronary Angiography

• C. Compas, T. Syeda-Mahmood, P. McNeillie, “Automatic detection of coronary stenosis in X-ray angiography through spatio-temporal tracking,” in ISBI 2014.
• T. Syeda-Mahmood, R. Kumar, D. Beymer, Y. Zhang, F. Wang, R. Lundstrom, E. J McNulty, J. Terdiman, "Automatic Extraction of Coronary Artery Tree in 2D X-ray Angiography", MMBIA 2012.
• D. Beymer, T. Syeda-Mahmood, F. Wang, R. Kumar, Y. Zhang, “AngioViewer: A Tool for Assessing the State of Coronary Artery Disease,” in Proc. AMIA Annual Symposium on Medical Informatics (AMIA), 2012

EKG, Heart Sounds

• T. Syeda-Mahmood, F. Wang, R. Reddy, "Shape-based Retrieval of Heart Sounds for Disease Similarity Detection", in Proc. European Conference on Computer Vision (ECCV2008), 2008.
• T. Syeda-Mahmood, D. Beymer, and F. Wang, "Shape-based Matching of ECG Recordings", in Proc. International IEEE Conference on Engineering in Medicine and Biology (EMBC2007), Lyon, France, August 23-26, 2007.

AALIM system

• T. Syeda-Mahmood, F. Wang, D. Beymer, A. Amir, M. Richmond, S. N Hashmi, “AALIM: Multimodal Mining for Cardiac Decision Support”, in Proc. Computers in Cardiology, Durham, North Carolina, 2007.

Back

Chest X-ray radiology

While AALIM was a purely data-driven approach to decision support, the next project explored the role of combining clinical knowledge and reasoning methods with data-driven approaches to offer decision support to radiologists at the point of care. Following the success of AALIM and Watson Jeopardy system, another large exploratory project called Medical Sieve Radiology Grand Challenge was started to prove that multimodal AI cognitive assistants could be trained to interpret and understand diagnostic imaging at the level of entry-level radiologists to aid in their decision making. This ambitious large-scale funded project developed sophisticated deep learning-driven medical AI reasoning models that interpreted data in breast and cardiac radiology at a level sufficient to pass the American Board of Radiology (ABR) exam. For this a large team of over 100 researchers from world-wide IBM Research labs were assembled along with over 40 clinicians in these specialties, software engineers, statisticians, and regulatory consultants. Several algorithms to detect and recognize many major breast and cardiac diseases in most common modalities including mammograms, chest X-rays, ultrasound, CT and MRI were developed. The corresponding clinical knowledge to drive the multimodal diagnostic reasoning and question answering for the American Board of Radiology (ABR) exam was then assembled. All this work was published in top conferences (MICCAI) and journals (over 100 publications, 5 best paper awards including MICCAI Young Scientist award in 2016). Much of this work also transitioned to define the core 510(k) products of Watson Health Imaging business such as Patient Synopsis, Clinical Review and their Cognitive Imaging Advisors.

While question answering in exam taking scenarios was a quantitative way to showcase automated image interpretation by AI models, a more commercially viable application was to produce an automated report describing the findings and impressions. This led us to the Chest X-ray Challenge in which we developed the first generative AI system for reading chest X-rays at the level of entry-level radiologists. We again assembled a team of clinicians, medical imaging specialists and data scientists to take a systematic approach to this problem by solving 4 key problems, namely, (a) development of a complete catalog of findings in chest X-rays, (b) assembly of a labeled chest X-ray datasets depicting the findings, and (c) development of a fine-grained deep learning model to classify the findings, and (d) generation of high quality realistic radiology reports from the detected findings and their implied impressions. Research from this work produced over 100 publications, and received the Homer Warner Award for outstanding contributions to biomedical informatics by the American Medical Informatics Association in 2020.

Currently, the focus in this work is on fact-checking of radiology reports through development of novel fact-checking models.

Representative publications in chest X-ray radiology:

Radiology report generation and verification

• R. Mahmood, Diego Machado-Reyes, Joy Wu, Parisa Kaviani, Ken C.L. Wong, Niharika D'Souza, Mannudeep Kalra, Ge Wang, Pingkun Yan, Tanveer Syeda-Mahmood, “Phrase-grounded Fact-checking for Automatically Generated Chest X-ray Reports,” in Proc. Medical Imaging and Computer-Assisted Intervention (MICCAI), 2025.
• T. Syeda-Mahmood, and L. Shi. "Searching for Fine-Grained Queries in Radiology Reports Using Similarity-Preserving Contrastive Embedding." In Jl. Of Machine Learning Research, vol.182:1–15, 2022. (IF=4.09)
• L. Shi, T. Syeda-Mahmood, T. Baldwin, “Improving Neural Models for Radiology Report Retrieval with Lexicon-based Automated Annotation,” in Proc. 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), 2022.
• T. Syeda-Mahmood, K. C. L. Wong, Y. Gur, J. T. Wu, Ashutosh Jadhav, A. Karargyris, A. Pillai, A. Sharma, A. Syed, O. Boyko, M. Moradi, "Chest X-ray Report Generation through fine-grained label learning," in Proc. Medical Imaging and Computer-Assisted Interaction, pp.561-571, (MICCAI 2020).

Finding classification models

• N. Srivathsa, R. Mahmood, T. Syeda-Mahmood, “Spatially preserving flattening for location-aware classification of findings in chest X-rays,” in Proc. ISBI 2022
• K. C. L. Wong, M. Moradi, J. Wu, A. Pillai, A. Sharma, Y. Gur, H. Ahmad, M. S. l Chowdary, J. Chiranjeevi, K. K. Reddy P., W. Venkateswar, D. C Reddy, T. Syeda-Mahmood, "A robust network architecture to detect normal chest X-ray radiographs," in Proc. (ISBI) 2020.
• C. Agunwa, Mehdi Moradi, Ken C.L. Wong, T. Syeda-Mahmood, “Body Part and Imaging Modality Classification for a General Radiology Cognitive Assistant”, SPIE Medical Imaging 2019.
• M. Moradi, K. C. L. Wong, A. Karargyris, and T. Syeda-Mahmood, "Quality controlled segmentation to aid disease detection", in SPIE Medical Imaging 2020.
• A. Karargyris, K. C. L. Wong, Joy T Wu, M. Moradi, T. Syeda-Mahmood, "Boosting the rule-out accuracy of deep disease detection using class weight modifiers", IEEE ISBI 2019.
• A. Harouni, H. Wang, T. Syeda-Mahmood, David Beymer, "Deep Network Anatomy Segmentation with Limited Annotations Using Auxiliary Labels", IEEE ISBI 2019.
• S. Kashyap, M. Moradi, A. Karargyris, Joy T Wu, M. Morris, B. Saboury, E. Siegel, T. Syeda-Mahmood, “Artificial Intelligence for Point of Care Radiograph Quality Assessment”, SPIE Medical Imaging 2019.
• C. Agunwa, Mehdi Moradi, Ken C.L. Wong, T. Syeda-Mahmood, “Body Part and Imaging Modality Classification for a General Radiology Cognitive Assistant”, SPIE Medical Imaging 2019.
• K. C. L Wong, M. Moradi, J. Wu, T. Syeda-Mahmood, “Identifying disease-free chest x-ray images with deep transfer learning”, SPIE Medical Imaging 2019: Computer-Aided Diagnosis.
• A Karargyris, S Kashyap, JT Wu, A Sharma, M. Moradi, T Syeda-Mahmood, “Age prediction using a large chest x-ray dataset”, SPIE Medical Imaging 2019: Computer-Aided Diagnosis.
• K. C. L Wong, M. Moradi, J. Wu, T. Syeda-Mahmood, “Identifying disease-free chest x-ray images with deep transfer learning”, SPIE Medical Imaging 2019: Computer-Aided Diagnosis.
• M. Moradi, A. Madani, Y. Guy, Y. Guo, T. Syeda-Mahmood, "Bimodal network architectures for automatic generation of image annotation from text", MICCAI 2018, pp. 449-456.
• A. Madani, M. Moradi, A. Karargyris, T. Syeda-Mahmood, “Semi-supervised learning with generative adversarial networks for chest X-ray classification with ability of data domain adaptation”, IEEE ISBI 2018.
• Ali Madani, M. Moradi, T. Syeda-Mahmood, A. Karargyris, “Chest x-ray generation and data augmentation for cardiovascular abnormality classification”, SPIE 2018.
• T. F. Syeda-Mahmood, R. Kumar, C.B. Compas, “Learning the correlation between images and disease labels from reports using ambiguous learning,” Medical Image Computing and Computer-Assisted Intervention, accepted for MICCAI 2015.

Clinical knowledge/ text analytics

• A. Jadhav, T. Baldwin, J. Wu, V. Mukherjee, T. Syeda-Mahmood, “Semantic Expansion of Clinician Generated Data Preferences for Automatic Patient Data Summarization”, AMIA 2021.
• Tanveer Syeda-Mahmood, Ken C. L. Wong, Joy T. Wu, Ashutosh Jadhav, Orest Boyko, Extracting and Learning Fine-grained Labels from Chest Radiographs, AMIA Annual Symposium, (AMIA) 2020. Won the Homer Warner Award for outstanding contribution to the field of biomedical informatics.
• A. Jadhav, K. C. L. Wong, J. Wu, M. Moradi, T. Syeda-Mahmood. "Combining Deep Learning and Knowledge-driven Reasoning for Chest X-Ray Findings Detection" in Proc. American Medical Informatics Association Annual Symposium 2020 (AMIA 2020).
• J. T. Wu, A. Syed, H. Ahmad, A. Pillai, Y. Gur, A. Jadhav, D. Gruhl, L. Kato, M. Moradi, T. Syeda-Mahmood, “AI Accelerated Human-in-the-loop Structuring of Radiology Reports,” AMIA Annual Symposium 2020.
• T. Syeda-Mahmood, H. M Ahmad, N. Ansari, Yaniv Gur, Satyananda Kashyap, Alexandros Karargyris, Mehdi Moradi, Anup Pillai, Karthik Sheshadri, Weiting Wang, Ken CL Wong, Joy T Wu, "Building a Benchmark Dataset and Classifiers for Sentence-Level Findings in AP Chest X-rays", IEEE ISBI 2019
• A. Jadhav, J. Wu, T. Syeda-Mahmood. "Automatic extraction of structured radiology reports". Society for Imaging Informatics in Medicine (SIIM) 2019, Denver, Colorado, USA. (Oral presentation)
• A. Jadhav, J. Wu, T. Syeda-Mahmood. "Knowledge-driven approach to boost the performance of solely image-based deep learning models" Society for Imaging Informatics in Medicine (SIIM) 2019, Denver, Colorado, USA.
• T. Syeda-Mahmood, J. Wu, M. Morris, B. Sabury, "Automatic structuring of radiology reports, “RSNA 2019.
• A. Jadhav, J. Wu, T. Syeda-Mahmood. "Knowledge-driven approach to boost the performance of solely image-based deep learning models" Society for Imaging Informatics in Medicine (SIIM) 2019, Denver, Colorado, USA.
• A. Jadhav, T. Baldwin, J. Wu, V. Mukherjee, T. Syeda-Mahmood, "Automatic Patient Data Summarization for Radiologist", RSNA 2019.

Regional finding detection

• J. T. Wu, Y. Gur, A. Karargyris, A.B. Syed, O. Boyko, M. Moradi, T. Syeda-Mahmood, "Automatic Bounding Box Annotation of Chest X-Ray Data for Localization of Abnormalities," in Proc. ISBI 2020
• S. Kashyap, A. Karargyris, Joy T. Wu, Y. Gur, A. Sharma, K. C.L. Wong, M. Moradi, T. Syeda-Mahmood, "Looking in the Right Place for Anomalies: Explainable AI Through Automatic Location Learning," in Proc. ISBI 2020.
• M. Moradi, K. C. L. Wong, A. Karargyris, and T. Syeda-Mahmood, "Quality controlled segmentation to aid disease detection", To appear in SPIE Medical Imaging 2020.
• A. Harouni, H. Wang, T. Syeda-Mahmood, and D. Beymer. "Deep Network Anatomy Segmentation with Limited Annotations using Auxiliary Labels." In 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), pp. 6-10. IEEE, 2019.

Lung cancer

• V. Subramanian, M. Do, T. Syeda-Mahmood, "Multimodal Fusion of Imaging and Genomics for Lung Cancer Recurrence Prediction," in Proc. ISBI 2020
• J. Francis, H. Wang, K. White, T. Syeda-Mahmood, R. Stevens, "Neural Network Segmentation of Cell Ultrastructure Using Incomplete Annotation," in Proc. ISBI 2020.

Tubes and Lines detection

• V. Subramanian, H. Wang, Joy Wu, K. CL Wong, A. Sharma, T. Syeda-Mahmood, "Automated detection and type classification of central venous catheters in chest X-rays", MICCAI 2019.

Fracture detection

Y. Cao, H. Wang, M. Moradi, P. Prasanna and T. Syeda-Mahmood. “Fracture Detection in X-ray Images through Stacked Random Forests Feature Fusion,” in Proc. International Symposium on Biomedical Imaging, ISBI 2015

Back

Healthcare informatics

With the large-scale multimodal data collected during the AALIM project, there was a rich opportunity to do comparative effectiveness research to ask questions regarding effectiveness of treatment, discovering a large number of co-morbidities between diseases or between drugs at the population level. Similarly, demographic distribution of diseases, and their outcomes could be measured. Predictive analytic models could be applied for varied problems ranging from hospital re-admissions, to likelihood of restenosis. Looking for clinical evidence across multimodal sources could reveals more patients that could benefit from treatments or interventions. Billing errors involving over or under billing could be discovered impacting revenue cycle management. Much of this work was done in collaboration with device vendors (Edwards Life Sciences) and hospital systems such as Kaiser Permanente and Cedars-Sinai.

• T. F. Syeda-Mahmood, Y. Guo, M. Moradi, D. Beymer, D. Rajan, Yu Cao, Y. Gur, M. Negahdar, “Identifying Patients at Risk for Aortic Stenosis Through Learning from Multimodal Data,” in Proc. MICCAI (3) 2016: 238-245

Back

Clinical decision support

We pioneered the concept of patient similarity-guided clinical decision support in the AALIM system. The AALIM system extracted information from different data sources from EMR, PACS, Laboratory systems, and reporting systems to form a longitudinal patient record. It used advanced analytics to extract diagnostic information from multimodal patient data and was the first system to find similar cases, diseases, comorbidities and medications using data-driven machine learning. AALIM brought the longitudinal and cross-sectional information of that population to the point of care. It provided clinicians with a consistent view of the patient's history as well as the ability to compare this history with that of other patients with similar test results and disease profiles. This comparison supplied the clinician with a refined insight as to the patient's diagnosis and the comparative effectiveness of different treatments on outcomes.

After AALIM, the next clinical decision support system combined data-driven reasoning with clinical knowledge. While current generative AI models are still trying to formulate diagnostic reasoning, the Medical Sieve project was an avant-garde clinical informatics system that collected clinical, textual and imaging data of patients from electronic health records systems. It then analyzed multimodal content to detect anomalies if any, and summarized the patient record collecting all relevant information pertinent to a chief complaint. The results of anomaly detection were then fed into a reasoning engine which used evidence from both patient-independent clinical knowledge and large-scale patient-driven similar patient statistics to arrive at potential differential diagnosis to help in clinical decision making. In compactly summarizing all relevant information to the clinician per chief complaint, the system retained provenance to the raw data for detailed review providing holistic summaries of patient conditions. The system was demonstrated in the domains of cardiology and breast radiology for differential diagnosis and imaging studies summarization. A demonstration called the Eyes of Watson was developed jointly with RSNA in 2016 in which 3000 radiologists participated.There were 260 million earned impressions, and 919,200 social impressions on Twitter and 10,934 views of this Eyes of Watson Video .

• J. Wu, K. C. L. Wong, Y. Gur, N. Ansari, A. Karargyris, A.Sharma, M. Morris, B. Sabury, H. Ahmad, O. Boyko, A. Syed, A. Jadhav, H. Wang, A. Pillai, S. Kashyap, M. Moradi, T. Syeda-Mahmood, "Artificial Intelligence versus Entry-level Radiologists for Full-fledged Preliminary Read of Frontal AP Chest Radiographs: A Comparative Study", Journal of American Medical Informatics Association, (JAMA) Network, October 9, 2020. (IF=13.35)
• Karina Kanjaria, Anup Pillai, Chaitanya Shivade, Marina Bendersky, Vandana Mukherjee and Tanveer Syeda-Mahmood, “Receptivity of an AI Cognitive Assistant by the Radiology Community: A Report on Data Collected at RSNA,” Best Industrial Paper Award at International Conference on Healthcare Informatics (part of BIOSTEC 2020).
• T. Syeda-Mahmood, “Role of Big Data and Machine Learning in Diagnostic Decision Support in Radiology”, Journal of the American College of Radiology, 15(3), 569-576, 2018. (IF=5.3)
• A. Pillai, A. Katouzian, A. Jadhav, M. Bendersky. K. Kanjaria, C. Shivade, V. Mukherjee, T. Syeda-Mahmood, “A knowledge-based question answering system to provide cognitive assistance to radiologists”, SPIE Medical Imaging 2019.
• A. Jadhav, T. Baldwin, J. Wu, V. Mukherjee, T. Syeda-Mahmood, "Automatic Patient Data Summarization for Radiologist", RSNA 2019.
• T. F. Syeda-Mahmood, E. Walach, D. Beymer, F. Gilboa-Solomon, M. Moradi, P. Kisilev, D. Kakrania, C. B. Compas, H. Wang, M. Negahdar, Y. Cao, T. Baldwin, Y. Guo, Y. Gur, D. Rajan, A. Zlotnick, S. Rabinovici-Cohen, R. Ben-Ari, G. Amit, P. Prasanna, J. Morey, O. B. Boyko, S. Y. Hashoul, “Medical sieve: a cognitive assistant for radiologists and cardiologists,” SPIE Medical Imaging: Computer-Aided Diagnosis 2016.
• T. Syeda-Mahmood, F. Wang, D. Beymer, A. Amir, M. Richmond, S. N Hashmi, “AALIM: Multimodal Mining for Cardiac Decision Support”, in Proc. Computers in Cardiology, Durham, North Carolina, 2007.

Back

Brain Imaging

In brain imaging, we covered diseases ranging from hematomas to Alzheimer's but our strongest work here was on building new mathematical formulations to address the segmentation and registration problems. We consistently led the leaderboards of BRATS segmentation challenge until 2017 and developed novel segmentation approaches through the eras of statistical ML, deep learning, and Fourier neural operators. Most of these publications appear in MICCAI, IPMI and their associated journals.

Representative publications in this area:

• Ken C. L. Wong, Hongzhi Wang, Tanveer Syeda-Mahmood, “HartleyMHA: Self-Attention in Frequency Domain for Resolution-Robust and Parameter-Efficient 3D Image Segmentation”, in International Conference of Medical Image Computing and Computer-Assisted Intervention, MICCAI, 2023.
• Ken C. L. Wong, Hongzhi Wang, and Tanveer Syeda-Mahmood, “FNOSeg3D: resolution-robust 3D image segmentation with Fourier neural operator,” in IEEE International Symposium on Biomedical Imaging, ISBI 2023.
• A. Karargyris and T. Syeda-Mahmood, “Saliency U-Net: A regional saliency map-driven hybrid deep learning network for anomaly segmentation,” Proc. SPIE, vol.10575, February 2018.
• H Wang, D Kakrania, H Tang, P Prasanna, T Syeda-Mahmood, “Fast Anatomy Segmentation by Combining Coarse Scale Multi-Atlas Label Fusion with Fine Scale Corrective Learning”, Computerized Medical Imaging and Graphics 68, 16-24, 2018.
• Ken C. L. Wong, M. Moradi, H. Tang, T. Syeda-Mahmood, "3D Segmentation with Exponential Logarithmic Loss for Highly Unbalanced Object Sizes", MICCAI 2018, pp. 612-619.
• K. C. L. Wong, T. Syeda-Mahmood, M. Moradi, “Building medical image classifiers with very unbalanced and limited data using segmentation networks”, Medical Image Analysis, vol. 49, pp. 105–116, 2018.
• H. Wang, M. Moradi, Y. Gur, P. Prasanna and T. F. Syeda-Mahmood, “A Multi-Atlas Approach to Region of Interest Detection for Medical Image Classification.” MICCAI 2017, pp. 168-176, 2017.
• K. C. L. Wong, M. Moradi, A. Karargyris, T. Syeda-Mahmood, Building Disease Detection Algorithms with Very Small Number of Positive Samples, in Proc. MICCAI 2017
• H. Wong, M. Moradi, Y. Gur, P. Prasanna, T. Syeda-Mahmood,” A multi-atlas approach to region of interest detection for medical image classification,” in Proc. MICCAI 2017
• R. Kumar, B. C. Vemuri, F. Wang, T. F. Syeda-Mahmood, P. R. Carney, T. H. Mareci, "Multi-fiber Reconstruction from DW-MRI data using a Continuous Mixture of Hyperspherical von Mises-Fisher Distributions," in Proc. Information Processing in Medical Imaging (IPMI) 2009.
• F. Wang, T. F. Syeda-Mahmood, B. C. Vemuri, D. Beymer, and A. Rangarajan, "Closed-form Jensen-Renyi Divergence for Mixture of Gaussians & Applications to Group-wise Shape Registration," in Proc. International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2009.
• F. Wang, B. C. Vemuri, and T. F. Syeda-Mahmood, "Generalized L2-divergence and Its Application to Shape Alignment," in Proc. Information Processing in Medical Imaging (IPMI), July 2009.

Back

Multimodal fusion for cancers and other diseases

In a complex disease such as cancer, the interactions between the tumor and host can exist at the molecular, cellular, tissue, and organism levels. Thus, evidence for the disease and its evolution may be present in multiple modalities across scale such as clinical, genomic, molecular, pathological and radiological imaging. Effective patient-tailored therapeutic guidance and planning using digital twins in the future will require bridging spatiotemporal scales through novel multimodal fusion formalisms. In the last few years, funded by an NCI-DOE grant on digital twins, we have expanded into multimodal AI for better prediction and modeling by integrating clinical patient data with imaging, genomic and lab information to address cancer and other diseases. We took a principled approach to multimodal fusion by modeling fine-grained and multi-faceted complex interactions, both within and across modalities and scales of observation for better outcome prediction. Specifically, my team and I developed a general theory of multiplexed graph neural networks expanding the typical inference propagation through graph convolutions to cover both within plane and cross-planar connections. The resulting fusion framework was demonstrated to fuse modalities in a superior way and applied to treatment outcome prediction in tuberculosis patients by combining clinical data with genomic and imaging data. This paper was a Young Scientist (Best Paper) finalist at MICCAI 2022 and also appeared in the Medical Image Analysis journal.

I am also interested in exploring the role of fusion for disease modeling through association discovery to obtain a fundamental understanding of the correlation between imaging and genomics data. Specifically, we explored the correlation between cardiac tissue morphology and genomic loci by integrating GWAS data with cardiac MRI data using convolutional mesh encoders. Using the UK Biobank data we identified 49 gene loci that had study-wide significance with left ventricular morphology seen in cardiac MRI. This landmark study was done jointly with the University of Leeds researchers and appeared on the cover of the March issue of Nature Machine Intelligence journal this year.

Representative publications in digital twins for cancer care and beyond:

• N.S. D’Souza, H. Wang, A. Giovannini, A. Foncubierta-Rodriguez, K. L. Beck, O. Boyko, T. Syeda-Mahmood, "Fusing Modalities by Multiplexed Graph Neural Networks for Outcome Prediction from Medical Data and Beyond Medical Image Analysis", Medical Image Analysis Journal, April 2024. IF=10.9
• R. Bonazzola, E. Ferrante, N. Ravikumar, Y. Xia, B. Keavney, S. Plein, T. Syeda-Mahmood, and A. F Frangi, “Unsupervised ensemble-based phenotyping enhances gene discoverability in imaging genetics: new associations from left-ventricular morphology,” in Nature Machine Intelligence, 6(3):291-306, March 2024, IF=23.8.
• E. Warner, J. Lee, W. Hsu, T. Syeda-Mahmood, C. E. Kahn, O. Gevaert, A. Rao, “Multimodal Machine Learning in Image-Based and Clinical Biomedicine: Survey and Prospects,” in International Journal of Computer Vision (IJCV), April 2024, IF=19.5
• E. Katsoulakis, Qi Wang, H. Wu, L. Shahriyari, R. Fletcher, J. Liu, L. Achenie, H. Liu, P. Jackson, Y. Xiao, T. Syeda-Mahmood, R. Tuli, J. Deng, "Digital twins for health: a scoping review,", NPJ Digital Medicine, vol.7, 77, 2024. IF=15.35
• V. Subramanian, T. Syeda-Mahmood, M.N. Do, "Modelling-based joint embedding of histology and genomics using canonical correlation analysis for breast cancer survival prediction," in Artificial Intelligence in Medicine, Mar:149:102787, 2024, IF=14.0
• Singh SB, Wang H, Baratto L, Wu JT, Vasyliv I, Adams L, Sarrami AH, Syeda-Mahmood T, Daldrup‐Link H.E. Deep Learning Algorithm for Automatic Pediatric Lymphoma Detection Using Multimodal FDG PET/MRI Images. Society for Pediatric Radiology (SPR) 2024, Supplement 2024.
• Hongzhi Wang, Amirhossein Sarrami, Joy Tzung-yu Wu, Lucia Baratto, Arjun Sharma, Ken C. L. Wong, Shashi Bhushan Singh, Heike E Daldrup-Link, Tanveer Syeda-Mahmood, “Multimodal Pediatric Lymphoma Detection using PET and MRI”, in AMIA 2023 Annual Symposium, 2023.
• Stahlberg, E.A., Abdel-Rahman, M., Aguilar, B., Asadpoure, A., Beckman, R.A., Borkon, L.L., Bryan, J.N., Cebulla, C.M., Chang, Y.H., Chatterjee, A. and Deng, J., 2022. Exploring approaches for predictive cancer patient digital twins: Opportunities for collaboration and innovation. Frontiers in Digital Health, 4, p.1007784. 2022, (IF=3.2)
• N. D'Souza, H. Wang, A. Giovannini, A. Foncubierta-Rodríguez, K. Beck, O. Boyko, T. Syeda-Mahmood, “Fusing Modalities by Multiplexed Graph Neural Networks for Outcome Prediction in Tuberculosis,” in Proc. MICCAI 2022. MICCAI Young Scientist Award Finalist.
• R. Bonazzola, N. Ravikumar, R. Attar, E. Ferrante, T. Syeda-Mahmood, A. F. Frangi, “Image-derived phenotype extraction for genetic discovery via unsupervised deep learning in CMR images,” in Proc. MICCAI 2021.
• V. Subramanian, T. Syeda-Mahmood, Minh N. Do, “Multimodal fusion using sparse CCA for breast cancer survival prediction, IEEE ISBI 2021.
• H. Wang. V. Subramanian, T. Syeda-Mahmood, Modeling uncertainty in multimodal fusion for lung cancer survival analysis, IEEE ISBI 2021.
• Hernandez-Boussard, Tina, Paul Macklin, Emily J. Greenspan, Amy L. Gryshuk, Eric Stahlberg, Tanveer Syeda-Mahmood, and Ilya Shmulevich. "Digital twins for predictive oncology will be a paradigm shift for precision cancer care", Nature Medicine, vol. 27, no. 12 (2021): 2065-2066. (IF=87.8)
• V. Subramanian, M. Do, T. Syeda-Mahmood, "Multimodal Fusion of Imaging and Genomics for Lung Cancer Recurrence Prediction," in Proc. ISBI 2020
• J. Francis, H. Wang, K. White, T. Syeda-Mahmood, R. Stevens, "Neural Network Segmentation of Cell Ultrastructure Using Incomplete Annotation," in Proc. ISBI 2020.
• E. Greenspan, C. Lauzon, A. Gryshuk, J. Ozik, N Collier, T. Syeda-Mahmood, I. Shmulevich, T. Hernandez-Boussard, P. Macklin, “Digital Twins for Predictive Cancer Care: an HPC-Enabled Community Initiative,” in Proc. The International Conference for High Performance Computing, Networking, Storage, and Analysis, Nov. 18-21, 2019.

Back

Precision AI in Interventional imaging

Despite the increased adoption of multimodal AI in healthcare, the impact to patient outcomes is yet to be fully seen. There is room for improvement both in the choice of problems and domains where AI can make a difference, as well as the influence of other inter-disciplinary fields such as neuroscience on the formation of novel AI architectures to address problems in healthcare and beyond. While much of AI in imaging work has focused on diagnostic radiology, there is an even greater need for AI assistance in many interventional radiology and cardiology procedures for interpreting pre-procedural and intra-procedural images and sequences. These are challenging areas for AI as high levels of precision are needed in decision making involving integration of multiple sources of information including operative and pre-operative images, knowledge of interventional devices and their dimensional constraints, modeling of the dexterity of the interventional specialists, and real-time feedback for action guidance. Our work in precision interventional AI was developed under a research collaboration between Boston Scientific and MIT. Specifically, we developed methods to detect stents and their malappositions in coronary arteries. We also developed novel neural networks for measuring continuous lumen boundaries in DVT ultrasound to select the correct size of interventional stents. Our stent detection method has been subsequently commercialized by Boston Scientific and introduced in clinics through their AVVIGO multi-modality guidance system.

As this is an emerging area of interest, our publications are more recent:

• Y. Chen, N. D’Souza, A. Mandepally, P. Henninger, S. Kashyap, N. Karani, N, Dey, M. Zachary, R. Rizq, P. Chouinard, P. Golland, T. Syeda-Mahmood, “Geo-UNet: A Geometrically Constrained Neural Framework for Clinical-Grade Lumen Segmentation in Intravascular Ultrasound,” in Proc. Machine Learning for Medical Imaging (MLMI) Workshop, MICCAI 2024.
• S. Kashyap, N. Karani, A. Shang, N, D’Souza, N. Dey, L. Jain, R. Wang, H. Akakin, D. Li, W. Li, C. Carlson, P. Golland, T. Syeda-Mahmood, “Feature selection for malapposition detection in intravascular ultrasound – A comparative study,” in Proc. Applications of Artificial Intelligence Workshop, MICCAI 2023, Vancouver, BC, pp. 165-178. 2023.

Back