More details
The app shows promise in the following areas:
- Research: The app can be used to benchmark the performance of AI models (in this case, GPT-4, though it can easily be edited to incorporate other fine-tuned LLMs in the future).
- Education: This app is a useful demonstration of the potential of AI (in particular LLM models) to revolutionize medicine.
- Clinical Support: Radiologists can use the app to double-check their findings, ensuring no critical details are overlooked.
While the app shows promise, it has the following limitations and is currently not intended for clinical use:
- Data Security/Privacy: OpenAI doesn’t use the data sent via the API to train/improve its models and the data is only retained for 30 days to monitor for abuse before being deleted, However, don't enter sensitive/identifiable information into the app or any third-person API. For more information on how OpenAI handles information sent via its API, see the “API Platform FAQ” section of https://openai.com/enterprise-privacy.
- Performance: The accuracy of the model, as reflected by the F1 scores, shows variability across different pathological findings. In some areas, the app performs well, while in others, it demonstrates that further refinement is necessary. See the paper Leveraging GPT-4 for Post Hoc Transformation of Free-text Radiology Reports into Structured Reporting: A Multilingual Feasibility Study for more information.
The core functionality of the app is based on the paper Leveraging GPT-4 for Post Hoc Transformation of Free-text Radiology Reports into Structured Reporting: A Multilingual Feasibility Study, published in April 2023 and cited by 51+ different papers so far. The corresponding Github repository was created by Keno Bressem, a board-certified radiologist; for more information about him, please see https://aim.hms.harvard.edu/team/keno-bressem.
Keno Bressem, a board-certified radiologist, created the app’s core functionality, which has been peer-reviewed. See https://aim.hms.harvard.edu/team/keno-bressem for more information about him.
Venkata Chengalvala created this app’s Steramlit interface. As an AI consultant for Health Universe, he ports high-quality peer-reviewed AI models helpful to clinicians, patients, and/or researchers to Health Universe's platform. He has a Bachelor of Science in Molecular, Cellular, and Developmental Biology (MCDB) and Computer Science from the University of Michigan-Ann Arbor. During his undergrad education, he engaged in medical research, co-authoring a literature review on tumor-derived exosomes that’s cited by 30+ people so far: https://www.sciencedirect.com/science/article/pii/S2211383521001398.
If you have any feedback or thoughts about the app, feel free to leave a comment below.
Warning: App may appear to work well but has not been peer reviewed. Not intended for clinical use. Use with caution.
- Diagnostics & Imaging