Holistic Evaluation of GPT-4V for Biomedical Imaging

6 major task evaluation experiments in GPT-4V with 8 modalities in 16 medical fields. Reference alone 245 page arXiv papers. I think there were similar papers in the past, but I think it’s a follow-up

제목: Holistic Evaluation of GPT-4V for Biomedical Imaging

Summary:
In this paper, we present a large-scale evaluation that examines the capabilities and limitations of GPT-4V for biomedical image analysis. GPT-4V is a breakthrough in artificial general intelligence (AGI) for computer vision and has been applied in the biomedical domain. We evaluate the performance of GPT-4V in 16 medical imaging categories, including radiology, oncology, ophthalmology, and pathology. It performs tasks such as modality recognition, anatomical localization, disease diagnosis, report generation, and lesion detection. Extensive experiments provide insights into the strengths and weaknesses of GPT-4V. Experiments show that GPT-4V is adept at modality and anatomical recognition but struggles with disease diagnosis and localization. GPT-4V has shown strength in image captioning techniques due to its excellence in generating diagnostic reports. While promising as biomedical imaging AI, GPT-4V requires further improvements and validation prior to clinical deployment. We emphasize responsible development and testing for reliable biomedical AGI integration. The rigorous evaluation of GPT-4V on a variety of medical images enhances understanding of multimodal large-scale language models (LLMs) and guides future work toward influential medical applications.

arXiv: https://arxiv.org/abs/2312.05256
Browse: https://browse.arxiv.org/pdf/2312.05256.pdf
PDF: https://arxiv.org/pdf/2312.05256.pdf
arXiv-vanity: https://www.arxiv-vanity.com/papers/2312.05256
Paper page: https://huggingface.co/papers/2312.05256
Papers with code: https://paperswithcode.com/paper/holistic-evaluation-of-gpt-4v-for-biomedical I think the robot also has a GPT moment. It’s a world where you attach GPT to a robot and express its movements in text. Sooner or later, there will be an era in which robots move by speaking and expressing, and then there will be robots that naturally communicate and act like humans.

제목: From Text to Motion: Grounding GPT-4 in a Humanoid Robot “Alter3”

Summary:
We report the development of Alter3, a humanoid robot that can generate spontaneous actions using LLM (large language models), specifically GPT-4. This achievement was realized by integrating GPT-4 into the proprietary Android Alter3, effectively grounding LLM to Alter’s body movements. In general, lower-level robot control depends on the hardware and is beyond the scope of the LLM corpus, thus presenting challenges for direct LLM-based robot control. However, for humanoid robots such as Alter3, direct control is possible by mapping linguistic representations of human actions to the robot’s body through program code. Surprisingly, this approach allows Alter3 to adopt different poses, such as a ‘selfie’ pose or ‘pretending to be a ghost’, and generate a sequence of actions over time without explicit programming for each body part. This demonstrates the robot’s ability to learn zero-shot. Furthermore, the pose can be adjusted through verbal feedback, so no fine-tuning is required.

arXiv: https://arxiv.org/abs/2312.06571
Browse: https://browse.arxiv.org/pdf/2312.06571.pdf
PDF: https://arxiv.org/pdf/2312.06571.pdf
arXiv-vanity: https://www.arxiv-vanity.com/papers/2312.06571
Paper page: https://huggingface.co/papers/2312.06571
GitHub: https://tnoinkwms.github.io/ALTER-LLM/

“Playing Metal Guitar” Video: https://www.youtube.com/watch?v=SAc-O5FDJ4k And obviously, as I expected, the wave of Generative AI and LLM also seemed to be huge in RSNA 2023.

“With the success of ChatGPT, generative AI (#GenAI) technology, especially large language models (LLMs), has dominated the discussion of artificial intelligence across the industry. The same is true of Radiology AI, so the impact of new GenAI technology has been evident in almost every encounter of RSNA.

Among other things, GenAI has significantly expanded its interest and expectations for AI within the radiology community. Previously, while the main drivers for new products and companies came from image analysis and processing tools, in 2023, discussions took place about future opportunities around LLM’s capabilities. These included AI generation results and expectations of workflow simplification, such as automatic report generation, which can greatly simplify evaluation by radiologists.

In conclusion, RSNA 2023 provided a comprehensive snapshot of the evolving environment of radiology AI, characterized by a mature industry, diverse delivery models, and the promise of LLM and GenAI.”

As it develops so rapidly, how to include Generative AI, LLM, and Foundation models in the areas of standards and regulation is likely to be proportional to the size of expectations and concerns.

Anyway, I think we need to hear more stories from those who have been back ….

Link: https://www.linkedin.com/pulse/reflections-future-radiology-ai-rsna-2023-wrapup-braunewell-x2chf/ In Germany, we published our findings in the European Journal of Radiology measuring how computer-aided detection systems for prostate MRI affect the workflow, workload and stress of radiologists, and the conclusion is highly controversial.

Conclusion: The implementation of AI-based detection aids had a low standardization level and had no effect on radiology specialists’ workload or stress over time. Expectations that AI would reduce radiology specialists’ workload have not been confirmed in real-world studies.

The CAD system in question was Quantib Prostate, an FDA-approved web-based deep learning MRI reading and reporting platform from Wenderott in the United States…

Sajok: But introducing and using MS Office products in a company, not all companies will just improve their workflow or reduce their workload or stress. Wouldn’t it be possible to think the other way around from that perspective?

논문: https://www.sciencedirect.com/science/article/pii/S0720048X23005661