Gemini: A Family of Highly Capable Multimodal Models

A comprehensive research and analysis of research trends on Retrial-Augusted Generation (RAG), one of the keywords that attracts attention these days. It’s 27 pages long, but it’s a good summary of the difference between fine tuning and RAG, as well as the analysis of research trends on major research trends and major components and technologies. There are a lot of survey papers like this in China these days.

제목: Retrieval-Augmented Generation for Large Language Models: A Survey

Summary:
Large-scale language models (LLMs) show powerful features, but in practical applications they still face challenges such as hallucinations, slow knowledge updates, and lack of transparency in their answers. Search augmentation generation (RAG) refers to searching for relevant information from an external knowledge base before answering a question with LLM. RAG has been proven to significantly improve the accuracy of answers and reduce model hallucinations, especially in knowledge-intensive tasks. Users can verify the accuracy of their answers by citing sources and increase confidence in the model outcomes. It also facilitates knowledge updates and domain-specific knowledge introduction. RAG is one of the most important methods for implementing large-scale language models by effectively combining parameterized and unparameterized external knowledge bases of LLM. In this paper, we summarize the development paradigm of RAG in the LLM era into three categories: Naive RAG, Advanced RAG, and Modular RAG. We then summarize and summarize the three main components of RAG: retrievers, generators, augmentation methods, and key techniques of each component. It also describes how to evaluate the effectiveness of the RAG model, introduces two evaluation methods for RAG, highlights key indicators and functions for evaluation, and presents the latest automatic evaluation framework. Finally, we introduce future research directions in three aspects: vertical optimization, horizontal scalability, the technology stack of RAG, and the ecosystem. The arXiv paper from Google Gemini team. 950 co-authors are registered together. This is a unique record

Of the 62 pages, 10 pages are the author’s list, and the introduction of the method is only one page (three pages including the learning method). ^^ I think the purpose of including the family word in the title was to emphasize the family in which the authors themselves express ‘Are we different!!’ 🙂

제목: Gemini: A Family of Highly Capable Multimodal Models

Summary:
In this report, we introduce Gemini, a new family of multimodal models that outperforms across image, audio, video, and text understanding. The Gemini family consists of ultra, pro, and nano sizes and is suitable for applications ranging from complex inference tasks to on-device memory-constrained use cases. Our evaluation on a variety of benchmarks shows that the most performing Gemini Ultra model outperforms the state-of-the-art on 30 of 32 benchmarks and is the first to achieve human expert-level performance, especially on the well-studied test benchmark, MMLU, and improves the state-of-the-art on all 20 multimodal benchmarks. We believe that new features of the Gemini model in cross-modal inference and language understanding will enable a wide range of use cases, and we discuss approaches to responsibly deploy them to users.

arXiv: https://arxiv.org/abs/2312.11805
Browse: https://browse.arxiv.org/pdf/2312.11805.pdf
PDF: https://arxiv.org/pdf/2312.11805.pdf
arXiv-vanity: https://www.arxiv-vanity.com/papers/2312.11805
Paper page: https://huggingface.co/papers/2312.11805
Papers with code: https://paperswithcode.com/paper/gemini-a-family-of-highly-capable-multimodal-1
There seems to be a lot of interest in the Foundation Model and AGI in China, too. A 169-page research preprint on Foundation Model-based reasoning written by 32 authors from 12 schools. More than 70 pages here are also references.

Figure 5 presents a classification of Reason Tasks based on the Foundation model, and introduces each research trend by dividing it into 7 sub-categories and 24 sub-categories. All papers are organized and provided again on Github, so it will be helpful for those interested.

제목: A Survey of Reasoning with Foundation Models: Concepts, Methodologies, and Outlook

Summary:
As an important ability for solving complex problems, inference plays a pivotal role in a variety of real-world settings, such as negotiation, medical diagnosis, and criminal investigation. It serves as a fundamental methodology in the field of artificial general intelligence (AGI). As the underlying models continue to be developed, there is a growing interest in exploring their capabilities in inference tasks. In this paper, we introduce important underlying models proposed or applicable for inference and highlight the latest developments in a variety of inference tasks, methods, and benchmarks. We then explore the potential future directions behind the emergence of reasoning capabilities within the underlying model. We also discuss the relevance of multimodal learning, autonomous agents, and super-alignments in the context of reasoning. By discussing these future research directions, we hope that researchers will be inspired to explore this field, facilitate the development of reasoning through underlying models, and contribute to the development of AGI.

Conclusion:
This survey study sheds light on the evolutionary path of the underlying model in the field of inference, showing remarkable advances in complexity and efficiency from its early stages to current developments. While acknowledging remarkable advances in data-driven thinking, it is important to objectively recognize the strengths and limitations of large-scale models. In this context, it is imperative to emphasize the importance of enhancing interpretability and security. We also note that in all the papers surveyed in this study, no consensus has yet been reached on how to consistently elevate the reasoning power of the underlying model to a superhuman level (e.g., one that can win an IMO medal or solve an open mathematical problem).
In conclusion, while the underlying model offers interesting possibilities in the task of inference, it is imperative to approach it critically when it comes to its development and application. It is important to recognize the challenges, limitations, and risks associated with LLM-based reasoning. This facilitates responsible and thoughtful development in this field, ensuring the development of robust and reliable inference systems.

arXiv: https://arxiv.org/abs/2312.11562
Browse: https://browse.arxiv.org/pdf/2312.11562.pdf
PDF: https://arxiv.org/pdf/2312.11562.pdf
arXiv-vanity: https://www.arxiv-vanity.com/papers/2312.11562
Paper page: https://huggingface.co/papers/2312.11562
Papers with code: https://paperswithcode.com/paper/a-survey-of-reasoning-with-foundation-models
Github: https://github.com/reasoning-survey/awesome-reasoning-foundation-models
arXiv: https://arxiv.org/abs/2312.10997
Browse: https://browse.arxiv.org/pdf/2312.10997.pdf
PDF: https://arxiv.org/pdf/2312.10997.pdf
arXiv-vanity: https://www.arxiv-vanity.com/papers/2312.10997
Paper page: https://huggingface.co/papers/2312.10997
Papers with code: https://paperswithcode.com/paper/retrieval-augmented-generation-for-large
Github: https://github.com/tongji-kgllm/rag-survey