ANALYSIS OF THE EFFICIENCY OF INTERACTIVE CHAT BOTS FOR WORKING WITH DOCUMENTS: COMPARATIVE ANALYSIS OF BASIC LLM, NAIVE RAG AND GRAPHRAG METHOD

Boyko Aleksey Yuryevich

the aim of the work is to conduct a systematic comparative analysis of approaches for interactive chat bots answering questions about documents using large language models (LLM). The study is aimed at assessing the effectiveness of three methods: LLM, naive Retrieval-Augmented Generation (RAG) and the advanced GraphRAG approach, which integrates the extraction of relevant fragments with the construction of a knowledge graph for organizing and synthesizing information. Methods. This paper analyzes three approaches: – LLM, which relies solely on its internal knowledge, which results in limited accuracy when working with unseen documents; – Naive RAG, which implements extraction of relevant text fragments by semantic search over vector representations, which improves accuracy, but is limited to one or several fragments; – GraphRAG, which uses a structured representation of the text corpus in the form of a knowledge graph with preliminary summarization of fragments, which allows taking into account complex relationships between information fragments and provides more complete coverage of broad queries. Results. Experimental data indicate that GraphRAG provides significantly more complete and comprehensive answers compared to naive RAG and basic LLM. Improvements are especially noticeable in scenarios requiring multi-stage reasoning and synthesis of information from several sources, which minimizes the effect of hallucinations and limitations in the size of the context window. Conclusions. The obtained results confirm that the integration of the graph knowledge structure into the response generation process significantly improves the efficiency of interactive document-based chatbots. The GraphRAG approach is a promising solution for problems related to processing large and complex document collections, which opens up new opportunities for further research in the field of combining data extraction, summarization, and structured presentation.

Keywords:large language models, Retrieval-Augmented Generation, GraphRAG, chatbots, document analysis, information extraction, knowledge graph, interactive QA.