Latest / 1 More Paper / DKC-LLM: Dynamic Knowledge Caching for Large Language Models in Business Applications — Khaliq & Adebayo (2026)

Transcript

DKC-LLM: Dynamic Knowledge Caching for Large Language Models in Business Applications — Khaliq & Adebayo (2026) · 1 More Paper

• You’re listening to “DKC-LLM: Dynamic Knowledge Caching for Large Language Models in Business Applications,” by A. Khaliq and K.J. Adebayo. Published in 2026. Abstract. Large Language Models (LLMs) often face severe latency and computational cost constraints which hinder their adoption in real-time enterprise applications. Retrieval-Augmented Generation (RAG) frameworks, while improving factual accuracy, further increase inference delays owing to additional retrieval and context integration steps. To address these challenges, we propose Dynamic Knowledge Caching for Large Language Models (DKC-LLM), a novel framework that integrates dynamic semantic caching with an adaptive cache management strategy, which detects and compensates for semantic drift to accelerate response generation while ensuring accuracy and freshness. We evaluate DKC-LLM on two distinct question-answering benchmarks: BankFAQs, a domain-specific dataset with 5,000 selected queries and HotpotQA, an open-domain dataset with 5,000 selected queries, totaling 10,000 evaluation queries. Experimental results demonstrate that DKC-LLM achieves 20-30% lower latency, i.e., 5.3–6.8 ms vs. RAG’s 8–10 ms, a 63-70.5% cache hit rate, and a 60% reduction in LLM compute overhead, while maintaining 83-90% response accuracy. Additionally, DKC-LLM reduced hallucination rates by 75%, outperforming baseline RAG by up to 15 percentage points. These findings posit that DKC-LLM is a cost-efficient, low-latency, and high-accuracy solution for real-time, high-frequency business scenarios such as customer support and enterprise information retrieval. Introduction. The rapid proliferation of Large Language Models (LLMs) has transformed artificial intelligence (AI), enabling a wide range of capabilities across diverse domains, particularly in business analytics and decision-support systems. Among recent innovations, Retrieval-Augmented Generation (RAG), has emerged as a pivotal paradigm for enhancing factual accuracy and contextual grounding in knowledge-intensive tasks. By dynamically retrieving relevant external information, RAG mitigates hallucinations, and facilitates domain-specific personalization. The growing adoption of RAG frameworks across indus-tries has led to significant advances in architecture and optimization techniques, positioning RAG as a key The associate editor coordinating the review of this manuscript and approving it for publication was Maria Chiara Caschera. enabler of reliable enterprise-scale conversational systems such as banking and customer support chatbots for high-traffic business environments. Despite its advantages, RAG introduces substantial computational and latency overhead owing to the retrieval and context-integration steps inherent in its design. These delays become critical in real-time, high-frequency environments, such as financial or enterprise service platforms, where millisecond-level response times are essential for user satisfaction and operational efficiency. Conse-quently, reducing query latency in RAG-based systems remains an urgent research problem, as even marginal performance improvements translate into measurable business and energy-efficiency gains. Recent studies have explored various approaches to accelerating inference and reducing redundant computations,,,,,,,,,,, including strategies for mobile-edge optimization and internal key–value cache management within LLMs. However, most existing approaches rely on static or chunk-based caching, which overlook the challenges of semantic drift and dynamic knowledge evolution, thereby limiting their effectiveness in continuously changing enterprise contexts. To address these challenges, we propose Dynamic Knowledge Caching for Large Language Models (DKC-LLM), a novel framework that integrates a dynamic semantic caching layer within the RAG pipeline to substantially reduce redundant LLM inferences and accelerate response generation. Unlike conventional caching mechanisms that depend on exact matching or retrieval reuse,,,, DKC-LLM stores and reuses semantically similar, frequently accessed responses. A central innovation of the framework is its adaptive caching strategy, which leverages both query usage statistics and semantic drift detection to maintain response freshness. This ensures information currency while preserving accuracy and effectively addressing the per-sistent issue of stale knowledge in rapidly evolving business domains. Our experimental results demonstrate that DKC-LLM significantly outperforms the traditional RAG and recent cache-augmented variants. It achieves an average 17% reduction in query latency (6.06 ms vs. 8.26 ms), a 64% cache hit rate, and a 60% decrease in LLM compute overhead. Furthermore, DKC-LLM maintains 83–90% response accuracy, surpassing RAG’s 85%, while reducing hallucinations by approximately up to 50%. We evaluate the framework on two representative datasets: a domain-specific BankFAQs corpus capturing frequent customer-service queries, and the HotpotQA benchmark, which reflects open-domain reasoning and multi-hop question answering. A. RESEARCH OBJECTIVES This research aims to achieve the following objectives: 1. Framework Design: Develop Dynamic Knowledge. Caching for LLMs (DKC-LLM), a hybrid architecture integrating dynamic semantic caching with RAG to mitigate latency and computational bottlenecks in real-time business applications. 2. Adaptive Caching Mechanism: Implement a seman-. tic drift-aware caching module that utilizes query usage statistics and content embeddings to sustain the relevance and freshness of cached responses in evolving knowledge environments. 3. Comprehensive Evaluation: Empirically compare DKC-LLM against baseline RAG and cached-RAG approaches across key metrics such as latency, compute efficiency, and response accuracy, using both domain-specific (BankFAQs) and open-domain (HotpotQA) datasets. 4. Applied Impact: Demonstrate DKC-LLM’s viability as a cost-efficient, low-latency, and scalable architecture for high-frequency, real-time enterprise applications such as customer support and enterprise search. B. CONTRIBUTIONS The major contributions of this work are summarized as follows: • We introduce DKC-LLM, a dynamic semantic caching framework that integrates RAG to substantially reduce the inference latency and computational overhead in LLM-driven applications • We designed a caching mechanism that incorporates usage-based statistics and semantic drift detection, ensuring the continuous relevance and accuracy of cached responses without manual intervention. • We conducted a large-scale experimental evaluation demonstrating the superiority of DKC-LLM over baseline RAG and state-of-the-art cache-augmented variants in terms of latency, efficiency, and accuracy across both domain-specific and general benchmarks. • We showcase DKC-LLM as a scalable, low-cost, and high-performance solution for real-time, high-throughput business environments, thereby providing new directions for sustainable LLM deployment. The remainder of this paper is structured as follows: Section II provides a review of related work. Section III presents the architecture and core components of the DKC-LLM framework, along with a description of the experimental setup. Section III-E1 discusses the results and key findings, Section III-E2 discusses the theoretical and implications of our proposed framework, and finally, Section IV and V conclude the paper and outline directions for future research. II. RELATED WORK. This section reviews the key literature underpinning this study. First, we discuss the evolution and applications of Retrieval-Augmented Generation (RAG), emphasizing its deployment within enterprise settings. We then examine the existing approaches for optimizing LLM inference and caching mechanisms, and highlight their limitations. Finally, we consider recent advances in semantic similarity modeling, which form the foundation of dynamic semantic caching. A. RETRIEVAL AUGMENTED GENERATION (RAG) The advent of large language models (LLMs) has revo-lutionized Natural Language Processing (NLP), enabling breakthroughs in generation, summarization, and question answering. Despite their success, LLMs exhibit several well-known limitations, including static parametric knowledge owing to a fixed training cut-off, susceptibility to hallucination, and an inability to access proprietary or dynamically changing information. To address these challenges, Retrieval-Augmented Generation (RAG) has been introduced as a hybrid paradigm that combines generative modeling with dynamic retrieval from external knowledge sources. A RAG system typically comprises a retriever, which iden-tifies semantically relevant passages or documents, and a generator, that synthesizes responses conditioned on both the user query and the retrieved context. Foundational work in dense retrieval, such as Dense Passage Retrieval (DPR), has laid the groundwork for efficient query–document matching. RAG has since been successfully applied across tasks ranging from open-domain question answering to specialized knowledge-intensive domains. Recent developments have advanced the RAG architectures along several fronts. Surveys such as synthesize innovations including iterative retrieval refinement, self-correcting RAG for robustness,, and structured retrieval pipelines that restrict evidence to verified cor-pora. Moreover, optimization efforts by researchers span retriever–generator alignment, improved passage re-ranking, and enhanced embedding models for semantic similarity. Despite these strides, system-level challenges persist, particularly regarding end-to-end latency, computational efficiency, and the adaptive handling of dynamic or time-sensitive knowledge, which are key barriers to deploying RAG in real-time enterprise contexts. B. RAG IN BUSINESS APPLICATION The integration of RAG into enterprise systems has gained considerable momentum owing to its ability to ground LLMs in organization-specific knowledge bases. Applications include intelligent customer support, enterprise searches, and internal knowledge management. Early evaluations, such as, demonstrated gains in the accuracy and efficiency achievable through RAG-powered chatbots. Further studies have extended these findings to customer service analytics, industrial knowledge management, business process automation, domain-specific assistants in finance, and healthcare. RAG’s value of the RAG in these domains lies in its ability to combine generative fluency with factual precision. However, as listed in Table 1, most enterprise-focused RAG implementations remain constrained by latency and redundancy issues. Frameworks such as CAPRAG and Retail-GPT enhance retrieval quality but lack semantic caching to optimize repeated query handling. Others, including DRAG and modular retrieval pipelines, improve contextual adaptation but do not address response reuse or stale knowledge detection. These limitations become acute in high-frequency business environments, where repeated retrieval and inference of similar queries inflate costs and degrade user experience. Thus, efficient caching and semantic reuse represent critical, yet underexplored, optimization avenues for enterprise RAG systems. Given the computational intensity of LLM inference, significant research has targeted efficiency improvements through architectural and algorithmic approaches. Beyond foundational innovations such as the transformer pow-ering recent models such as Gemma 3, Llama 4, and Phi 4, optimization strategies include quantiza-tion, knowledge distillation, and high-throughput serving frameworks. Caching has emerged as a promising method to reduce redundant computations. Traditional caches store exact query–response pairs; however, natural language variability necessitates semantic caching, where vector embeddings identify semantically similar queries. This approach enables the reuse of pre-computed responses across linguis-tically diverse but semantically equivalent requests. Recent research into caching can be grouped into three main categories: (a) Internal LLM Caching. Techniques such as key-value (KV) cache optimization target long-context inference and throughput enhancement by managing the memory more effectively. For example, ARKVALE introduced a recallable eviction strategy to reduce re-computation, whereas explored multi-tier KV management for heterogeneous hardware. Similarly, ChunkAttention eliminates redundant prefix computation. These approaches improve inference efficiency but focus on model internals rather than, end-to-end RAG pipelines. (b) Query and Response Caching. Frameworks such as RagCache, PipeRAG, CacheCraft, and CacheBlend aim to reuse retrieval results or generated responses. Other studies propose approximate caching or semantic caching of intermediate representations to accelerate multi-hop reasoning. While these techniques reduce retrieval cost, they often still require fresh LLM inference for each query, which maintains high computation overhead and variability in responses. Moreover, most lack mechanisms for adaptive refresh when the underlying knowledge changes. (c) Context-Aware Caching. Emerging work has explored policy-driven caching to anticipate user needs. Chan et al. introduced cache-augmented generation as a potential substitute for retrieval in certain scenarios, while the Adaptive Contextual Caching (ACC) framework employs deep reinforcement learning to manage caching policies dynamically in mobile-edge environments. These innovations reveal the potential of caching beyond exact reuse but fall short in addressing semantic drift and knowledge staleness, which are issues central to large-scale enterprise deployments. Moreover, complementary research on hallucination miti-gation,, emphasizes the value of architectures that inherently enhance factual reliability through vali-dated knowledge integration. However, the role of adaptive semantic caching in reducing hallucination by preferentially reusing previously verified responses remains underexplored. It is essential to distinguish algorithmic frameworks such as DKC-LLM from infrastructure-level solutions in the likes of Amazon MemoryDB and Azure Cosmos DB, etc. While these platforms offer scalable and reliable vector storage, they do not natively address cache validity, semantic drift, or reuse policies, instead delegating such responsibilities to external application logic. DKC-LLM is explicitly designed to operate as this governing semantic control layer, providing principled mechanisms for cache admission, reuse, and invalidation. Moreover, unlike system-level optimizers such as RagCache, which primarily manage internal key–value states or document tree structures, DKC-LLM targets end-to-end response-level caching. This design choice enables direct optimization of downstream inference costs and latency at the level most relevant to user-visible performance. This distinc-tion motivates our comparison with VC-RAG, as it allows us to isolate the performance benefits attributable specifically to semantic reuse and governance mechanisms, rather than to conventional exact-match or infrastructure-assisted caching. In summary, although caching has proven effective in improving RAG throughput, existing systems lack a uni-fied framework that dynamically manages semantic freshness and usage-adaptive caching. This gap motivates our proposed Dynamic Knowledge Caching (DKC-LLM) framework, which integrates semantic drift detection and adaptive cache refresh into the RAG to ensure both efficiency and accuracy in evolving knowledge environments. C. SEMANTIC SIMILARITY AND SEMANTIC MODELS Effective semantic caching critically depends on robust embedding models that capture fine-grained semantic rela-tionships between queries. Modern Sentence-BERT architectures, encode contextual meanings into dense vector representations. In this study, we adopted all-MiniLM-L6-v2, which offers a strong balance between accuracy and computational cost, making it suitable for high-throughput enterprise applications. For efficient similarity retrieval, we utilized FAISS, an open-source optimized vector search library that enables a sub-millisecond nearest-neighbor search over large-scale embeddings. While larger or domain-specific models (e.g., Col-Bert and, ModernBert), offer potential accuracy gains, they impose higher inference costs. Thus, the combination of lightweight Sentence-BERT embeddings and efficient FAISS indexing provides an effective trade-off for dynamic, low-latency semantic caching. Further research has explored embedding adaptation for drift detection and adaptive semantic caching in dynamic retrieval contexts,, both of which inform the adaptive mechanisms implemented within DKC-LLM. D. CHALLANGES AND RESEARCH GAP Synthesizing the discussed literature, three major challenges persist in the deployment of RAG systems for enterprise use. First, the retrieval latency inherent in standard RAG pipelines remains a bottleneck for real-time applications. Second, existing caching solutions often rely on static or exact-match mechanisms, failing to address the semantic variability of natural language queries. Finally, there is a lack of adaptive drift detection mechanisms capable of maintaining knowledge freshness in rapidly evolving business domains without manual intervention. This study addresses these specific gaps by proposing a dynamic, semantic, and drift-aware caching framework. III. METHODOLOGY. This section outlines the experimental design used to evaluate the effectiveness of the proposed DKC-LLM framework. First we describe the datasets employed for both the domain-specific and general knowledge evaluations. Next, we detail the architectural components of the DKC-LLM and baseline models used for comparison. Finally, we spec-ify the experimental setup and metrics used to quantify performance. A. DATASETS We used two distinct datasets to evaluate the model performance across different complexity levels and domain specificities. 1) BANKFAQS DATASET. To assess performance in a domain-specific business envi-ronment, we used the publicly available BankFAQs dataset, which contains over 1,770 question-answer pairs frequently encountered by banking customers. The dataset was structured into Question, Answer, and Class columns, covering topics such as savings and current accounts, loans, credit cards, online banking services, and general banking policies. This dataset provides a realistic representation of static yet frequently accessed knowledge which is typical of customer support scenarios. 2) HOTPOTQA DATASET. To evaluate complex multi-hop reasoning and retrieval capabilities, we used HotpotQA. This dataset contains 90,447 question-answer pairs, along with gold supporting facts and distracting passages, requiring models to reason across mul-tiple documents. HotpotQA allows us to assess the retrieval accuracy and the ability of models to synthesize information in noisy, multi-document contexts. B. MODEL ARCHITECTURES We implemented and compared the following architectures: 1) NAÏVE RETRIEVAL-AUGMENTED GENERATION (RAG). The Naïve RAG baseline1 follows Lewis et al. and integrate a dense retrieval module with a generative language model. The architecture contains the following modules: a: DOCUMENT INDEXING Relevant documents from the target knowledge base (i.e., the BankFAQs and HotpotQA datasets articles) were prepro-cessed and segmented into passages. Each passage is encoded into a high-dimensional vector using the All-MiniLM-L6-v2 model, from the Sentence Transformers library. These embeddings were indexed using FAISS2, enabling efficient approximate nearest-neighbour retrieval. b: RETRIEVER Incoming queries are embedded using the same encoder, and the top-5 relevant passages are retrieved based on the inner-product similarity. c: RE-RANKER To improve relevance, the retrieved passages were re-scored using a cross-encoder-based re-ranker adapted from Nogueira and Cho. This model jointly encodes the query and each candidate passage to compute the fine-grained relevance scores. The top-3 re-ranked passages were selected as input to the generative model. d: GENERATOR We employed Gemma 3n, a lightweight decoder-only LLM optimized for edge-based scenarios.3 Our pipeline, receives the user query along with the top-3 re-ranked context passages as input and produces fluent, factually grounded responses from the re-ranked passages. This baseline provides a standard against which we mea-sure improvements in latency, efficiency, and accuracy. C. VANILLA CACHED RAG (VC-RAG) VC-RAG integrates a simple caching mechanism to avoid redundant computation for repeated queries,,,,. Figure 1 shows the workflow architecture of VC-RAG. The caching mechanism is implemented as a lightweight key value store during the query processing stage. Upon receiving a new user query, the system executes the following steps: 1) Cache Lookup: An incoming query is first checked. against cache. A cache hit is registered if an identical or highly similar query (cosine similarity > 0.98) has been processed previously. 1For clarity, the terms standard RAG, baseline RAG, naive RAG and full RAG are used interchangeably throughout the manuscript to denote the same system configuration, which corresponds to our baseline retrieval-augmented generation pipeline (VC-RAG) without caching. 2FAISS (Facebook AI Similarity Search). Available at: the linked source 2) Cache Hit: If a match is found, the corresponding. cached output, including the retrieved documents and the generated response, is returned directly, bypassing the retrieval, re-ranking, and generation stages. 3) Cache Miss: Queries not found in the cache pass. through the Naïve RAG pipeline, and the resulting pair is stored for future use. Although this baseline efficiently handles repeated or near identical queries, it cannot generalise across semantically equivalent but syntactically diverse inputs, a limitation frequently encountered in real-world open-domain and conversational retrieval settings. D. DYNAMIC KNOWLEDGE CACHING (DKC-LLM) Our proposed framework, DKC-LLM, is designed to significantly reduce the latency and computational overhead in LLM-powered applications by intelligently integrating a dynamic semantic caching layer. Unlike traditional RAG or VC-RAG baselines, DKC-LLM’s core innovation lies in its ability to serve semantically similar queries directly from a pre-computed and adaptively updated cache, thereby bypassing both the retrieval and generation steps for a substantial portion of queries. The architectural workflow is illustrated in Figure 2. The framework comprises the following components: 1. User Query & Embedding: An incoming user query is. first embedded into a high-dimensional vector using the All-MiniLM-L6-v2 model. 2. Semantic Cache Lookup: This query embedding is. then used to perform a real-time semantic similarity search against the embeddings of cached questions within the Semantic Cache. The cache utilises the FAISS index for an efficient similarity computation. A cosine similarity threshold of 0.85 is applied such that if a cached question’s embedding is found to be above this threshold, it is considered a cache hit. 3. Cache. Hit Scenario: If a semantically similar query is identified in the cache, the corresponding pre-computed answer is instantaneously returned. This direct retrieval significantly reduces the latency and completely bypasses the computationally expen-sive RAG pipeline (retrieval, re-ranking, and LLM inference). 4. Cache Miss & Population: If no match is found, the. query goes through the RAG pipeline, and a new query-answer embedding is added to the cache. 5. Adaptive Cache Update: A distinguishing feature of. DKC-LLM is its adaptive mechanism for maintaining cache currency and relevance, addressing the issue of ‘‘semantic drift’’ or outdated information. This is achieved as follows: ◦ Usage Statistics: The cache monitors query frequency. Less frequently accessed or old entries are candidates for eviction. ◦ Semantic Drift Detection: Cache entries undergo revalidation based on two primary conditions: (i) when the time-to-live (TTL) threshold of 24 hours is reached, a duration chosen as a practical default given that enterprise knowledge is typically refreshed on a daily basis. This period represents a balance between ensuring cache freshness and maintaining computational efficiency, and (ii) when updates to the underlying knowledge base are detected. To optimize computational resources, drift checks are conducted in batches after every 100 queries, thereby reducing the frequency of unnecessary recalculations. Cached responses are compared to regenerated outputs using cosine similarity; if the similarity score falls below a predefined threshold, the entry is flagged for either an update or invalidation. Notably, this revalidation process intro-duces minimal computational overhead (i.e., <0.5 ms per query on average), ensuring that it does not significantly impact end-to-end latency, even under high-throughput scenarios. To maximise effectiveness, the proposed framework employs a pre-computation and pre-loading mechanism that initialises the semantic cache with frequently asked questions. These entries are dynamically managed and updated at runtime. In contrast to purely static pre-computation approaches or context-aware caching techniques, which primarily focus on optimising the internal state of the language model, our DKC-LLM continuously refines its cache based on semantic similarity and proactive content freshness. This dynamic adaptability makes the framework particularly well-suited for environments charac-terised by evolving knowledge and frequently changing information. The algorithm below presents the complete deployment workflow of the DKC-LLM framework, detailing the inference pipeline, semantic caching, and asynchronous drift detection mechanisms. E. EXPERIMENTAL SETUP Our experiments were designed to simulate realistic usage patterns within customer support environments and to facilitate a rigorous comparative analysis of the proposed DKC-LLM framework against RAG and VC-RAG systems. 1) DATASETS AND QUERY SIMULATION. We simulated 10,000 queries by leveraging a bootstrapped subset of 1,000 diverse questions. This subset was com-posed of two distinct parts: 200 queries from the BankFAQs dataset, which represent common domain-specific customer inquiries, and 800 queries from the HotpotQA dataset, covering more complex, reasoning-intensive general knowledge questions. To more closely reflect real-world customer support behaviour, where a relatively small set of popular queries dominate traffic, we expanded the subset into 5,000 queries per dataset, 200 queries were designated as cold-start inputs, while the remaining 4,800 queries were generated through paraphrasing, synonym replacement, and automated data augmentation. This approach aimed to simulate repeated but semantically varied customer inquiries (e.g., ‘‘What are the requirements to open a savings account?’’ → ‘‘How do I start a new savings account at the bank?’’). For the HotpotQA dataset, a similar procedure was fol-lowed. Of the 1,000 queries, 200 were reserved as cold-start examples, while the remaining seed queries were augmented using paraphrasing techniques to generate a set of 4,800 semantically varied queries. These were designed to simulate warm-cache behaviour under multi-hop reasoning scenarios (e.g., ‘‘Which company was founded earlier, Tesla or Ford?’’ → ‘‘Between Ford and Tesla, which one was established first?’’). This process resulted in a balanced evaluation set of 10,000 queries (5,000 per dataset), enabling a fair assessment of both cold-start latency and warm-cache efficiency. Additionally, over 200 semantically similar queries from the Bank-FAQs dataset were specifically analyzed to evaluate cache effectiveness in handling repetitive, domain-specific inquiries. 2) DATA PREPROCESSING AND EMBEDDING. All textual data, including incoming user queries, under-went minimal preprocessing to retain the semantic and syntactic features necessary for transformer-based embeddings. Specifically, we applied whitespace normalization and Unicode standardization while deliberately retaining case, punctuation, and special characters, as these elements carry contextual cues essential for accurate semantic representation. For generating dense vector representations of both queries and candidate passages, we employed the all-MiniLM-L6-v2 model from the Sentence Transformers library. This model was selected for its computational efficiency and robust performance on semantic similarity tasks, offering an optimal balance between accuracy and inference speed. To reduce redundancy and better mimic real-world usage patterns, duplicate and semantically similar queries were clustered using cosine similarity with a threshold of 0.85. The resulting embeddings were indexed using FAISS, allowing efficient retrieval based on vector similarity. At run-time, the system embeds the incoming user query, retrieves the top-k most relevant FAQ entries from the FAISS index, and composes a template-based response accordingly. All experiments were conducted on a workstation equipped with an Intel Xeon E3-1505M v5 CPU, 64 GB of RAM, and an NVIDIA Quadro P6000 GPU (24GB VRAM), using PyTorch and the Hugging Face Transformers ecosystem4 for model deployment. We make all experimental artifacts publicly available,5 including source code, configuration files, dataset splits, FAISS indexes, and benchmark logs. The repository also contains timing harnesses for latency profiling across indi-vidual pipeline stages (embedding, retrieval, re-ranking, and generation), along with comprehensive environmental specifications. These resources facilitate full reproducibil-ity, independent validation, and extension of the DKC-LLM framework to additional domains. F. EVALUATION METRICS To provide a comprehensive and quantitative assessment of DKC-LLM’s performance relative to our established baselines, we employ the following metrics: Latency (Response Time): This is measured as the end-to-end duration from the query input to the generation of a complete response. The latency was recorded in milliseconds using Python’s time module. The average latency over all simulated queries was reported, providing a direct measure of the system responsiveness. XN  Latency = 1 |1=1| ·Li N where N is the total number of queries and Li is the latency for query i. Computational Efficiency: This metric quantifies the efficiency gains achieved by avoiding redundant LLM infer-ences and retrieval operations. This is primarily reflected by the cache hit rate, which measures the percentage of queries served directly from the cache. NumberofCacheHits CacheHitRatio = TotalNumberofQueries We also quantified the computational savings by measuring the redundancy score, which captures the number of repeated queries avoided by precomputations. ComputationalSavings = RC RAG − RC DKC −LLM RCRAG where: - RC RAG is the total number of retrieval and generation computations required by the standard RAG pipeline, - RC DKC −LLM represents the same metric for the DKC-LLM system. For experiments involving the VC-RAG pipeline, RC RAG is substituted with RC VC −RAG, denoting the retrieval and generation count for that model. Accuracy: The factual correctness and completeness of the generated answers were evaluated using a combination of automated and manual methods. For both datasets, accuracy was assessed via exact match and semantic equivalence against predefined gold-standard answers. Semantic similarity was computed by comparing the embeddings of the generated and reference answers using cosine similarity. For the HotpotQA dataset, accuracy was further evaluated by comparing the generated answers not only to the ground-truth answers but also to their asso-ciated supporting facts, providing a more comprehensive assessment of the model’s performance. In addition to automated evaluation, manual verification was carried out by domain experts to assess the quality of the answers relative to the gold-standard references. Specifically, two graduate students with backgrounds in business and finance rated the responses based on relevance, correctness, and completeness, following a standardized rubric. To assess the reliability of the manual evaluation, we cal-culated the inter-rater agreement using Cohen’s Kappa (κ), which yielded a score of approximately 0.82, indicating a moderate level of agreement. This score reflects dis-crepancies in 18% of the cases, which were subsequently resolved through discussion to ensure consistency in the final evaluation. Finally, response accuracy was computed as a weighted combination of the Exact match score, the Semantic similarity score, and the Manual expert ratings. The following equations define the unweighted and weighted accuracy scores: Accuracy = Number of Correct Responses Total Number of Responses WeightedAccuracy = α · ExactMatch + β · SemanticSimilarity + γ · ManualScore where α + β + γ =1, and the weights are determined based on validation agreement among evaluators and the confidence in each signal. Cache Responsiveness to Updates: To validate DKC-LLM’s capability to maintain knowledge currency and over-come staleness, we tracked the responsiveness of the cache to updated answers. This involved introducing deliberate modi-fications to a subset of BankFAQs answers and observing the time and accuracy with which DKC-LLM’s adaptive cache mechanism reflected these changes, in contrast to a static cache baseline. IV. RESULTS. We now present an evaluation of the proposed DKC-LLM framework against Naïve RAG and VC-RAG baselines. Our experiments systematically measured key performance indi-cators across the BankFAQs and HotpotQA datasets, focusing on latency, computational efficiency (via cache hit rate), and response accuracy. A comprehensive quantitative overview of all evaluation metrics is summarized in Table 2. A. LATENCY PERFORMANCE 1) AVERAGE LATENCY WITH CONFIDENCE INTERVAL. The most significant performance advantage of DKC-LLM is in its superior latency. As shown in Figure 3, DKC-LLM consistently demonstrated lower end-to-end response latency than both Naïve RAG and VC-RAG. Latency, recorded as the time taken from the query input to the generation of a response, is a critical indicator in real-time applications. For BankFAQs which contains 5,000 total queries, 200 queries were processed under cold-cache conditions and 4,800 under warm-cache conditions. As shown in Figure 3, DKC-LLM demonstrated consistently lower response latency than both baseline systems. During the warm-cache oper-ation, DKC-LLM achieved an average response time of 6.3 milliseconds, with a 95 percent confidence interval between 6.1 and 6.5 milliseconds. This represents a clear improvement over the 6.9 milliseconds recorded by VC-RAG and the 8.4 milliseconds observed for the Naïve RAG baseline. Even when operating under cold-cache conditions, where limited prior cache entries are available, DKC-LLM maintained a latency of 7.2 milliseconds, compared with 7.6 milliseconds for VC-RAG and 9.1 milliseconds for Naïve RAG. Across all trials, the framework sustained an average throughput of approximately 2,280 tokens per second with responses averaging 18 tokens in length, underscoring its capacity for real-time, high-frequency business interactions. As shown in Figure 4, the performance trends were consistent in the HotpotQA experiments, which included an equivalent number of queries. Under warm-cache conditions, DKC-LLM produced responses with an average of 6.8 milliseconds, with a 95 percent confidence interval ranging from 6.6 to 7.0 milliseconds. This performance exceeded that of VC-RAG, which averaged 7.4 milliseconds, and Naïve RAG, which required 9.8 milliseconds per query. Under cold-cache scenarios, DKC-LLM maintained an average latency of 7.6 milliseconds, compared to 8.1 milliseconds for VC-RAG and 10.4 milliseconds for Naïve RAG. The model achieved a mean throughput of approximately 2,310 tokens per second, generating responses that averaged 23 tokens in length. Taken together, these results indicate that DKC-LLM consistently delivers 20–30 percent faster response times than conventional RAG pipelines. The observed improvements in both latency and throughput highlight the practical advantage of dynamic semantic caching in sustaining real-time performance in both domain-specific and knowledge-intensive question-answering environments. 2) THRESHOLD BASED LATENCY ANALYSIS. Figure 5 presents the results of the threshold-based latency analysis, illustrating DKC-LLM responsiveness under vary-ing time constraints. On the BankFAQs dataset, which reflects high-frequency, domain-specific queries, DKC-LLM demonstrated a marked advantage over both baseline models. Within the two-millisecond threshold, the proposed framework successfully resolved 23% of all queries, whereas neither the Naïve RAG nor the VC-RAG baseline produced any responses within this range. Expanding the threshold to five milliseconds further accentuated this difference: DKC-LLM successfully processed 67.4% of queries within this limit, compared with only 2.6% for Naïve RAG and 22.7 percent for VC-RAG. These results highlight the efficiency of DKC-LLM in serving repetitive or semantically aligned queries, which are common in customer-facing domains such as banking. On the HotpotQA dataset, the overall latency increased across all models owing to the additional computational complexity of the multi-hop reasoning. Even under these demanding conditions, the, DKC-LLM maintained a mea-surable performance advantage. At the two-millisecond threshold, it successfully handled 10.3% of queries, while both baselines achieved no successful completions within this range. By the five-millisecond threshold, DKC-LLM achieved 21.0% query coverage, compared with 0.1% for Naïve RAG and 2.5% for VC-RAG. These findings under-score DKC-LLM’s capacity to maintain rapid response capabilities even in complex reasoning scenarios, confirming that the integration of a dynamic semantic cache substantially reduces latency across diverse task complexities. 3) TAIL LATENCY. To complement the mean latency results, we conducted a tail latency analysis to assess the stability and predictability of each model’s response time. Tail latency, measured at the 50th (P50), 95th (P95), and 99th (P99) percentiles, reflects the time within which 50%, 95%, and 99% of queries are completed, respectively. This analysis provides insight into how each framework performs under high-load or outlier conditions that critically affect real-time user experience. On the BankFAQs dataset, we can see from Figure 7 that the Naïve RAG model exhibited pronounced long-tail delays, with P95 latency reaching 42.0 milliseconds and P99 latency extending to 45.1 milliseconds, indicating frequent slow responses under peak load. The VC-RAG baseline reduced this variance, achieving P50, P95, and P99 latencies of 6.8, 13.4, and 15.5 milliseconds, respectively. DKC-LLM demonstrated the most stable latency distribution, with corresponding values of 6.2, 10.8, and 12.9 milliseconds, reflecting its ability to minimize performance degradation even under heavy query repetition. As shown in Figure 8, on the HotpotQA dataset, which involves multi-hop reasoning and more complex retrieval dependencies, the Naïve RAG again produced the slow-est tail responses, achieving P95 and P99 latencies of 29.2 ms and 32.5 ms respectively. Both VC-RAG and DKC-LLM maintained substantially lower tail latencies, with DKC-LLM slightly outperforming VC-RAG across all per-centiles. Specifically, DKC-LLM recorded P50, P95, and P99 values of 6.7, 9.6, and 10.4 milliseconds, compared to 7.3, 10.4, and 11.8 milliseconds for VC-RAG. These results indicate that DKC-LLM not only reduces average response time but also ensures greater temporal stability, which is a critical property for latency-sensitive applications such as conversational assistants and real-time customer support systems. Moreover, the adaptive cache update mechanism introduced minimal computational overhead, adding on average less than 0.5 milliseconds per query, corresponding to less than 1% of the total processing time. This negligible cost demonstrates that the proposed semantic drift detection and cache-refresh procedure does not compromise the latency advantages of DKC-LLM, even when operating under high-throughput, real-world workloads. B. CACHE HIT RATIO AND COMPUTATIONAL EFFICIENCY Beyond raw latency improvements, the efficiency of the DKC-LLM framework is further demonstrated by its significantly higher cache hit rate, which directly translate into reduced redundant retrieval and LLM inference operations. Across both evaluation datasets, the DKC-LLM consistently demonstrated superior caching performance. On the BankFAQs dataset, it achieved a 63% cache hit rate, whereas on HotpotQA, it reached 67%. In comparison, the VC-RAG baseline attained 50% and 55%, respectively, and the Naïve RAG system, which lacked any caching component, was 0%. These improvements directly correlate with consider-able computational efficiency. As defined in Equation, DKC-LLM reduces retrieval and generation computations by approximately 60% relative to Naïve RAG. This reduction is particularly advantageous in high-frequency business environments, such as banking and financial services, where users often issue semantically similar queries related to account operations, transactions, or product eligibility. In such scenarios, DKC-LLM effectively mitigates backend load, minimizes response latency, and significantly improves throughput in enterprise-scale deployments. Figures 9 and 10 compare the cache hit ratios across all evaluated systems. The results clearly demonstrate DKC-LLM’s superior ability to detect semantically equivalent queries and reuse previously generated responses, leading to consistently higher cache efficiency across both datasets. In terms of memory overhead, the semantic caching layer remains lightweight, requiring approximately 150 MB to store 5,000 cached entries for either dataset and, scaling linearly to approximately 300 MB for 10,000 entries. This footprint is well within the capacity of standard GPU or enterprise server configurations, confirming that DKC-LLM’s semantic caching can be deployed at scale without imposing substantial memory constraints. 1) SIMILARITY THRESHOLD SENSITIVITY. To investigate the influence of similarity thresholds on cache effectiveness, we conducted a parameter sweep over cosine-similarity values of 0.70, 0.75, 0.80, 0.85, and 0.90 for both datasets. For each configuration, we measured cache-hit ratio, hit precision, hit recall,and F1 scores. The results, summarized in Table 2, reveal a clear trade-off. Lower thresholds (≤ 0.75) maximize recall and hit ratio, 84.2 percent on BankFAQs and 76.7 percent for HotpotQA, but allow semantically weaker matches, reducing precision. In contrast, stricter thresholds (≥ 0.85) improve precision to over 94 percent, but at the cost of reduced recall and cache utilization. The intermediate range of 0.80–0.85 offers the most bal-anced outcome, preserving both high retrieval accuracy and cache effectiveness. In particular, DKC-LLM operating at 0.85 maintains > 94 percent hit precision, high recall, and stable F1 scores across datasets. This sensitivity analysis empirically supports the choice of 0.85 as the default similarity threshold, ensuring a robust balance between cache reliability and semantic generalization while minimizing the risk of drift-induced inaccuracies. C. RESPONSE ACCURACY The accuracy of generated responses remains a critical evaluation dimension in retrieval-augmented generation (RAG) frameworks. In this work, we adopt Exact Match (EM) and F1 scores as the primary evaluation metrics, following standard QA benchmark practice. EM measures the percentage of system outputs that exactly match the gold-standard reference answers, while F1 captures partial matches based on token overlap, balancing precision and recall. We report dataset-level averages with 95% confidence intervals to ensure that the improvements are statistically grounded. As summarized in Table 3, DKC-LLM consistently out-performed both baseline models across all evaluated datasets. On the BankFAQs dataset, DKC-LLM achieved an Exact Match score of 82.4 ± 0.9 percent and an F1 score of 87.1 ± 0.8 percent, exceeding the performance of Naïve RAG, which reached 72.6 percent EM and 79.3 percent F1, and VC-RAG, which obtained 78.2 percent EM and 83.6 percent F1. On the more challenging HotpotQA dataset, we see that DKC-LLM maintained its advantage with 70.8 ± 1.0 percent EM and 76.4 ± 0.9 percent F1, compared with 65.4 and 71.5 percent for Naïve RAG, and 69.1 and 75.2 percent for VC-RAG. Moreover, bootstrap resampling analysis over 1,000 itera-tions confirmed that these improvements were statistically significant (p < 0.05). Beyond quantitative accuracy metrics, we also assessed factual consistency through a focused hallucination analysis on a randomly sampled subset of 20 HotpotQA queries. Two independent human raters were employed to manually annotate hallucination instances. A high level of inter-rater agreement was observed, with a Cohen’s κ value of 0.82, indicating substantial reliability between both raters. The annotations revealed a marked reduction in erroneous content generation across models. Specifically, Naïve RAG exhibited a 20% hallucination rate, VC-RAG produced 15%, and DKC-LLM achieved the lowest rate of 5%. These findings align with recent findings on retrieval fidelity and hallucination mitigation in RAG architectures, and further suggest that the semantic caching and adaptive update mechanisms in DKC-LLM not only enhance response accuracy but also contribute to improved factual robustness by mitigating hallucination propagation. D. EVALUATION THROUGH ABLATION STUDIES To isolate the contribution of each architectural component within DKC-LLM, we conducted an ablation study on both the BankFAQs and HotpotQA datasets. Three controlled configurations were evaluated: 1) Full DKC-LLM (Full), incorporating both semantic. caching and adaptive cache management. 2) Without Adaptive Cache Management (–ACM), in. which cached entries were not refreshed based on usage statistics or semantic drift detection. 3) Without Semantic Cache (–SC), where caching is. limited to exact query matches, the semantic similarity layer is entirely removed As summarized in Table 4, the full DKC-LLM setup achieved the best overall performance, with average latencies of 5.3 ms on BankFAQs and 6.2 ms on HotpotQA, cache hit ratios of 63 and 67%, and F1 scores of 90.1 and 83.0%, respectively. We observed that removing the adaptive cache management led to moderate degradation, increasing latency by approximately one millisecond and lowering cache hit ratios by approximately 16 percentage points, confirming its role in maintaining cache freshness. Similarly, eliminating the semantic cache produced the most significant performance loss, reducing hit ratios below 30% and decreasing F1 by more than 15 points on both datasets. This result demonstrates that semantic caching is the principal mechanism enabling DKC-LLM to efficiently handle paraphrased and semantically diverse queries. Overall, these results confirm that both semantic caching and adaptive cache management modules are essential for realizing full efficiency and effectiveness of the DKC-LLM framework. When either component is removed, overall performance regresses toward that of simpler cache-augmented RAG systems such as VC-RAG, aligning with prior observa-tions reported in RagCache Building upon the component-level analysis from the ablation study, we conclude the experimental evaluation by providing a consolidated summary of the system’s deployment environment and overall performance metrics. To ensure reproducibility, Table 5 outlines the hardware specifications, system throughput, and canonical latency results across both the BankFAQs and HotpotQA datasets. These results highlight the stability of DKC-LLM under varying computational loads. V. DISCUSSION. In this section, we provide an integrated interpretation of the experimental findings, emphasizing the unique strengths of the DKC-LLM framework and its implications for theory, system design, and practical deployment in knowledge-intensive, LLM-powered applications. A. INTERPRETATION OF KEYFINDINGS As detailed in Section III-E1, DKC-LLM delivers consistent improvements in latency, computational efficiency, and response accuracy compared to baseline models. Figures 7 and 8 and Table 2 collectively illustrate these gains across both the BankFAQs and HotpotQA datasets. One of the major contributions is the substantial reduction in end-to-end latency. DKC-LLM achieves average response times of approximately 5.3 ms on BankFAQs and 6.2 ms on HotpotQA, which are notably faster than Naïve RAG (8.3 ms and 9.0 ms) and VC-RAG (6.9 ms and 7.2 ms). This improvement can be attributed to the dynamic semantic caching of the DKC-LLM mechanism, which serves 63–67% of queries directly from cache, effectively bypassing the computationally intensive retrieval, re-ranking, and generation stages. To generalize this finding, we observe that the latency reduction is an inherent feature of the architecture, scaling linearly with the cache hit ratio. This is because the semantic lookup time is negligible compared to the RAG generation time. These near-instantaneous cache hits make DKC-LLM particularly well-suited for latency-sensitive business applications, such as customer support, where user experience is highly dependent on responsiveness. Beyond latency, the higher cache hit ratio translates into significant computational savings, reducing LLM inference and retrieval overhead by approximately 60%. This efficiency advantage is particularly valuable in enterprise settings, where a small set of high-frequency queries dominates operational traffic. The improvement over VC-RAG at 50–55% hit rate stems primarily from DKC-LLM’s semantic similarity– based lookup, which effectively identifies paraphrased or syntactically diverse but semantically equivalent queries that simpler caching mechanisms often overlook. Importantly, these efficiency gains do not come at the expense of accuracy. DKC-LLM consistently achieves the highest accuracy, ranging between 90–93% while exhibiting the lowest hallucination rate at 5%, compared to 10% for Naïve RAG and 15% for VC-RAG. These gains arise from two factors: pre-validated cached responses provide stable and factually grounded answers, and reduced reliance on frequent LLM inference inherently mitigates hallucination risk. Together, these mechanisms contribute to a more reliable and consistent user experience. B. CACHE RESPONSIVENESS AND KNOWLEDGE UPDATES A key differentiator of DKC-LLM is its adaptive cache update mechanism, which is designed to prevent semantic drift and ensure knowledge freshness. Experiments on the BankFAQs dataset demonstrate its effectiveness. When factual updates were introduced to 25 frequently queried items (e.g., interest rates, loan policies), DKC-LLM’s drift detection mechanism correctly flagged 92% of outdated cache entries within 20 queries. The accuracy temporarily decreased from 90% to 78% immediately after the update but recovered to 88% following automated cache re-validation. In contrast, a static cache baseline exhibited persistent degradation (>25%), continuing to serve stale responses. The adaptive mechanism introduces negligible computational overhead which is less than 1% of total cost, and can be scheduled during off-peak cycles, making it practical for continuous operation. This experiment confirms that DKC-LLM autonomously maintains content validity and reliability in dynamic, knowledge-evolving environments. C. COMPARISON WITH PRIOR WORK AND THEORETICAL IMPLICATIONS DKC-LLM advances the caching paradigm for retrieval-augmented generation by addressing gaps in prior systems such as RagCache, PipeRAG, CacheFocus, Pen-sieve, and CacheCraft. These approaches primarily optimize low-level operations (e.g., chunk caching, key-value reuse, prefetching) or treat caching as a static process, often lacking mechanisms to maintain knowledge freshness or semantic adaptation. In contrast, DKC-LLM leverages semantic similarity–based caching combined with adaptive updates driven by usage patterns and drift detection. Theoretically, this represents a shift toward adaptive knowledge-aware LLM serving, which is a framework that balances responsiveness, accuracy, and knowledge currency through proactive cache management. This approach not only improves efficiency but also enhances factual reliability. By serving pre-validated, semantically consistent responses, DKC-LLM implicitly enforces a form of verified grounding, offering an alternative to prompt engineering or post-hoc hallucination filters. This contributes to a broader understanding of how static and dynamic knowledge can be integrated within LLM architectures for long-term robustness. D. PRACTICAL IMPLICATIONS AND BUSINESS VALUE The practical benefits of DKC-LLM are substantial for enterprise-scale deployments: 1) OPERATIONAL COST REDUCTION. By reducing LLM inference operations by approximately 60%, DKC-LLM directly lowers computational and API expenses. 2) ENHANCED USER EXPERIENCE. Millisecond-level latency provides smoother, more respon-sive, and engaging interactions. 3) SCALABILITY. High cache hit ratios reduce backend load, enabling higher throughput without proportional increases in infrastructure requirements. 4) RELIABILITY AND CONSISTENCY. Reduced hallucination rates, coupled with automatic cache refresh, ensure factual stability and consistent performance. 5) DEPLOYMENT VERSATILITY. Demonstrated effectiveness across both domain-specific and open-domain datasets makes DKC-LLM adaptable to a wide range of applications, including enterprise search, knowledge management, and customer-facing assistants. Collectively, these advantages position DKC-LLM as a viable approach for cost efficient toward cost-efficient, responsive, and reliable AI-driven services in real-world business environments. E. LIMITATIONS 1) Despite its advantages, DKC-LLM has some limitations. During initial deployment, performance mirrors that of the baseline RAG pipeline until the cache is sufficiently popu-lated. For entirely novel or unseen domains, the framework reverts to the baseline RAG pipeline, incurring standard latency. 2) While semantic drift detection ensures knowledge. freshness, its effectiveness depends on the frequency of re-validation and the robustness of divergence detection. In rapidly changing domains, increased update frequency may overhead may raise computational overhead. Finally, reliance on a fixed cosine similarity threshold (0.85) intro-duces potential boundary cases, where subtle semantic dif-ferences may occasionally lead to misclassifications between cache hits and misses. VI. CONCLUSION. In this study, we introduced Dynamic Knowledge Caching (DKC-LLM), a framework designed to optimize information retrieval in read-heavy, business-centric environments by integrating semantic caching with generative models. DKC-LLM addresses key challenges in traditional Retrieval-Augmented Generation (RAG) pipelines, including high latency and computational overhead, while ensuring the continuous relevance and freshness of cached knowledge through an adaptive cache management mechanism. We evaluated DKC-LLM against two baselines: Naïve RAG and Vanilla Cache-RAG (VC-RAG), using both domain-specific (BankFAQs) and multi-hop reasoning (Hot-potQA) datasets. Experimental results demonstrate that DKC-LLM achieves significant reductions in query latency (5.3 ms on BankFAQs vs. 8.3 ms for RAG, 6.2 ms on Hot-potQA vs. 9.0 ms for RAG) and decreases LLM inference calls by approximately 60% due to a high cache hit rate (63–67%). Crucially, these efficiency gains are accompanied by improved accuracy (90–93%) and reduced hallucination rates (5%, representing a 70% improvement over RAG and 66% over VC-RAG), highlighting its ability to provide reliable and consistent responses. The core of DKC-LLM is its dynamically managed semantic cache, which intelligently reuses responses for semantically similar queries and adapts based on usage patterns and semantic drift detection. This mechanism directly addresses knowledge staleness in dynamic business contexts and establishes a more efficient and reliable paradigm for LLM deployment. Overall, DKC-LLM provides a cost-effective, low-latency, and high-accuracy solution for enterprise applications, involving repetitive queries and stringent real-time performance requirements. Sents a significant advancement toward scalable, economically viable, and robust AI systems, with immediate applicability in customer support, knowledge management, and other knowledge-intensive operations. VII. FUTURE WORK. Building on the strengths of DKC-LLM, several promising directions for future research are apparent: A. ADVANCED RAG INTEGRATION Integration with more sophisticated RAG variants, including multi-modal retrieval, conversational context understanding, or knowledge graph-based reasoning, could enable richer, context-aware responses and improved handling of multi-turn interactions. B. OPTIMIZED EMBEDDING MODELS AND SIMILARITY METRICS Incorporating domain-specific or computationally efficient transformer models for embeddings, and exploring hybrid similarity metrics that combine semantic embeddings with lexical or structural information, could enhance both cache accuracy and hit rates. C. ADAPTIVE CACHE POLICY LEARNING Developing intelligent, adaptive cache eviction and re-validation strategies using reinforcement learning or other machine learning techniques could dynamically optimize cache size, update frequency, and freshness checks, tailoring DKC-LLM to evolving workloads and volatile knowledge domains. D. EXTENDING BEYOND ANSWER CACHING Future work could explore caching intermediate RAG components, such as retrieved passages or re-ranking scores, to optimize sub-components of the pipeline and further reduce latency. E. SUSTAINABILITY AND RESOURCE MANAGEMENT Evaluating the energy efficiency of DKC-LLM and inves-tigating dynamic resource allocation strategies based on cache hit rates could improve both ecological and operational sustainability. F. CACHE SECURITY CONCERNS DKC-LLM may be vulnerable to cache poisoning attacks, where malicious queries compromise cached entries. Future research should investigate security measures such as prove-nance verification, trust scoring, and periodic revalidation to ensure cache integrity. Collectively, these avenues aim to balance knowledge currency, computational efficiency, and response speed, paving the way for more intelligent, robust, and economically sustainable LLM-powered system.