Summary of Semantic Caching Implementation and Results
This document details the implementation and evaluation of a semantic caching system designed to reduce LLM API costs and improve response times. Here’s a breakdown of the key aspects:
Problem:
* Naive caching based on exact string matches is insufficient for LLM applications due to paraphrasing and variations in user queries.
* A single similarity threshold for all query types is suboptimal, leading to either missed cache hits or incorrect responses.
Solution:
* Semantic Embedding: Uses an embedding model to represent queries as vectors,allowing for similarity comparisons based on meaning rather than exact text.
* Vector Store: Stores query embeddings and associated responses for efficient similarity search.
* Query-type Specific Thresholds: Dynamically adjusts the similarity threshold based on the type of query (FAQ, Product Search, Support, Transactional) to balance precision and recall.
* Query Classification: Employs a QueryClassifier to determine the query type.
* Cache Invalidation: Implements both time-based TTL and event-based invalidation to ensure cached responses remain accurate.
Key findings & Results:
* Threshold Tuning: Optimal thresholds were persistent thru a rigorous methodology involving:
* Sampling query pairs with varying similarity scores.
* Human labeling of query pairs as “same intent” or “different intent.”
* Calculation of precision/recall curves for different thresholds.
* Selection of thresholds based on the cost of errors for each query type.
* Threshold Values:
* FAQ: 0.94 (High precision)
* Product Search: 0.88 (High recall)
* Support: 0.92 (Balanced)
* Transactional: 0.97 (Very high precision)
* Latency:
* Cache lookup overhead: 20ms (p50), 47ms (p99)
* LLM API call: 850ms (p50), 2400ms (p99)
* The overhead is negligible compared to the time saved by avoiding LLM calls.
* Performance Enhancement:
* 65% net latency improvement.
* Significant cost reduction due to fewer LLM API calls.
* Cache Invalidation Strategies:
* Time-Based TTL: Uses different expiration times based on content type (e.g., 4 hours for pricing, 14 days for general FAQs).
* Event-Based Invalidation: invalidates cache entries when underlying data changes, triggered by a CacheInvalidator class and on_content_update method.
Components:
* AdaptiveSemanticCache class: manages thresholds, query classification, and cache retrieval.
* QueryClassifier class: Classifies the type of query.
* embedding_model: Used to encode queries into vector embeddings.
* vector_store: Stores query embeddings and associated response IDs.
* response_store: Stores the actual cached responses.
* CacheInvalidator class: Handles event-based cache invalidation.
* compute_precision_recall function: Calculates precision and recall for threshold evaluation.
the implementation demonstrates a accomplished approach to semantic caching, considerably improving performance and reducing costs while maintaining accuracy through careful threshold tuning and robust cache invalidation strategies.