An LLM cache you can watch working
This is a proxy sitting in front of the OpenAI API. When a question has been asked
before, it retrieves the answer from a multi-tiered Redis cache instead of paying
for a new API call.
Every answer is tagged with the Redis cache tier that served it:
Tier 1
Exact match. The text matches a previous question perfectly. The answer
is retrieved instantly from the Redis string cache, bypassing the LLM entirely.
(Note: follow-up questions skip this tier because they rely on conversational
context.)
Tier 2
Semantic match. Different words, but identical meaning. Served from the
Redis vector cache. The response shows a distance metric indicating how close
the match is (0 is identical).
Miss
Nothing matched, so the question goes upstream to OpenAI. Takes a few
seconds.
How it works
- Prompt normalization. Tier 1 lowercases the text and collapses
whitespace before hashing it, ensuring messy formatting still hits the exact same
Redis key.
- Vector search. Tier 2 embeds the question and runs a nearest-neighbor
search against the Redis vector index, accepting anything within a 0.15 cosine
distance.
- Query rewriting. The proxy uses chat history to turn vague follow-ups
into complete standalone questions. For example, “How does it work?” is
rewritten into “How does Redis work?” before it is embedded, so answers
aren't mixed up between different topics.