Skip to content

Intelligent optimization

Phase 6 adds three opt-in helpers that make the cache smarter and safer. All are additive — leave them at their defaults and the cache behaves exactly as before, including deriving byte-identical keys so existing cached entries keep hitting.

Single-flight (stampede protection)

When several concurrent requests miss the same key at once, executing the expensive underlying work once and sharing the result — instead of every caller paying — is a pure reliability and cost win. MemoryCache.get_or_compute provides that coalescing.

from memory_reuse import CacheConfig, MemoryCache

cache = MemoryCache(CacheConfig(single_flight=True))

async def answer(question: str) -> str:
    return await cache.get_or_compute(
        ["qa", question],
        compute=lambda: call_llm(question),   # runs once for a concurrent burst
        scope="global", scope_id=None, ttl=3600,
    )
  • A burst of concurrent misses for the same key runs compute once; every waiter receives that one result, which is stored before returning.
  • A key that already has a stored value never runs compute.
  • If compute raises, the exception propagates to all waiters, nothing is stored, and the key is cleared so a later call retries.

Distributed lock (across processes / microVMs)

For several processes sharing one Redis backend, add distributed_lock=True to coalesce across processes too:

cache = MemoryCache(CacheConfig(
    backend="redis", redis_url=..., single_flight=True, distributed_lock=True,
))

The distributed lock is strictly fail-open: a lock failure or timeout degrades to running the compute rather than blocking or raising, and the lock carries an expiry so a crashed holder cannot deadlock other processes. Enabling distributed_lock without a Redis backend raises a configuration error at construction.

Version-aware cache identity

Fold a version token into every cache key so a redeploy with new prompts, models, or tools never serves a result generated by incompatible older logic:

cache = MemoryCache(CacheConfig(cache_version="agent-v2"))
  • With cache_version unset (the default), keys are byte-identical to a pre-Phase-6 config, so existing cached entries keep hitting after upgrading.
  • A version change invalidates reuse uniformly across the exact, tool, and semantic layers — including the semantic vector namespace, so a version-A vector is never returned as a match for a version-B query.
  • Two caches on the same version and inputs derive the same key, so a shared backend is reused correctly across processes on the same version.

Effectiveness analyzer + recommendations

Record per-operation hit/miss observations, then ask for actionable, advisory recommendations — what to cache, with what settings, and what not to cache.

# On each call, tell the analyzer what happened:
cache.record_operation("search_confluence", hit=False, input_hash=args_hash)
cache.record_operation("get_stock_price", hit=False, input_hash=sym, volatile=True)

report = cache.analyze()
for rec in report.recommendations:
    print(rec.operation, "->", rec.action, "—", rec.reason)

Example output:

search_confluence -> enable_cache   — 73% of calls repeat identical inputs...
get_stock_price   -> avoid_semantic — data changes; similarity != equivalence

The rules are deterministic and threshold-driven:

Signal Recommendation
High call count + high repeat ratio, not yet cached enable_cache
Already cached but hit rate below the low threshold increase_ttl
Operation flagged volatile avoid_semantic
Fewer than min_calls observations nothing emitted

Two important properties:

  • Advisory only. Producing or reading recommendations never changes any cache behaviour, TTL, scope, or key — you act on them yourself.
  • Honest signals. The cached and volatile flags are caller-supplied, because the analyzer cannot infer a tool's caching status or data-freshness on its own. This keeps recommendations grounded rather than guessed.

Recording is best-effort (a failure never breaks a cache operation) and bounded (max_tracked_operations caps how many distinct operation aggregates are kept), and it never touches the hit/miss counters or the Phase 5 analytics totals.

Faithful result serialization (zero-config)

By default the cache stores values as JSON, which stringifies non-JSON objects. That matters most for wrap_graph over a LangChain/LangGraph agent: the final state contains message objects, so a naive cache hit would replay strings while a miss returns message objects — breaking result["messages"][-1].content on a hit.

wrap_graph handles this automatically — no setup required. When langchain-core is installed, a wrapped graph transparently round-trips LangChain message objects, so a hit returns the same object shapes as a miss:

from memory_reuse import MemoryCache

cache = MemoryCache()
cached_agent = cache.wrap_graph(agent, scope="user", key_fields=["messages"])
# A cache hit returns real LangChain message objects — plug-and-play.
  • This is the default (serialize_messages="auto"): the codec is applied when langchain-core is importable and silently skipped when it is not (no hard dependency, existing behaviour preserved).
  • Pass serialize_messages=False to opt out, or serialize_messages=True to require it (raising a clear error naming memory-reuse[langchain] if the dependency is missing).
  • The codec applies only to that wrapped graph's store/load — it never changes serialization for the cache's exact/tool/semantic layers.

Manual hooks (any framework)

For non-graph caching or other frameworks, configure a (serializer, deserializer) pair directly on the cache. An explicit configured pair always wins over wrap_graph's automatic codec:

from memory_reuse import CacheConfig, MemoryCache
from memory_reuse.integrations.langchain_serde import langchain_message_codec

serializer, deserializer = langchain_message_codec()   # pip install "memory-reuse[langchain]"
cache = MemoryCache(CacheConfig(serializer=serializer, deserializer=deserializer))
  • With both hooks unset (the default), stored bytes are byte-identical to before — existing entries and values are unaffected.
  • The hooks apply uniformly across the exact, tool, and semantic layers.
  • The hooks are generic: supply your own pair for any framework object.

Packaging

The single-flight, version identity, analyzer, and generic serialization hooks are standard library only; the optional distributed lock reuses the existing redis extra. The only new extra is langchain (for the optional message codec), folded into all.