Holographic Memory for an AI Agent on a Potato VPS

I have a so-called Potato VPS — a cheap server with minimal RAM. It runs an AI agent — Hermes — that needs to remember context between sessions: facts about users, project configurations, preferences, decisions. Not just "write to a file and grep it," but semantic search that understands paraphrasing and multilingualism.

The problem: ChromaDB consumes 400 MB just at startup. Pinecone is SaaS, and I want local. FAISS lacks keyword indexing. I need a hybrid: full-text search + vector semantics + compositional algebra. All of this on a modest server, including the embedding model itself.

The solution is the holographic-memory plugin — four search strategies in a single SQLite file. Here's how it works and why.

Architecture: Four Strategies, One Query

Hybrid scoring isn't "vector search with FTS fallback" — it's four independent channels whose results are combined with weights:

Final score: relevance × trust_score × temporal_decay, where relevance = fts×0.3 + jaccard×0.2 + hrr×0.2 + semantic×0.3.

Why four when you could get by with just vector search? Because vector search performs poorly on short, precise queries ("nginx deployment order"), while FTS5 doesn't understand paraphrasing ("how to roll out nginx to prod"). HRR provides algebraic operations — probing by entity, finding connections between facts, multi-entity JOINs. Jaccard is a cheap noise filter.

Search Pipeline

1. FTS5 MATCH → limit×3 candidates (AND semantics: all terms required)
2. If FTS5 returns empty + semantic available → _semantic_candidates() (full cosine scan)
3. Pre-compute query embedding (~46 ms)
4. Reranking: relevance = fts×0.3 + jaccard×0.2 + hrr×0.2 + semantic×0.3
5. Final: score = relevance × trust × temporal_decay

Key point: if FTS5 returns 0 (query is in Russian while facts were recorded in English, or it's a paraphrase), the pipeline automatically falls back to pure semantic search. Semantic isn't a luxury — it's a load-bearing component for multilingual support.

Why Not Off-the-Shelf Solutions

ChromaDB

Two processes (Chroma + Hermes), ~800 MB before the agent has even remembered anything. On a Potato VPS, that's half the RAM. Plus Chroma pulls in HNSW, which builds its index in memory. For facts (hundreds, maybe thousands of records) — it's using a cannon to kill a mosquito.

Pinecone

SaaS. Requires an API key, internet access, and trusting a third party with your agent's data. For a hobby project on a cheap VPS — overkill and vendor lock-in.

FAISS

Excellent library for vector search, but no full-text indexing. You'd have to bolt FTS5 on top separately. And if you're already using SQLite for both text and vectors — why bother with FAISS?

Custom Solution

SQLite + FTS5 + WAL is already in Python's stdlib (sqlite3). Add fastembed for embeddings and numpy for HRR algebra. One database file, one process, zero infrastructure.

Model Selection: mpnet vs MiniLM

I tested two multilingual fastembed models:

MiniLM: 680 MB RSS, 481 MB residual. mpnet: 1440 MB RSS, 693 MB residual. On a Potato VPS, mpnet doesn't fit — after loading the model + Hermes + the OS, only ~300 MB remains, and the OOM killer comes knocking.

MiniLM scores 0.511 for semantically similar phrases and 0.008 for unrelated ones — sufficient separation. Not ideal (mpnet is ~20% better), but functional.

Residual refers to ONNX Runtime, which doesn't release memory pools even after del model + gc.collect(). 481 MB is the price of a single fastembed invocation during the process lifetime. Hence the strategy: lazy-load on first request, keep in memory until shutdown.

Lazy-Loading and Model Lifecycle

_model = None
_available: Optional[bool] = None

def is_available() -> bool:
    """Check without loading the model."""
    if _available is not None:
        return _available
    try:
        import fastembed
        _available = True
    except ImportError:
        _available = False
    return _available

def _get_model():
    """Lazy-load on first embed_text() call."""
    global _model
    if _model is None:
        from fastembed import TextEmbedding
        _model = TextEmbedding("sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2")
    return _model

def embed_text(text: str) -> np.ndarray:
    model = _get_model()
    vec = list(model.embed([text]))[0]
    return vec / np.linalg.norm(vec)  # normalize → unit vector

First embed_text() call takes ~1.3 s (loading the ONNX model). All subsequent calls take ~46 ms. The model stays in memory until shutdown.

Unloading on Shutdown

# In the plugin's __init__.py, on shutdown hook:
import embedder, gc

def on_shutdown():
    embedder._model = None
    gc.collect()
    # Frees ~200 MB (model weights), but ONNX residual (481 MB) remains

In practice: after shutdown, RSS drops from ~1.1 GB to ~620 MB. ONNX Runtime doesn't give back memory — this is a known quirk. 620 MB is a workable baseline for a Potato VPS.

Storage: Two Vectors Per Fact

Each fact in SQLite stores two vectors:

CREATE TABLE facts (
    fact_id INTEGER PRIMARY KEY,
    content TEXT NOT NULL,
    category TEXT DEFAULT 'general',
    tags TEXT DEFAULT '',
    trust_score REAL DEFAULT 0.5,
    retrieval_count INTEGER DEFAULT 0,
    helpful_count INTEGER DEFAULT 0,
    hrr_vector BLOB,        -- 1024 × float64 = 8192 bytes
    semantic_vector BLOB,   -- 384 × float32 = 1536 bytes
    created_at TEXT,
    updated_at TEXT
);

HRR vector (8 KB per fact) — SHA-256 phase encoding. Content tokens → atom bundle → bind with ROLE_CONTENT and ROLE_ENTITY. Used for algebraic operations: probe ("all facts about X"), related ("what's connected to X"), reason ("what do X, Y, Z have in common"). Runs on numpy, no model needed.

Semantic vector (1.5 KB per fact) — fastembed output. Normalized unit vector for cosine similarity. Used in hybrid scoring.

Total: ~9.7 KB per fact. For 1000 facts — ~10 MB. Negligible.

HRR: Compositional Algebra Without Neural Networks

HRR (Holographic Reduced Representations) is a way to encode structure into a fixed-size vector. Instead of training — SHA-256 hashes of tokens converted into phase vectors.

def encode_atom(token: str, dim: int = 1024) -> np.ndarray:
    """Token → unit vector in phase space."""
    h = hashlib.sha256(token.encode()).digest()
    rng = np.frombuffer(h, dtype=np.uint8).astype(np.float64)
    # Phase code: cos + j*sin → unit vector in complex space
    phases = rng[:dim] / 255.0 * 2 * np.pi
    return np.cos(phases) + 1j * np.sin(phases)

def bind(a, b):
    """Binding: circular convolution in frequency domain."""
    return np.fft.ifft(np.fft.fft(a) * np.fft.fft(b))

def bundle(vectors):
    """Bundling: element-wise sum + normalization."""
    result = np.sum(vectors, axis=0)
    return result / np.linalg.norm(result)

Why bother when you have fastembed? Because HRR supports algebraic operations that vector models can't do:

• probe(entity) — "give me all facts bound to entity X." Bind/unbind with the entity atom.
• related(entity) — "what structurally neighbors X." Via HRR similarity.
• reason(e1, e2, e3) — "what do multiple entities have in common." Multi-entity JOIN, min(scores).

The semantic model gives "similarity in meaning," HRR gives "connectedness by structure." These are different axes.

Jaccard: A Cheap Filter

Jaccard is the intersection of query and fact tokens divided by their union. Cost: O(n) over tokens, zero memory allocations. Works as a coarse filter: if the query and fact share no tokens — apply a penalty.

def jaccard(query_tokens: set, fact_tokens: set) -> float:
    if not query_tokens or not fact_tokens:
        return 0.0
    return len(query_tokens & fact_tokens) / len(query_tokens | fact_tokens)

Weight of 0.2 — not the primary channel, but cuts through noise. In practice: query "deploy nginx" and fact "DNS configuration" get jaccard=0, and rightly so.

FTS5: Full-Text Indexing

SQLite FTS5 is built-in full-text indexing. AND semantics: all query words must appear in the fact. Strict, but predictable.

CREATE VIRTUAL TABLE facts_fts USING fts5(content, content=facts, content_rowid=fact_id);

-- Search:
SELECT fact_id, rank FROM facts_fts WHERE facts_fts MATCH 'deploy nginx'
ORDER BY rank LIMIT 30;

The problem with FTS5: it doesn't understand paraphrasing. "How to roll out nginx to prod" won't match "nginx deployment via CI/CD." That's why falling back to semantic search when FTS5 returns empty is critical.

Resource Usage on a Potato VPS

Typical memory footprint with the agent running:

Plenty left for the OS, swap, and other processes. Comfortable.

On shutdown, the model is unloaded and RSS drops to ~620 MB. ONNX residual isn't freed — that's the cost of a single import fastembed per process lifetime.

Weight Auto-Redistribution

If fastembed is unavailable (not installed, or numpy missing), weights are redistributed automatically:

def _redistribute_weights(self):
    if not embedder.is_available():
        # Semantic unavailable: 0.3 → FTS +0.15, Jaccard +0.1, HRR +0.05
        self.fts_weight = 0.45
        self.jaccard_weight = 0.30
        self.hrr_weight = 0.25
        self.semantic_weight = 0.0
    elif not _HAS_NUMPY:
        # HRR unavailable: 0.2 → FTS +0.1, Semantic +0.1
        self.fts_weight = 0.40
        self.jaccard_weight = 0.20
        self.hrr_weight = 0.0
        self.semantic_weight = 0.40

Graceful degradation: the system works without embeddings (FTS + Jaccard) and without HRR (FTS + Jaccard + Semantic). But the full quartet is optimal.

Practical Pitfalls

1. FTS5 AND Is Too Strict

Query "compact message format" requires all three words present. If the fact was recorded as "concise responses" — FTS5 stays silent. Semantic saves the day, but only if the model is loaded.

Solution: Don't rely on FTS5 as the sole channel. Always keep semantic enabled.

2. Missing Semantic Vectors After Update

The agent was updated, but the old process is still running. New facts are written without semantic_vector. Symptom: Russian queries return empty results.

SELECT COUNT(*) FROM facts WHERE semantic_vector IS NULL;

If > 0 — run a backfill:

import embedder, sqlite3

conn = sqlite3.connect(db_path)  # path to your database
rows = conn.execute('SELECT fact_id, content FROM facts WHERE semantic_vector IS NULL').fetchall()

for fid, content in rows:
    vec = embedder.embed_text(content)
    conn.execute('UPDATE facts SET semantic_vector = ? WHERE fact_id = ?',
                 (embedder.vector_to_bytes(vec), fid))

conn.commit()

3. fastembed Pooling Change

Version 0.8.0+ changes pooling from CLS to mean. Old vectors are incompatible with new ones. Solution: full re-embedding of all facts after upgrading fastembed.

4. ONNX Memory Leak (It's Not a Leak)

del model + gc.collect() frees the weights but not the ONNX Runtime pools. This isn't a leak — it's how ONNX works. 481 MB residual is normal. Don't try to "fix" it.

5. RSS ≠ Used Memory

ru_maxrss shows the peak, not current consumption. For accurate measurement:

def current_rss_mb():
    with open('/proc/self/statm') as f:
        pages = int(f.read().split()[1])
    return pages * 4096 / 1024 / 1024

Web Interface for Viewing Facts

For debugging, I built a standalone htmx app on stdlib's http.server. Dark theme, monospace font, FTS5 search, inline editing, feedback buttons, color-coded category badges. Zero dependencies — just Python stdlib. Launches with a single command, listens on a local port.

Summary

Four search strategies in a single SQLite file. 680 MB RSS with the model loaded. Lazy-loading, graceful degradation, unloading on shutdown. No external services, no docker-compose, no Pinecone API keys.

For a hobby project on a Potato VPS — this is the only sensible option. Not because it's "better than ChromaDB," but because ChromaDB doesn't fit in modest RAM, and Pinecone is someone else's computer.

A custom SQLite plugin means control. Control over memory, over indexing, over the model lifecycle. And when the OOM killer comes knocking at 3 AM — you know exactly who's to blame and what to do.

The holographic-memory plugin is part of the Hermes Agent project.