AI Agent Memory: When to Use Profiles, History or Vector Search

An AI application’s memory is the state its software stores and later makes available to the model. A vector database is one way to retrieve that state, not a mandatory component and not a replacement for accurate records, access controls or backups.

Reviewed October 14, 2026: This article replaces blanket claims about model memory, arbitrary vector dimensions and unverified VPS capacity. It focuses on choosing and validating a memory design; runnable database examples are in the linked labs.

Separate the model from the application

A model can use the context supplied to a request. The surrounding application may keep conversation history, summaries, preferences or other persistent state and supply it in later requests. It is therefore misleading to say that every AI product necessarily starts from zero after each conversation.

Storing a fact in a database does not retrain the model. The application still needs a deliberate retrieval or context-building step. Letta’s stateful-agent documentation illustrates one implementation with persisted state and memory blocks; other frameworks make different choices.

Choose storage based on the question

Need	Useful starting point	Important check
Continue the current conversation	Ordered messages and a bounded context window	Preserve important details when summarizing or trimming.
Remember an explicit preference	A structured profile field	Let the user update or remove it; do not infer it from unrelated text.
Find an exact account or ticket	Authorized structured lookup	Use the correct account scope and current system of record.
Find related passages across documents	Keyword, vector or hybrid retrieval	Evaluate relevant matches and plausible-but-wrong matches.
Recover after an outage	Durable storage and tested backups	Demonstrate restoration, not just a successful write.

For example, a support assistant should look up an invoice’s status in the authorized billing system rather than retrieve an old chat that mentioned it. A response-length preference usually needs a profile field, not similarity search. Vector retrieval is more appropriate when the question is which documentation passages discuss a symptom in different words.

What vector memory adds

An embedding model maps text to numerical vectors; retrieval compares a query vector with stored vectors. The embedding model determines the dimensions and input limits. A made-up vector close to another made-up vector only tests database behavior, not whether two sentences have related meanings.

A typical document flow is: identify authorized sources, split them into suitable passages, embed them, store source metadata, retrieve within the user’s permitted scope, then build a bounded prompt. The answer still needs to be checked against the source. Similarity is not certainty, and a top result exists even when the corpus lacks the answer.

Design remembering and forgetting together

• Ownership: Derive the user or tenant identity from authenticated application state, not the prompt. Enforce it on writes, queries, updates and deletions.
• Provenance: Keep the source, version and update time so a retrieved claim can be checked and stale entries replaced.
• Purpose: Store only information needed for the feature. Avoid passwords, API tokens and unnecessary personal data.
• Expiry: Exclude expired entries from retrieval and separately remove them according to the retention policy. A query filter alone does not physically delete data.
• Correction: Updating a preference should replace the old value, not leave two contradictory memories competing for retrieval.
• Deletion: Account for source records, vector payloads, caches and backups. Document what is deleted immediately and what expires later.

Retrieved text is untrusted input. A document that tells an agent to ignore its rules should not gain authority just because it came from the memory store. Apply the controls discussed in OWASP’s prompt-injection guidance, especially when the agent can invoke tools.

Acceptance tests before adding real customer data

1. Create synthetic users Alice and Bob with deliberately conflicting preferences. Alice’s query must not return Bob’s data.
2. Update Alice’s preference and verify that retrieval does not return the obsolete value.
3. Expire and delete test entries, then verify the intended retrieval and deletion behavior.
4. Restart the service and verify persistence. Separately test restoring a backup.
5. Ask questions that the stored material cannot answer. Check that the application can decline instead of inventing a response.
6. Test a retrieved passage containing hostile instructions. Ensure it cannot authorize privileged actions.

Our memory-store implementation lab tests ownership, updates, expiry, deletion and restart persistence with synthetic data. It does not claim to solve authentication or model behavior. The document retrieval lab separately tests a real local embedding model.

Hosting considerations

Measure memory, storage growth, indexing time and query latency for your dataset before selecting a VPS. Add the generation model’s resource needs separately. Self-hosting the database does not keep data local if the application sends excerpts to an external model, logging system or connector.

Keep the database private, restrict application credentials, monitor failed jobs and plan recovery. For connecting an assistant to InterServer account tools, rather than implementing memory, use the InterServer MCP Server documentation.