Long Term Memory

A key aspect of building lifelike and engaging AI characters is their ability to relate with the end users and form relationships over time. Addressing this challenge, Convai is introducing a new feature: Long Term Memory (LTM) for characters. With LTM enabled, characters can now remember interactions with users, build memories from them that can be referred to in future conversations, and develop and evolve their own personalities over time. 

‍

Challenges with existing LLM systems

• Limited Memory Span: Traditional LLMs can only manage a brief history of interactions within a single prompt, causing them to forget information over time and exhibit short-term memory traits.
• Information Loss: Even when the LLM context is large enough to hold a sizable amount of conversation history, they often suffer from selective recollection depending on where the information is present in the prompt.
• Benchmark Limitations: Some advanced LLMs with very large context lengths use "needle-in-haystack" searches, in which random UUIDs or other "needle" information is placed randomly within the context and the model is asked to find the value. The effectiveness of these models in practical scenarios, like remembering past conversation contexts, is often not reflected in benchmark assessments.

Beyond Basic Retrieval: The Nuances of Memory

To supplement the prompts with relevant information, a common approach is the use of Retrieval Augmented Generation systems powered by vector databases. However memories are more nuanced than just referencing old information. Some of the factors that make a memory different from just information are:

• Temporal Influence: Like human memory, AI characters' recollections are subject to recency bias, with newer interactions remembered more vividly than older ones.
• Emotional Significance: Memories with emotional or significant impacts are prioritized, influencing the character's personality development.

Learned Behaviors: Through repeated interactions, AI characters develop and incorporate behaviors into their subconscious, simulating the human learning process.

‍

Addressing Memory and Privacy Concerns

An AI character can interact with many different users in different scenarios. So the natural question that may arise is: "Can the AI character reveal my information from its memories to someone else?". 

To safeguard user privacy while enabling rich, personalized interactions, we've implemented a memory tree system. This system ensures that memories are tied to unique speaker profiles, keeping interactions compartmentalized and private. This will allow developers building with Convai to manage individual memory profiles and offer privacy controls to their end users.

‍

Memory Architecture

Convai's LTM utilizes a hybrid approach, combining Retrieval Augmented Generation (RAG) with a custom ranking system. This architecture accounts for factors like recency and emotional impact, enhancing memory relevance. Here's how it works:

• Request Processing: Upon authentication, the API server forwards the user's request (audio / text) to the RAG processor.
• Memory Retrieval and Ranking: Using the speaker_id in the user's request, the RAG processor accesses the memory tree to retrieve memories across three dimensions—recent, long-term, and latent. These memories are then ranked based on their relevance.
• Prompt Construction: The selected memories are compiled into a prompt, enabling AI characters to respond in a more personalized manner.

Memory Update: Upon interaction completion, it is fed back into the character's memory tree via the RAG processor, forming new memories or enriching older ones.

‍

Memory Management

We are also building a set of tools and controls that will make it easy for developers to control the memory trees of different users / speakers.

• Memory Tree Control: Using the speaker_id, developers have the ability to build separate memory trees for different users. They can also address requests to purge / delete memory trees based on user requests.
• Memory Versioning: Developers will also have the ability to backup and restore memory trees up to a specific point to correct unintentional changes.
  ‍

Conclusion: The Future of Conversational AI

Convai's Long Term Memory feature is set to revolutionize user engagement with AI characters. Though still in its early stages, LTM has already shown promising results in enhancing conversational coherence and depth. We're excited

‍

‍