Memory System

​

Memory Architecture

​

User Memory (Personal)

​

Collective Memory (Shared)

​

Memory Components

​

1. Episodes

• What: Summaries of conversation segments that capture key events, decisions, and context
• Purpose: Provide conversational continuity within a session
• Scope: Session-specific
• Max stored: 8 most recent episodes per session
• Example: “User requested a story about a clockmaker and provided feedback that they preferred shorter paragraphs”

​

2. Directives (User Memory)

• What: Persistent user preferences and behavioral instructions
• Purpose: Customize agent behavior for individual users
• Scope: User-specific, persists across all sessions
• Consolidation: Automatically merged when 5+ directives accumulate (production) or 3+ (development)
• Example: “Always ask Jack for permission before generating videos”

​

3. Facts (Collective Memory)

• What: Atomic, verifiable, unchanging information relevant to the collective memory context
• Purpose: Build a permanent knowledge base
• Scope: Shared across all users
• Storage: FIFO queue with max 50 facts per memory shard (production)
• Example: “The project deadline is May 1st (source: Alice)” or “Maximum budget is $1000 (source: Bob)“

​

4. Suggestions (Collective Memory)

• What: New ideas, proposals, or context updates for the collective memory
• Purpose: Allow the collective memory to evolve based on community input
• Scope: Shared across all users
• Consolidation: Automatically integrated when 16+ suggestions accumulate (production)
• Example: “Bob proposed adding a feature that allows users to export data in CSV format”

​

Memory Formation Flow

​

Automatic Trigger

• Message threshold: 45+ messages since last memory formation (production)
• Token threshold: 1000+ tokens since last memory formation (production)

​

Extraction Process

​

Consolidation Process

1. Load all unabsorbed directive memories
2. Use LLM to merge them with existing consolidated user memory
3. Remove redundancies and contradictions (newer overrides older)
4. Update the fully formed user memory
5. Clear unabsorbed directives

1. Load all unabsorbed suggestions and current facts
2. Use LLM to integrate suggestions into the consolidated memory blob
3. Maintain structure and preserve existing information
4. Update version number
5. Generate fully formed memory shard

​

Customizing Memory Behavior

​

Memory Instructions

• What information to pay attention to
• What types of facts should be remembered
• What kinds of suggestions are relevant
• The overall purpose and context of this memory shard

This collective memory is for tracking the development of our community art project "Digital Dreams".

Important to remember:
- Artist contributions and their specialties
- Technical requirements and constraints
- Agreed-upon artistic direction and themes
- Deadlines and milestone dates
- Budget allocations
- Community feedback and preferences

Focus on facts that won't change (like budget limits, deadlines) and suggestions that help guide the project's evolution (like new theme ideas, workflow improvements).

​

User Interface Guide

​

Managing User Memory

• The main text area shows your consolidated user memory
• Below that, “Recent Memory Context” shows new directives awaiting consolidation
• These recent directives will be automatically merged into your main memory

1. Click the edit button (appears on hover)
2. Modify the memory text in the modal
3. Confirm your changes

• Click edit to modify a directive before consolidation
• Click delete to remove unwanted directives
• These will automatically merge into your main memory when the threshold is reached

​

Managing Collective Memory

1. Click the edit button in the Memory Instructions section
2. Update the prompt to better describe what should be remembered
3. Save your changes
4. This affects future memory extractions immediately

• The Memory section shows the main consolidated collective memory
• This evolves automatically as suggestions are consolidated
• Version number indicates how many consolidation cycles have occurred

• Facts are shown with their age (e.g., “5 days old”)
• Click edit to modify a fact
• Click delete to remove irrelevant facts
• Facts follow FIFO (first in, first out) - oldest facts are automatically removed when the limit is reached

• Shows new suggestions awaiting consolidation
• Tagged with “Will be integrated soon”
• Click edit to refine suggestions before consolidation
• Click delete to remove off-topic suggestions
• These automatically consolidate when threshold is reached

• Use the toggle switch to activate or pause a memory shard
• Paused shards don’t extract new memories but retain existing content
• Useful for temporary memory contexts or testing

​

Use Cases

​

Personal AI Assistants

​

Collaborative Projects

​

Community Agents

​

Creative Workflows

​

Memory Context Assembly

1. Load the fully formed user memory (consolidated + recent directives)
2. Include username and memory age metadata
3. Format as XML structure

1. Load all active memory shards
2. For each shard, include:
   • Shard name and Memory Instructions
   • Current facts with age information
   • Consolidated memory content
   • Recent suggestions awaiting consolidation
3. Format as hierarchical XML

1. Load up to 8 most recent episodes from the current session
2. Order chronologically (oldest to newest)
3. Format as conversation context

<memory_contents description="Your complete memory context for this conversation">

  <collective_memory description="Shared memory across all your conversations">
    <memory_shard name="project_name">
      ## Shard facts:
      [Facts with age]

      ## Current consolidated shard memory:
      [Consolidated memory content]

      ## Recent shard suggestions:
      [Unabsorbed suggestions]
    </memory_shard>
  </collective_memory>

  <user_memory description="Memory and context specific to this user">
    [Consolidated user memory]

    ## Recent user directives:
    [Unabsorbed directives]
  </user_memory>

  <current_conversation_context description="Recent exchanges from this conversation">
    - [Episode 1]
    - [Episode 2]
    - [Episode N]
  </current_conversation_context>

</memory_contents>

​

Advanced Configuration

​

Memory Shard Settings

• Maximum number of facts to store in FIFO queue
• When limit is reached, oldest facts are removed
• Can be customized per shard

• Number of suggestions before automatic consolidation
• Higher values mean less frequent but larger consolidations
• Can be customized per shard

​

Memory Formation Timing

• 45+ messages have been exchanged (production) OR
• 1000+ tokens worth of conversation has occurred (production)

• User messages: 1.0x weight (highest priority)
• Tool results: 0.5x weight
• Agent messages: 0.2x weight (memories should come from users, not agents)

​

Token Limits

• Episode: 50 words target
• Directive: 25 words target
• Suggestion: 35 words target
• Fact: 30 words target
• User Memory Blob: 250 words target
• Agent Memory Blob: 750 words target

​

Best Practices

​

For User Memory

1. Be clear and specific when giving the agent instructions
2. Phrase directives as rules rather than requests (e.g., “Always X” instead of “Could you X?”)
3. Review recent directives periodically to catch unwanted extractions
4. Keep consolidated memory focused on truly persistent preferences

​

For Collective Memory

1. Start simple - Write initial Memory Instructions that are clear and narrow in scope
2. Iterate frequently - Refine Memory Instructions based on what gets extracted
3. Remove bad facts - Clean up irrelevant facts regularly to maintain quality
4. Review suggestions before consolidation - Edit or delete suggestions that don’t align with your goals
5. Monitor memory version - Track how often consolidation occurs and adjust thresholds if needed
6. Use specific language in Memory Instructions about what qualifies as a fact vs suggestion
7. Provide examples in Memory Instructions of the types of information you want

​

General Tips

1. Let the system work - Give it time to learn from multiple conversations
2. Trust the consolidation process - Manual edits should be rare
3. Use descriptive names for memory shards
4. Pause unused shards rather than deleting them
5. Monitor memory growth - If memories grow too large, refine extraction criteria

​

Troubleshooting

• Solution: Make Memory Instructions more specific about what’s relevant
• Solution: Review and delete bad memories before consolidation
• Solution: Reduce max_items in memory type configuration (requires system access)

• Solution: Explicitly mention in Memory Instructions what types of information matter
• Solution: Ensure users are providing information clearly in conversations
• Solution: Check that the memory shard is active (not paused)

• Solution: Manually edit to reduce length
• Solution: Adjust word count targets in Memory Instructions
• Solution: More frequently review and remove outdated information

• Solution: Check that enough suggestions/directives have accumulated
• Solution: Verify the memory shard is active
• Solution: Check system logs for consolidation errors

• Solution: Create separate memory shards for different topics
• Solution: Make Memory Instructions more focused and specific
• Solution: Use descriptive shard names that clearly indicate purpose

​

API Reference

​

User Memory Endpoints

• GET /api/agents/{agent_id}/memory - Get user memory
• POST /api/agents/{agent_id}/memory - Update user memory
• PATCH /api/agents/{agent_id}/user-memory-enabled - Toggle user memory on/off
• DELETE /api/agents/{agent_id}/memory/{directive_id} - Delete a directive

​

Collective Memory Endpoints

• GET /api/agents/{agent_id}/collective-memory - Get all memory shards
• POST /api/agents/{agent_id}/collective-memory - Create a new memory shard
• PATCH /api/agents/{agent_id}/collective-memory/{shard_id} - Update shard settings
• DELETE /api/agents/{agent_id}/collective-memory/{shard_id}/memory/{memory_id} - Delete a fact or suggestion

​

Memory Sessions

• PATCH /api/memory-sessions/{memory_id} - Edit individual memory content

​

Technical Details

​

Models Used

• Fast model (gpt-4o-mini): Episode, directive, and user memory consolidation
• Slow model (gpt-4o): Fact, suggestion extraction, and collective memory consolidation

​

Structured Output

• Max length constraints enforced per memory type
• Dynamic limits based on conversation length
• Pydantic models ensure type safety

​

Traceability

• Source session ID
• Source message IDs (which messages contributed to this memory)
• Related user IDs (who was involved)
• Timestamp (when it was created)
• For collective memories: Shard ID (which memory shard it belongs to)