Memory Health

The Memory Health panel provides operator-level insights into storage efficiency, data quality, and compliance with your organization's retention policies.

Health Metrics

Deduplication Ratio

Definition: Percentage of potential duplicate observations that were merged into a single record.

Dedup Ratio = (Duplicates Merged) / (Total Observations Processed) × 100%

Interpretation:

• High ratio (>30%) - Agents are learning similar patterns multiple times; good deduplication is working
• Low ratio (<5%) - Either agents learn very diverse patterns, or dedup is disabled
• Sudden spike - May indicate a feedback loop where the same observation is being re-generated

Why it matters:

• High dedup saves storage and improves retrieval performance
• Low dedup with high contradiction rate may indicate inconsistent learning
• Dedup ratio helps estimate the true "knowledge diversity" of your system

Retention Compliance

Definition: Percentage of observations within your organization's retention window.

Compliance = (Observations Under Retention Limit) / (Total Observations) × 100%

Interpretation:

• 100% compliance - All observations are within policy; no archival needed
• <100% compliance - Some observations are aged out and queued for purge or soft-delete
• Declining compliance - Your observation generation rate exceeds purge rate; may need policy adjustment

Common policies:

• No limit (default) - Retain indefinitely
• 90-day rolling window - Auto-purge observations older than 90 days
• 1-year snapshot - Annual archival for compliance

Storage Efficiency

Definition: Approximate observations per unit storage (observations/MB).

Efficiency = Total Observations / (Database Size for observations table)

Interpretation:

• High efficiency (>500/MB) - Compression, dedup, or small observation payloads
• Low efficiency (<100/MB) - Large observation content or metadata; may warrant review
• Trending down - Growing observation size; check metadata bloat

Active Observation Count

Definition: Total live observations (not soft-deleted or archived) in your org/project.

Why monitor:

• Runaway growth may indicate a learning loop or import error
• Sudden drops may signal a purge or data loss (audit the audit trail)
• Per-project counts help balance learning load across teams

Health Thresholds

The dashboard uses these reference ranges:

Common Health Issues and Fixes

Viewing Health Metrics

The health panel appears on the Memory Dashboard in the fourth row, visible only when you have observations. It updates on every dashboard refresh.

Operator-only note: Non-operators see only the high-level dashboard summary without health metrics.

API Reference

Endpoint: GET /api/memory/analytics/health

Query Parameters:

• projectId (optional) - Filter to a specific project; omit for org-wide stats

Response:

{
  totalObservations: number
  estimatedBytes: number           // sum of content byte lengths
  distinctContentHashes: number    // count of unique content_hash values
  deduplicationRatio: number       // 1 - distinctContentHashes / totalObservations (0-1)
  latency: {
    p50Tokens: number | null       // p50 token count from recent injection logs
    p95Tokens: number | null       // p95 token count from recent injection logs
    sampleCount: number
    note: string                   // caveat: token counts, not latency ms
  }
  retention: {
    overdueCount: number           // observations older than 365 days still present
    checkedDays: number            // retention window checked (365)
  }
  projectId: string | null
}

Example:

curl -X GET "https://api.rensei.ai/api/memory/analytics/health" \
  -H "Authorization: Bearer rsk_..."

Response:

{
  "totalObservations": 14250,
  "estimatedBytes": 3421000,
  "distinctContentHashes": 9405,
  "deduplicationRatio": 0.34,
  "latency": {
    "p50Tokens": 312,
    "p95Tokens": 689,
    "sampleCount": 1000,
    "note": "Token counts from context_injection_logs (retrieval latency ms not yet instrumented)"
  },
  "retention": {
    "overdueCount": 187,
    "checkedDays": 365
  },
  "projectId": null
}

Retention Policy Configuration

Organizations can set retention policies via the admin panel:

observations:
  retention:
    window_days: 90  # null for unlimited
    archive_on_delete: true  # Soft-delete instead of hard-delete
    auto_purge_job: "0 2 * * *"  # Cron: 2 AM daily

Modifications to the retention policy take effect on the next purge job run.

Audit Trail Integration

All health metrics are backed by:

• observations table - Live observations with created_at, updated_at, metadata
• observation_feedback table - User corrections and weight adjustments
• audit_events - Hash-chain immutable log of metric changes and policy updates

Query the audit trail for historical health snapshots:

curl -X GET "https://api.rensei.ai/api/audit/events?resource=memory_health&since=2026-05-01T00:00:00Z" \
  -H "Authorization: Bearer rsk_..."

Best Practices

1. Monitor weekly - Set a recurring calendar reminder to check health metrics
2. Track trends - Use export data to plot dedup/compliance over weeks
3. Respond to yellow flags - Investigate before metrics drop to red
4. Archive high-value observations - Export top observations before purging to preserve knowledge
5. Audit dedup aggressively - False merges silently lose information; set thresholds conservatively

Related Pages

• Top Observations - Quality scores of individual observations
• Drift Alerts - Contradictions that may indicate quality issues
• Memory Export - Backup high-value observations before purge
• Audit Trail - Immutable log of all memory system changes

Rate Limits

The health API enforces a 100 req/min quota per organization. Health calculations are cached for 5 minutes.