Feedback Impact

The Feedback Impact sub-view measures how effectively your feedback on observations improves their usefulness in agent decision-making. It tracks injection rates (how often observations make it into agent context) and rank delta (how much feedback changes their relevance ranking).

Understanding Feedback Loop

When agents run, they retrieve and inject relevant observations into their prompts. Feedback you provide on observations - upvotes, corrections, outdated flags - improves the ranking of future retrievals:

Observation → Retrieved → Injected into prompt → Agent uses it → Feedback → Reranked

This is your feedback loop. Strong feedback impact means:

Your feedback is accurate (agents benefit from corrected patterns)
Ranking algorithms respond appropriately
Fleet learning is converging toward shared understanding

Key Metrics

Injection Rate

Definition: Percentage of observations that were injected into at least one agent session during your time window.

Injection Rate = (Injected Observations) / (Total Observations) × 100%

Interpretation:

High rate (>70%) - Most of your knowledge is actively used; good coverage
Medium rate (40-70%) - Balanced mix of active and exploratory knowledge
Low rate (<40%) - Many observations are dormant; consider pruning or improving observability

Factors affecting rate:

Work volume - More sessions = more injection opportunities
Memory injection toggle - If disabled, rate drops to 0% (by design)
Observation age - Older observations are downweighted in retrieval
Agent scope - Agents with broader scope will inject more observations

Rank Delta

Definition: Average change in observation rank position before and after feedback is applied.

Rank Delta = (Mean Rank Before Feedback) - (Mean Rank After Feedback)

Interpretation:

Positive delta (e.g., +15) - Feedback is moving observations higher in rank; feedback is effective
Zero or negative delta - Feedback has minimal impact; ranking algorithm may need tuning
Large positive delta (>50) - Dramatic rank improvements; feedback is transformative

Example:

Before feedback: observation ranked #47 in retrieval list
After upvote: re-ranked to #12
Rank Delta: +35 (moved up 35 positions)

Interpreting Low Impact

If your feedback isn't improving injection rates, it may indicate:

Symptom	Cause	Fix
Upvote doesn't increase rank	Ranking weights favor other factors (recency, corroboration)	Check if the observation is recent/corroborated; upvote may help long-term
Downvote/correction has no effect	Observation still has high confidence elsewhere	Mark it misleading to explicitly suppress; or delete it
New observations never get injected	Newness penalty or low confidence	Upvote high-quality new observations to boost them
Feedback doesn't persist across time ranges	Feedback is working, but decay is fast	More frequent feedback cycles; regular positive reinforcement

Common Workflows

Validate and Amplify

Review top observations
Upvote ones you want agents to rely on heavily
Monitor next week: have injection rates increased?
Use this signal to identify "champion patterns"

Clean Up Low-Value Observations

Check which observations have low injection likelihood even after feedback
Mark as outdated or delete if they're stale
High-frequency feedback on low-injected observations is wasted effort

Drive Adoption of New Patterns

After introducing a new best practice, inject a corresponding observation
Upvote it regularly in the feedback panel
Monitor injection likelihood climbing as agents adopt it
Use this data to prove the pattern is working

Visualization

The feedback-impact panel shows:

Quality metric card: the top-quartile injection rate - how often the highest-weighted observations actually get injected
Top-10 rank-delta table: each top observation side by side with its baseline (unweighted) vs current (feedback-weighted) retrieval rank, with a Δ badge showing how far feedback moved it up or down
Project and time-range filters (7/30/90 days) plus manual refresh

API Reference

Endpoint: GET /api/memory/analytics/feedback-impact

Query Parameters:

projectId (optional) - Filter to a specific project
since (optional) - ISO-8601 date lower bound (default: 30 days ago)
until (optional) - ISO-8601 date upper bound (default: now)

Response:

{
  sessionCount: number               // distinct sessions in the window
  topObservations: Array<{
    observationId: string
    contentPreview?: string          // first 200 chars of content
    retrievalCount: number           // how often retrieved in the window
    avgRelevanceScore: number        // mean relevance score from retrieval_ab_logs
    avgFeedbackWeight: number        // mean feedback weight
    avgWeightedRank: number          // mean rank position after weighting
    avgUnweightedRank: number        // mean rank position before weighting
  }>
  qualityMetric: {
    topQuartileInjectionRate: number // share of injections drawn from top-quartile observations
  }
  projectId: string | null
  since: string
  until: string
}

Example:

curl -X GET "https://api.rensei.ai/api/memory/analytics/feedback-impact?since=2026-05-01T00:00:00Z" \
  -H "Authorization: Bearer rsk_..."

Response:

{
  "sessionCount": 142,
  "topObservations": [
    {
      "observationId": "obs_abc123",
      "contentPreview": "Always validate user input before database operations",
      "retrievalCount": 38,
      "avgRelevanceScore": 0.87,
      "avgFeedbackWeight": 0.94,
      "avgWeightedRank": 2.1,
      "avgUnweightedRank": 4.3
    }
  ],
  "qualityMetric": {
    "topQuartileInjectionRate": 0.62
  },
  "projectId": null,
  "since": "2026-05-01T00:00:00Z",
  "until": "2026-06-01T00:00:00Z"
}

The avgWeightedRank minus avgUnweightedRank difference (computeRankDelta) measures how much feedback weighting moves each observation up in rank. A negative value means feedback is promoting the observation (lower rank number = higher in list).

Feedback Types

Upvote ✅

Effect: Increases weight and injection likelihood
Use when: Observation is accurate and you want agents relying on it more
Impact timeline: Immediate for new retrievals; full impact within 24h

Downvote / Mark Incorrect ❌

Effect: Decreases weight; may lower injection likelihood
Use when: Observation is wrong or less important than alternatives
Impact timeline: Immediate; usually prevents injection by next session

Mark Outdated ⏳

Effect: Preserves observation for historical/audit purposes but reduces weight
Use when: Information was once correct but is no longer current
Impact timeline: Immediate; downweights but doesn't delete

Mark Misleading 🚩

Effect: Explicitly suppresses from injection; flags for review/deletion
Use when: Observation is actively harmful (e.g., bad security advice)
Impact timeline: Immediate and strong; agents will not inject

Feedback Frequency

Optimal feedback patterns:

Weekly review - Check top/bottom observations once per week
Real-time on critical - Immediately mark safety-critical observations as misleading if wrong
Batch upvoting - Upvote 5-10 high-quality observations at a time (don't over-feedback, dilutes signal)
Avoid churn - Upvote, then downvote, then upvote again (confuses the algorithm)

Best Practices

Feedback is a long-term investment - Don't expect immediate rank changes; compound effect over weeks
Prioritize safety - Always flag misleading observations in security/compliance domains immediately
Validate before feedback - Incorrect feedback teaches the system wrong patterns
Track your impact - Use this dashboard to see if your feedback efforts are paying off
Involve the team - Feedback from multiple people converges faster than single-person feedback

Memory Health - Dedup and retention (feedback helps clean up duplicates)
Top Observations - Quality rankings (feedback directly improves these)
Trends - Volume context (feedback doesn't change volume, only utility)
Context Budget Pareto - Token efficiency (feedback improves which observations are worth injecting)

Rate Limits

The feedback-impact API enforces a 100 req/min quota per organization.

On this page