Tool Analytics

Understand how agents use tools and where they fail. Tool Analytics breaks down tool call frequency and error rates to identify bottlenecks and reliability issues in your workflows.

Overview

Tool analytics measure how often each tool is called and what fraction of calls fail. The Performance dashboard surface shows:

• Tool call counts - Total invocations per tool in the time window
• Error rates - Failed calls ÷ total calls, per tool
• Ranked table - Tools sorted by frequency or error rate, with sparklines for trend

This helps you identify:

• Over-used tools - Optimization candidates or workflow design issues
• Brittle tools - High error rates indicate integration problems, flaky external services, or incorrect configuration
• Tool redundancy - Similar tools in your palette; candidates for consolidation

Metrics

Tool Call Count

Count of successful and failed calls to each tool, from agent-executed workflows.

• Data source: factory_events table, specifically agent.tool-summary events emitted on session completion
• Scope: Filtered by project, time window, workflow, and work-type
• Granularity: Aggregated per unique tool name (e.g., linear.issue-create, github.pr-open)

Each tool-summary event carries:

{
  "toolName": "linear.issue-create",
  "callCount": 5,
  "errorCount": 0,
  "avgDurationMs": 1240
}

Error Rate

Fraction of tool calls that failed, per tool.

• Formula: errorCount / callCount * 100
• Range: 0-100%
• Color coding: Green <5%, Yellow 5-15%, Red >15%

Errors include:

• Credential errors - Missing or invalid authentication
• Rate limit errors - API quota exceeded
• Network errors - Timeout, connection refused
• Business logic errors - Invalid argument, constraint violation (e.g., creating a Linear issue with missing required field)
• Transient errors - Retried and eventually succeeded, but counted in the attempt count

Tool Usage Table

The Tool Analytics panel displays a ranked table:

Trend shows 7d change (if viewing 30d window); helps spot emerging issues.

Sorting: By default, tools are sorted by call count (highest first). Optional: sort by error rate to surface reliability issues.

Display & Filtering

Available on the Performance → Tool Analytics panel with:

• Time window - 7d, 30d, 90d
• Project filter - Restrict to one project
• Workflow filter - Optional; show only tools called by one workflow
• Work-type filter - Optional; compare research vs. dev vs. qa
• Sort order - Call count (default) or error rate
• Error threshold - Show only tools with error rate >X% (default: all)

Data Pipeline

Collection:

1. Agent completes a session with a workResult
2. Session lifecycle hook emits agent.tool-summary event with aggregated tool metrics
3. Event is written to factory_events table

Aggregation:

1. Dashboard fetches events for the time window and project
2. Events are grouped by tool name
3. Call and error counts are summed, error rate is calculated

Retention: 90 days default (configurable per org); older events are archived.

Interpreting Results

High Call Count (Frequent Tool)

• ✓ Expected if the workflow heavily uses that tool
• ⚠ If unexpected, check:
  • Redundant tool calls in the workflow definition (the agent is calling it unnecessarily)
  • Loop exit conditions (the agent is retrying the same tool in a loop instead of moving forward)
  • Workflow state corruption (the agent is repeating work across sessions)

Low Error Rate (<5%)

• ✓ Tool is reliable and well-configured
• ✓ Credentials are correct
• ✓ External service is healthy

High Error Rate (>15%)

• ⚠ Credential issue: Check /org/integrations to confirm the API key or OAuth token is still valid
• ⚠ Configuration issue: Verify the tool's configuration in the workflow (e.g., required fields, scope, org/project bindings)
• ⚠ External service degradation: Check the service's status page (Linear, GitHub, etc.)
• ⚠ Rate limit: If error message mentions rate limiting, the service quota was exceeded; consider batching or backoff
• ⚠ Transient spikes: One-off errors that auto-retry may inflate the count; check trends over weeks, not minutes

Spike in Error Rate

• Recent credential rotation or OAuth refresh failure
• External API change (e.g., endpoint deprecation)
• Workflow change that now sends invalid arguments

Debugging

For a specific tool with high errors:

1. Open Sessions → find a session that called the tool
2. Open Session Detail → Inspector tab → Activity stream
3. Filter by tool name; inspect the tool input (arguments) and output (error message)
4. If arguments look wrong, check the workflow node that calls the tool; verify expressions and input binding

For a systematic issue (many tools failing):

1. Check Integrations page - is the credential still valid? (Check last_used timestamp)
2. Check Auth page - has your token expired? (Some platforms auto-rotate; re-connect OAuth)
3. Check external service status page (GitHub, Linear, etc.)

Example: Investigating a High-Error Tool

Tool linear.issue-update has a 15% error rate over 30 days. Steps:

1. Click "linear.issue-update" in the Tool Analytics table
   → Drill-down view shows last 10 failed calls

2. Inspect one error:
   Args: { issueId: "LIN-123", status: "In Progress" }
   Error: "Linear API: Invalid status 'In Progress' for team PLAT"

3. Root cause: Your team uses "In Dev" not "In Progress"
   Fix: Update the workflow node's mapping: status = "In Dev"

4. Verify: Re-run affected sessions; check error rate drops next hour

API & Programmatic Access

The metrics are aggregated server-side in the Performance dashboard component. There is no separate API endpoint for tool analytics; data is fetched as part of the performance dashboard query.

To fetch raw factory_events for analysis:

curl -H "Authorization: Bearer $RENSEI_API_KEY" \
  'https://api.rensei.ai/api/factory/events?eventType=agent.tool-summary&workspaceId=ws_123&after=2026-05-03T00:00:00Z'

Returns array of agent.tool-summary events with nested tool metrics.

Related Pages

• Agent Reliability - Overall session success/crash rates
• Integrations - Manage API credentials for tools
• Workflow Nodes - Tool node reference
• Session Inspector - Deep-dive into tool invocation traces