Metrics

This page lists every Prometheus metric Agent Kourier exports at /metrics, with its labels and what it counts. Setting up the scrape is Export metrics and traces.

Metric labels carry configured names (Bindings, connections) and values from closed sets, never prompts, answers, user names or error text.

Connection, configuration and turns

Metric	Type	Labels	Definition
agentkourier_chat_connected	gauge	connection	1 while the chat connection (Slack Socket Mode) is up, else 0, by ChatConnection. /readyz does not say this.
agentkourier_turn_outcomes_total	counter	binding, outcome	Turns that ended, by Binding and how: the terminal state of the task, or the class of what stopped it. outcome is one of completed, failed, rejected, canceled, credentials_rejected, forbidden, unreachable, session_gone, binding_gone, wedged, uncancellable, pause_ended, answer_rejected, refused, no_final, response_too_large, lookup_incomplete. It is counted before the session turn ended log line.
agentkourier_config_stale	gauge		1 before the first config load, and while the running config may lag its source: the latest reload failed, or the source says it cannot be reached.
agentkourier_config_reload_errors_total	counter		Config reloads that failed.
agentkourier_config_invalid_objects	gauge	kind	Objects that the broker refuses or holds past their deletion, by kind (AgentBackend, ChatConnection or Binding): a Binding of this broker (it names this broker's connection, or none that exists) that fails validation; any AgentBackend in the cluster that fails it, including one no Binding of this broker uses; this broker's own ChatConnection when it fails validation or was deleted and is held; and, while that connection is held, each Binding of it and each AgentBackend they use that was deleted and is held with it; a held Binding that lost its channel or whose backend is gone is counted and not served. Always 0 while the config is read from files, where one bad object fails the whole reload and counts under agentkourier_config_reload_errors_total instead.
agentkourier_config_held_objects	gauge	kind	Objects that were deleted and that the broker holds at their last good generation, by kind, while its own ChatConnection is deleted and held: the connection itself, the Bindings of it that are gone, and the AgentBackends that they and the Bindings of it that are still there use and that are gone. A nonzero value means the broker holds from memory what the cluster no longer has; a held Binding whose backend went first, or that lost its channel, is held and not served, and the broker's log at the start of the hold says how many. Always 0 while the config is read from files, and 0 when nothing is held.
agentkourier_config_binding_rejections	gauge	reason	Bindings that are not accepted as written, by reason: InvalidSpec, BackendNotFound, BackendRejected, ConnectionNotFound, ConnectionRejected, NamespaceNotAllowed or ChannelConflict, and other for a reason outside that set (none is written today). A Binding that is served from an older generation of itself counts too. Always 0 while the config is read from files. A Binding that is not accepted has no agentkourier_binding_ready series, since that gauge is the prober's and only a served Binding is probed: alert on this one.
agentkourier_binding_status_writes_total	counter	result	Writes of a Binding's status by the leader, by result: written, put_back (written again to the words it already had, because the status was changed since: a person's edit, or another writer; steady growth is a fight, which the troubleshooting page explains), or failed (the API server refused the write or could not be reached; the write is tried again after a wait that starts at 10 seconds and doubles up to 5 minutes). A write that finds the Binding already deleted counts as none of them. It has series only after the leader of a broker that reads the custom resources has tried a write, so it has none while the config is read from files.
agentkourier_crd_schema_state	gauge	crd, state	What the broker found when it compared the schema hash on each of the three CRDs with the one it was built with, at start and again every minute: 1 for the state found and 0 for the other three, by crd and state (match; mismatch: another hash, so the CRDs and this image are not from the same chart release, and the log warns when a CRD's result changes to it, naming the CRD and both hashes; it warns again if the CRD goes unknown and back; unannotated: the CRD has no hash, as a CRD from an older chart or file; unknown: the CRD could not be read). The hash is an annotation that the generated CRD files carry, so a Helm install and a kubectl apply apply it (a GitOps tool that applies the files should too: that is what such a tool should do, and none has been run): it says which files were applied, and is not proof of what the API server serves (a CRD edited by hand keeps it). During an API outage the series goes to unknown, so mismatch reads 0 and an alert on it resolves, then fires again when the API server answers. A broker that reads files has no series. A broker with mismatch serves as usual: update the CRDs the way they were installed, to those of the chart release that matches the image (helm upgrade with crds.install for CRDs Helm made, kubectl apply --server-side -f of that release's files/crds for CRDs kubectl made), and the next comparison clears it.
agentkourier_credential_rejections_total	counter	binding	Credentials an agent rejected, by Binding: the front door refused the Binding's token.
agentkourier_audit_write_failures_total	counter	kind	Audit entries that could not be written, by kind.
agentkourier_approval_outcomes_total	counter	binding, outcome	Pauses a Binding's policy turned away (auto_denied, ask_user_cancelled), or tool approvals decided (approved, denied, partially_approved) or timed out (timed_out), by Binding and outcome.
agentkourier_interaction_decisions_total	counter	result	Presses decided on an interaction, by result. unrecorded counts a press ended because its decision audit row cannot be written; no decision is sent.
agentkourier_interaction_timeouts_total	counter		Interactions that timed out unanswered.
agentkourier_outbox_abandoned_total	counter	reason	Outbound posts given up on, by reason: the outbox's own invalid (not postable, or an unreadable payload), lookup_timeout and expired (still unposted after 7 days), or the platform's channel_gone, too_long, invalid_content and other.
agentkourier_leader	gauge		1 while this replica does Agent Kourier's work (it leads, or it is the only replica: SQLite), 0 while it is a standby.
agentkourier_leader_takeovers_total	counter		Times this replica took the leadership: it won the lease and was given the next epoch.

Operations, intake, delivery and triggers

Metric	Definition
agentkourier_operation_duration_seconds	Duration of a finite operation, including failures and retries, by operation (the span name, such as agentkourier.agent.send or agentkourier.store.GetSession) and outcome. Agent execution ends at a terminal state, input-required or auth-required; both pause states are non-error outcomes. Agent and store calls are no longer also recorded in agentkourier_agent_operation_duration_seconds and agentkourier_store_operation_duration_seconds, which duplicated this histogram: select them with operation=~"agentkourier.agent.*" and operation=~"agentkourier.store.*"
agentkourier_agent_first_text_duration_seconds	Attempt start to first nonempty text observed on send/answer; no observations for follow snapshots
http_client_request_duration_seconds	HTTP request through response EOF or close; an SSE response includes its stream lifetime
agentkourier_intake_total	Message/press dispositions; source="reply_queue",disposition="duplicate" counts durable duplicates separately; a bot's message is trigger_matched, trigger_unmatched (no trigger took it) or ignored (an unbound channel, or a reply in a thread); a person's reply that a mention-only Binding turned away is unmentioned
agentkourier_slack_ack_duration_seconds	The envelope taken off the connection to completion of the ack attempt, or return with ack withheld, by outcome; so it includes the wait behind the earlier events of the same channel. It measures queueing the ack, not confirmation from Slack. An event that is shed or dropped at shutdown is left unacknowledged and gets no sample: agentkourier_slack_events_shed_total counts the shed ones
agentkourier_slack_events_shed_total	Slack events left unacknowledged, so that Slack may send them again, because the adapter already held as many as it may, by reason: channel_backlog (the channel's own events fill its limit) or adapter_backlog (the events of all channels fill theirs, and the channel already holds one). The limits are on Limits. A steady count means a channel's events take longer to handle than they arrive
agentkourier_turn_queue_wait_seconds	Enqueue to first execution attempt in this process; a restart may observe the same waiting reply again
agentkourier_sessions_active / agentkourier_sessions_paused	Current in-process runners and runners parked on input
agentkourier_stream_reconnects_total	Completed reconnect attempts, by outcome
agentkourier_session_recovery_total	Adopted tasks, recording failures and reconciliation diagnostic notices
agentkourier_reply_queue_depth / agentkourier_reply_queue_oldest_age_seconds	Durable undelivered reply count and oldest age
agentkourier_outbox_pending / agentkourier_outbox_oldest_age_seconds	Unposted, nonabandoned rows and oldest age; includes incomplete stream rows
agentkourier_outbox_delivery_duration_seconds	Row creation to successful delivery/adoption attempt; replay after a crash can produce another observation
agentkourier_render_final_total	Final rendering outcomes: delivered, empty output, permanent failure, retry exhaustion, deadline exceeded or cancelled
agentkourier_retention_removed_total	Stored content that retention removed, by what: stranded_replies (undelivered replies dropped after 30 days, a turn that was never sent: worth an alert on any increase), cleared_interactions (settled interactions whose request and answer were emptied), audit_entries (purged past AGENTKOURIER_AUDIT_RETENTION; an increase with the setting at keep is a fault), and expired_sessions with the rows that went with them, session_replies, session_interactions, session_presses and session_outbox (sessions deleted 90 days after their threadTTL), and stale_tasks with the stale_task_interactions cancelled with them (a task that a crash left on a session past that horizon, ended so that the session can be deleted; a standing increase means crashes). See Stored data
agentkourier_interactions_pending	Durable pending interactions
agentkourier_interaction_wait_seconds	Creation to first committed resolution, by kind and outcome (the state it resolved to); retries do not add another observation; the pending interactions that retention cancels with a stale task are not observed
agentkourier_chat_api_calls_total	Slack adapter operations and results, including logical API errors over HTTP 200; cached user lookups are adapter operations too. Outcomes include rate_limited, auth_rejected, channel_gone, not_streaming, api_error, and mode_refused (Slack refused the chunks mode of a stream, which makes Agent Kourier fall back to markdown_text and show no more steps). The ThreadBetween operation is a turn's read of its thread's earlier messages, for a mentioned turn in a chat.threadReplies: mention Binding and for a bot's mention on any Binding (conversations.replies, at most 5 pages, with an auth.test), which on a bot's first turn in a thread includes what opened it: rate_limited on it is why a turn went out with the line that the replies could not be read. The SetThreadStatus operation is assistant.threads.setStatus, which sets and clears the is working… status of an alert thread; its duration is agentkourier_operation_duration_seconds{operation="agentkourier.chat.SetThreadStatus"}. A refusal is counted as api_error, and after a refusal that turns the status off for the connection no more calls are made or counted. The ChannelAccess operation is Binding verification's read of a channel (conversations.info, one a pass for each Binding): channel_gone there is a channel that does not exist or that the bot cannot see, and api_error includes a missing scope. Its duration is agentkourier_operation_duration_seconds{operation="agentkourier.chat.ChannelAccess"}
agentkourier_thread_unmentioned_total	Replies in a thread Agent Kourier owns that a Binding with chat.threadReplies: mention, or a session that a bot's mention began on any Binding, ignored because they did not mention Agent Kourier, by binding: nothing was written or sent, no reaction was added and the thread's threadTTL was not extended. A reply in an expired thread counts too (it gets no expiry notice). A typed answer admitted to a pending question that takes text replies (the a2a dialect's free-text question) is not counted here: it needs no mention and is queued as that question's answer. (A redelivery of such an answer's event after the question closed is counted, once, as it is then an ignored reply.) It is the engagement of a thread that people discuss without waking the agent, so a rising count with a flat turn count is the feature working, and a Binding whose people complain that Agent Kourier ignores them shows it here.
agentkourier_mentions_total	What the names in a turn's final reply came to on a Binding that sets chat.mentions, by binding and outcome: made (a message that mentions the person was queued, which is not yet a delivery or a notification), undeclared (an @name no entry declares, which mentions nobody) and capped (a declared person past chat.maxMentionsPerTurn, who is not mentioned). A turn that failed, was stopped or was rejected counts nothing. A turn rendered again after a restart may count undeclared and capped twice, and a crash just after a mention is queued can leave it out of made.
agentkourier_thread_context_total	Turns that asked for what was posted in their thread since the previous turn: mentioned turns of a Binding with chat.threadReplies: mention, and a bot's mention on any Binding, which reads every author's messages since the previous bot's mention and, on the thread's first, what opened it. By binding and outcome: read (a block of replies went with the turn), empty (there was nothing to add: nothing the turn would show since the previous turn, or a window with none left, as after two near-simultaneous mentions), cut (a block went, with a last line that says how many earlier replies it left out: more than 50, past the block's 6000-unit cap, or older than the newest 1000 messages of the window; a reply longer than 1000 units is shortened, not left out; what opened the thread is never left out), failed (the turn went with the fixed line "Agent Kourier could not read the earlier replies in this thread.": the platform's read of the window or of what opened the thread failed or was rate limited past one wait of at most 5 s, the thread's window was past the 5 pages it will read, or the reply queue could not say which messages were turns) and unsupported (the platform cannot read a thread, so the turn went with the mentioned message alone; one info log line for each Binding). A turn is counted once when it is first sent (a resend after the agent was busy does not ask again), and again if a restart sends it anew. Nothing a person wrote is in a log line. A rising failed with agentkourier_chat_api_calls_total{operation="ThreadBetween",outcome="rate_limited"} says the reads are rate limited, which Slack counts against the same conversations.replies budget as the outbox's recovery
agentkourier_trigger_matched_total / agentkourier_trigger_unmatched_total	Bot messages a Binding's botMessage trigger matched, and bot messages in its channel that matched none (the heartbeat an exclude turns away counts here), by binding. A rising unmatched count after a template change means the trigger's regexes no longer fit; the debug log line names the channel, the message, and the sender's ID and kind, never the text. A bot's mention that a Binding takes counts as matched
agentkourier_trigger_decisions_total	What a matched bot message did to its key's thread, by binding and decision: started (a new investigation), repeat (a note that it fired again), closed (a note that it resolved), close_unknown (a close with no open thread, ignored), dedup (a message delivered again) empty_key (a keyed trigger whose key came out empty, so the message is an investigation of its own) and not_started (the session hook said the turn will not start, such as a refused prompt: the event is acknowledged, no cooldown begins) and limited (a start that one of the trigger's limits stopped: no turn, no key held, nothing posted in the alert's thread; see agentkourier_trigger_limited_total). A bot's mention has only started (a turn, in a new or an owned thread), limited and not_started (its session cannot take a turn, which uses none of the limits, or its prompt was refused), plus dedup. Flapping shows as repeat and closed rising while started stays flat; the key is in the debug and info log lines only as a short hash
agentkourier_trigger_limited_total	Starts that a limit stopped, so no investigation began, by binding, trigger (its index in the Binding's chat.triggers) and limit: rate (the trigger started its rateLimit.maxRuns in the last rateLimit.per), concurrent (rateLimit.maxConcurrent of its investigations are running) or daily (it started maxRunsPerDay in the last 24 hours); when more than one is reached the daily cap is reported, then the rate, then the concurrency. A repeat, a close and a note are never limited. The alert is not posted about: it reached people through its own receiver. Once for each window, which opens with the first stopped start and lasts the trigger's rateLimit.per, one line in the channel (top level, through the outbox) says how many were not investigated and why, for example "7 bot messages not investigated in the last 10m (rate limit).", and names no alert. A count that keeps rising means the limits are too tight for the storm, or that the investigations are slow (concurrent), or that one source is noisy (daily). A bot's mention counts under the index of the Binding's mention trigger (0 for an empty chat.triggers), with its fixed limits and a window of 30 minutes, and the audit log has an entry for each one stopped. Its concurrent counts a mention while its own turn is queued or is the session's task, not a later turn in its thread
agentkourier_trigger_refused_total	Starts that began no turn because the first prompt failed its safety check or could not be rendered, by binding, trigger (its index in the Binding's chat.triggers), ref (the value that failed, as a template names it without the dot: Fields.alertname, Key, Link, State, Channel, Source, or template for a template that failed to run) and reason (characters, too_long, not_link, render_error or other). ref and reason come from closed sets, and the entry names are the Binding's own. Each refusal is also a note in the alert's thread, "Not investigated: ...", and an audit entry that names the ref, reason and length of the value and never the value; a refusal opens no cooldown, so a fire of the key more than a minute later starts an investigation. A rising count after a change to the template or to how the alerting tool formats its messages means a value the template reads no longer fits its check
agentkourier_investigation_feedback_total	Votes cast on the final message of an investigation (the thread a trigger started) or of an answer (a Binding with feedback.enabled; a turn that another bot's mention started is one), by binding, agent, alertname and verdict (useful or not_useful). It counts votes cast, so a person who changes their vote adds one under the new verdict and the first stays counted; pressing the button of the verdict already recorded counts nothing. It therefore reads as activity and as an upper bound, and never as the standing of an alert, which is read from the feedback table: it keeps one latest vote for each person and message (Stored data). A vote counts only from a member of the channel and never from a bot. alertname comes from the session origin's bounded subject only for a validated alertname on the botMessage origin (which every trigger start records), on alertmanager (kept only by sessions created before that origin was retired), or on legacy (migrated from the older session schema); a generic source subject is none for an answer or unknown for an investigation. A Binding has at most 100 alert names of its own in this process, after which the votes count under other

Binding verification

The leader probes each Binding of the connection it serves, when it starts, when the config or a Secret it refers to changes, and every 5 minutes (with up to 10% jitter) after a pass begins. A change triggers a pass at most once every 30 seconds, and a Binding that a reload adds is a change like any other: one that comes sooner waits, and a new Binding has no series until then; the pass for it, which sees the last change, runs when the 30 seconds are over. A probe fetches the agent's card on a new connection and reads the Binding's channel as the bot (conversations.info on Slack); it sends the agent no message and posts nothing. A standby exports none of these series.

A Binding has agentkourier_binding_condition series for a condition once its first probe has ended. Its Ready series, with agentkourier_binding_ready, come once every first probe has ended, or once a first result has found it false: a result that is false sets agentkourier_binding_ready to 0 at once, even while the other probe is in flight, and a result that is not yet known (unknown) is hidden until the other probe has ended too, whichever probe ends first. A rule on agentkourier_binding_ready == 0 therefore fires for a Binding that one probe has found false, and does not fire for one whose other first probe is still in flight and whose first result is not yet known. Once both first probes have ended, a result that is not yet known counts as not ready and sets the gauge to 0. A Binding's series are deleted when a reload removes it or moves it to another ChatConnection than the one served, and the leader logs one INFO line stopped verifying a Binding for each; so a series that is gone can mean this, and its absence is not health.

A probe is not retried. One failure holds agentkourier_binding_ready at 0 until the next pass, which begins 4.5 to 5.5 minutes after the last one began (and up to one 5-second tick later). That includes a single 5xx, 409 or refused connection, as an agent in the middle of a rollout gives, which reads false, as does a backend that does not answer within its 8-second bound, and a result that is not yet known (for example an agent's 408, 425 or 429, a chat rate limit or deadline, or the Prober's own timeout), which reads unknown. A turn's connect waits up to 30 seconds, so a backend that takes longer than 8 seconds to serve its agent card reads false although a turn could still reach it. status="false" of agentkourier_binding_condition leaves the results that are not yet known out, but not the transient failures, so an alert on either should use a for: well beyond one interval, such as 10m, since the next pass can take a while to reach a Binding when backends are slow to answer.

The reason for a result is not a label; it is in the log line of the change. Troubleshoot an install lists each reason and what to do about it.

• a WARN a condition is false for a condition that went false;
• a WARN a condition is unknown for ChatCredentialsChanged (the chat credentials Secret holds other tokens than the ones in use: restart Agent Kourier) and for ChatCredentialsUnusable (the Secret, or a key of it, is gone or empty, or the Secret is of a type that is never read: the connection keeps the tokens it read, and a restart fails until the Secret is mended);
• an INFO a condition is unknown for any other result that is not yet known: for the agent, a 408, 425 or 429; for the chat platform, a rate limit, a deadline, or the chat platform read failed, with its cause in the line's cause attribute (for Slack, its code such as no_permission when it answered, or the transport's error such as connection refused when it did not);
• an INFO a condition is now true, with the reason it had before in was, for a condition that was false, or unknown after a first result. A first result that is true has no line.

A probe whose check could not tell leaves the condition unknown with no such line: its line is a probe failed, with the error, and a failure of the prober itself has its own line, the prober failed. A Binding that a reload removed during a pass has no a probe failed line for its agent probe, which finds the Binding gone. Its channel probe runs for the pass's own list of Bindings, so it can still log a change for the Binding before the stopped verifying line.

Metric	Type	Labels	Definition
agentkourier_binding_ready	gauge	binding	1 while every condition of the Binding is true, 0 while one is false or not yet known. It is the prober's result for a Binding the broker serves, and so says nothing of Accepted: a Binding that was refused after an edit and is served from its last good generation can read 1, and one that is not served has no series. See agentkourier_config_binding_rejections.
agentkourier_binding_condition	gauge	binding, condition, status	1 for the status the condition has and 0 for the other two, by Binding. condition is BackendReachable (the agent card was fetched and is one Agent Kourier can use), ChannelJoined (the bot is in the Binding's channel, which exists and is not archived) or Ready. status is true, false or unknown.
agentkourier_binding_probe_duration_seconds	histogram	condition	How long a probe took, by BackendReachable or ChannelJoined, whatever it found.

Postgres store

Only with AGENTKOURIER_STORE=postgres.

Metric	Type	Definition
agentkourier_store_deposed_total	counter	Writes the store refused because another replica has led since: its epoch was stale.
agentkourier_store_pool_conns	gauge	Connections of the store's pool, in use, idle or being made.
agentkourier_store_pool_acquired_conns	gauge	Connections of the store's pool in use.
agentkourier_store_pool_idle_conns	gauge	Idle connections of the store's pool.
agentkourier_store_pool_max_conns	gauge	Largest size of the store's pool.
agentkourier_store_pool_acquires_total	counter	Connections taken from the store's pool.
agentkourier_store_pool_empty_acquires_total	counter	Takes that had to wait for a connection.
agentkourier_store_pool_canceled_acquires_total	counter	Takes that a context ended before they got a connection.
agentkourier_store_pool_acquire_seconds_total	counter	Time spent taking connections from the store's pool.

Runtime

Metric	Definition
go_*, process_*	The Go runtime and process collectors.
go_sql_*	SQLite only: database pool statistics, separately for the read and write pools (db_name). Postgres exposes agentkourier_store_pool_* instead.

Reading them

Duration histograms cover milliseconds through one hour. Use separate latency queries for execution, human wait and visible delivery. The terminal turn outcome counter describes the backend task; agentkourier_render_final_total describes what happened to the output in chat. A completed task can have a failed final render.

Backlog scrapes query the store, SQLite or Postgres, with a one-second timeout. A failed query produces an invalid metric and a failed scrape rather than reporting a false zero backlog.

Reference: OpenTelemetry HTTP span conventions and span links.