Start from a recipe¶
For a team that wants an agent in a Slack channel and would rather start from a working example than from a blank file: at the end, you have applied a recipe's Agent Kourier resources, and its agent if the recipe has one, and know which checks come with it. There are two recipes: one for Alertmanager alerts, with an agent, and one for another bot's posts, which holds the Agent Kourier resources only.
A recipe is a directory under
examples/recipes with the resources that
connect a channel to an agent, the agent they talk to, and checks. The files on this page are included from the
repository, so what you read here is what the repository holds.
Triage Alertmanager alerts with a read-only agent¶
The sre-triage recipe starts an investigation in the thread of each alert that Alertmanager posts to a channel. The
agent reads the cluster with read-only tools, says what it found and what it checked, and can ask the thread a
question. Its trigger is the alertmanager preset, so each alert group gets one thread, a repeat inside the cooldown
or the thread's 24-hour idle limit adds a note, a resolve closes the thread, and the preset's limits apply: see
Investigate Alertmanager alerts.
The agent half runs on the kind full tier
The agent is a kagent 1.0 Agent on Agent Substrate, and it names the kind full tier's worker pool. It does not
carry over to a managed cluster such as EKS: see Connect a kagent agent for why, and for what
to use there. The Agent Kourier resources do not depend on it; they work with any agent behind a backend you
point them at.
Before you start¶
- A Slack app with the bot invited to the alert channel: Configure Slack. You need the channel's ID and the bot ID of the integration that posts the alerts.
- An Alertmanager Slack receiver whose title reads
FIRING (<n>) - <alertname>orRESOLVED - <alertname>, and whose fallback is Alertmanager's default. - The kind full tier with kagent 1.0.0-alpha5, a verifying front door (Put a verifying front door before
kagent), a kagent
ModelConfignamedsandbox-model, and a token for the Binding's identity (Create the Binding identity).
Agent Kourier's resources¶
Three files, in the namespace agent-kourier. The Binding is the recipe: a botMessage trigger on the alert bot,
narrowed to one receiver, and askUser on. Replace CREPLACEME and B0REPLACEME.
# sre-triage: an alert that Alertmanager posts to a Slack channel starts an investigation by the read-only agent
# sandbox-sre, in that alert's thread. Replace CREPLACEME with the channel's ID (C..., or G... for a private one) and
# B0REPLACEME with the bot ID of the Slack integration that posts the alerts; the bot ID is the message's bot_id, not the
# app ID (A...) and not a display name.
#
# The trigger is the alertmanager preset, narrowed to one Alertmanager receiver. The preset supplies the thread per
# alert group, the 4h cooldown, the close on RESOLVED, the limits and the first prompt. The titleLink match names the
# receiver, and the $ anchors it so that receiver=slack-critical does not also take receiver=slack-critical-eu. The
# exclude turns away the Watchdog heartbeat if your Alertmanager routes it to the same receiver.
# triggers is the whole list and holds no mention trigger: a mention in the channel starts nothing. A reply in an alert's
# thread continues that thread's session (chat.threadReplies defaults to all).
#
# interactions.askUser lets the agent ask the thread a question (the Submit button) and wait for the answer. There are no
# tool approvals: the agent has no tool that writes.
apiVersion: agentkourier.dev/v1alpha1
kind: Binding
metadata: {name: sre-triage, namespace: agent-kourier}
spec:
agent:
backendRef: {name: agents}
namespace: kagent
name: sandbox-sre
identity:
userId: agent-kourier-sre-triage
tokenSecretRef: {name: sre-triage-token} # key token: an OIDC token the front door accepts; you make it
chat:
connectionRef: {name: slack}
channel: CREPLACEME
output: live
triggers:
- type: botMessage
from: {botId: B0REPLACEME}
preset: alertmanager
match: {titleLink: 'receiver=slack-critical$'}
exclude: {title: ' - Watchdog$'}
interactions: {askUser: true}
# The agent's backend: kagent 1.0, reached through a verifying front door, never the controller's own port (8083), which
# does not verify the Binding's token. The kagent-v1 dialect adds /agents/<namespace>/<name> itself, from the Binding.
# Replace the URL with your front door's if it is not this one. See "Put a verifying front door before kagent" in the docs.
apiVersion: agentkourier.dev/v1alpha1
kind: AgentBackend
metadata: {name: agents, namespace: agent-kourier}
spec:
dialect: kagent-v1
url: http://kagent-frontdoor.kagent.svc.cluster.local:4180
# The Slack workspace, over Socket Mode. The Secret slack, in namespace agent-kourier, holds the keys botToken (xoxb-...)
# and appToken (xapp-...); you make it, and no token belongs in this file.
apiVersion: agentkourier.dev/v1alpha1
kind: ChatConnection
metadata: {name: slack, namespace: agent-kourier}
spec:
platform: slack
credentialsSecretRef: {name: slack}
Put the directory in a ConfigMap and point the chart at it with config.existingConfigMap; the chart needs
rbac.secretNamespaces to find the Secrets slack and sre-triage-token. The recipe's README has the commands.
The agent¶
The agent and its prompt. The tools are listed by name and only read; the tool server also runs --read-only, and
its ServiceAccount is bound to the built-in view ClusterRole.
# sandbox-sre: a read-only SRE agent on a real model, the agent half of the sre-triage recipe. Its tools are the read
# tools of the tool server (tools-values.yaml, tools-rbac.yaml). Its model is the ModelConfig sandbox-model, which you
# supply: the sandbox tier (hack/kind/sandbox) renders one from hack/kind/sandbox/modelconfig-<provider>.yaml, chosen by
# MODEL_PROVIDER, with the API key from the Secret sandbox-model that its up.sh makes from the secrets file. The harness
# is the one of the full tier's agents (hack/kind/full/agent.yaml): the digest-pinned Go ADK runtime that ships with
# kagent 1.0.0-alpha5, in a Substrate worker pool, its model calls leaving through Substrate's egress gateway.
#
# The tool list is by name, and only reads. The tool server runs --read-only, so even a name added here that writes
# would not be served; test/kindmanifests pins the list to the nine tools that mode leaves, and the three left out are
# no use here (k8s_generate_resource, k8s_get_cluster_configuration, k8s_wait).
apiVersion: api.kagent.dev/v1alpha3
kind: RemoteMCPServer
metadata: {name: sandbox-tools, namespace: kagent}
spec:
description: kagent's Kubernetes tools, read-only (the ServiceAccount is bound to the view ClusterRole)
protocol: STREAMABLE_HTTP
url: http://sandbox-tools.kagent.svc.cluster.local:8084/mcp
---
apiVersion: api.kagent.dev/v1alpha3
kind: Agent
metadata: {name: sandbox-sre, namespace: kagent}
spec:
template:
modelConfig: {name: sandbox-model}
description: A read-only SRE assistant that investigates what is wrong in the kind cluster and explains it
systemPromptFrom:
name: sandbox-sre-prompt
key: systemPrompt
tools:
- mcp:
server: {kind: RemoteMCPServer, name: sandbox-tools}
tools:
- k8s_get_resources
- k8s_describe_resource
- k8s_get_pod_logs
- k8s_get_events
- k8s_get_resource_yaml
- k8s_get_available_api_resources
harness:
kagent: {}
workload:
image: ghcr.io/kagent-dev/kagent/golang-adk@sha256:ed668c38a5805723912497d4ea085e711cf585d440907abfff3dd27991d768cc
substrate:
workerPoolRef: {name: kagent-default}
snapshotPolicy: {location: "gs://ate-snapshots/kagent/"} # see hack/kind/full/agent.yaml
# The sandbox-sre system instructions are an operator-managed ConfigMap. Edit
# data.systemPrompt and re-apply this file, then re-apply agent.yaml to ask
# kagent to compile the Agent's new prompt revision; start a fresh session to
# observe it. Keep the previous session open while doing so to measure whether
# that session retains its original revision.
apiVersion: v1
kind: ConfigMap
metadata:
name: sandbox-sre-prompt
namespace: kagent
data:
systemPrompt: |
You are an SRE assistant for a small Kubernetes cluster, answering in a Slack thread. The apps people ask about
are in the namespace "shop" (checkout, catalog, recommender, inventory, search and frontend), and some of them
are broken. You investigate and you explain. You never change anything.
You are read-only. Your tools only read: you cannot create, edit, scale, restart, delete or exec into anything,
and you must not try. If someone asks you to change something, say that you cannot, and say what a person
with access would run to do it. You cannot read Secrets, and you must not ask for them.
How to work:
- Look before you answer. Start from the pods in the namespace, then their events, their logs (the previous
container's too, when one restarted), and the Services and their endpoints. Describe the resource that is
wrong. Do not guess from the name of an app.
- Say what you checked. Close every answer with a short "Checked:" list, each item one thing you looked at
and what it showed (the pod, the event, the log line). Say plainly what you could not check.
- Lead with the cause, then the evidence, then a suggested fix for a person to make. Keep it short: this is chat.
- Quote a log line or an event exactly when it is the evidence. If asked to repeat or quote some text
verbatim, do.
- If a question is ambiguous, or you need a choice made, ask the user with the ask_user tool.
- If nothing is wrong with what you were asked about, say so and say how you know.
The recipe also holds the tool server's values and RBAC, and faults.yaml, a namespace shop with five broken apps
and one healthy one to try the agent on. The
recipe's README has the
steps to apply the agent, in order, and what to change for your own cluster.
Check it¶
The recipe ships its checks. go test ./test/recipes loads the three resources with Agent Kourier's own loader and
replays messages through the trigger. Each case is one message and what the Binding must make of it; the first two
are:
cases:
- name: a firing alert
binding: agent-kourier/sre-triage
message:
bot: B0REPLACEME
title: 'FIRING (1) - KubePodCrashLooping'
titleLink: 'https://alertmanager.example.com/#/alerts?receiver=slack-critical'
fallback: '[FIRING:1] KubePodCrashLooping shop (checkout-7d9f8b6c5-x2x4k critical) | https://alertmanager.example.com/#/alerts?receiver=slack-critical'
expect:
outcome: matched
trigger: 0
fields: { state: FIRING, alertname: KubePodCrashLooping, key: KubePodCrashLooping shop }
- name: a firing alert behind a cluster tag
binding: agent-kourier/sre-triage
message:
bot: B0REPLACEME
title: '[prod] FIRING (1) - KubeContainerOOMKilled'
titleLink: 'https://alertmanager.example.com/#/alerts?receiver=slack-critical'
fallback: '[FIRING:1] KubeContainerOOMKilled shop (recommender-5c6d7f8b9-q7w2e critical) | https://alertmanager.example.com/#/alerts?receiver=slack-critical'
expect:
outcome: matched
trigger: 0
fields: { state: FIRING, alertname: KubeContainerOOMKilled, key: KubeContainerOOMKilled shop }
These cases judge one message at a time. They do not evaluate the thread per alert, the cooldown, the close or the limits. Against your own channel, replay its real history through the trigger with a dry run before you rely on it.
The recipe also holds quality scenarios, one for each seeded fault and one for the healthy app, as data for a person to read an agent's answer against. Nothing runs them yet, and their shape may change. One of them:
# A quality scenario of sre-triage, as data. The shape is provisional: nothing runs it yet, and a runner will fix it.
# The Deployment `checkout` of ../../agent/faults.yaml is the case; a person reads the agent's answer against `cause` and `evidence`.
name: checkout crashes at start
app: checkout
alert: KubePodCrashLooping
cause: 'checkout cannot resolve its dependency payments-db.shop.svc.cluster.local (no such host), logs a FATAL line and exits 1, so the container restarts in a loop. No Service called payments-db exists in shop.'
evidence:
- 'the pod waits in CrashLoopBackOff with restarts counted'
- 'the log line FATAL: cannot resolve payments-db.shop.svc.cluster.local: no such host; refusing to start'
- 'there is no payments-db Service in shop'
Start investigations from a CI bot's failure posts¶
The custom-bot-trigger recipe is for a bot that has no preset: Alertmanager is the only one with a preset. A CI
system posts a message to a channel when a pipeline fails, and the recipe starts an investigation in a thread of its
own for each pipeline and branch. It holds Agent Kourier's resources only and ships no agent, so the backend's url
has to point at an A2A agent you run. The trigger is written in full, as in
Start investigations from any bot's messages: match and exclude pick out the bot's failure
posts, extract pulls out a state, a pipeline and a key, thread gives one thread to each key and closes it when the
state is PASSED, limits bounds how many investigations start, and promptTemplate writes the first prompt.
The message shape is made up
The regexes read a post that the recipe invented, and no real product posts it. Replace every regex with one
written against your own bot's messages, fetched as JSON (conversations.history returns them).
Before you start¶
- A Slack app with the bot invited to the channel: Configure Slack. You need the channel's ID and
the
bot_idof the CI bot's posts. - An A2A agent, and its URL for the AgentBackend. For kagent, set
dialect: kagent-v1as in Connect a kagent agent.
Agent Kourier's resources¶
The recipe has three files, in the namespace agent-kourier: the Binding below, an AgentBackend, and a
ChatConnection that names the Secret slack-ci. Replace CREPLACEME and B0REPLACEME.
# A CI bot's failure posts start an investigation in the channel, one thread per pipeline and branch.
# The trigger is written by hand: there is no preset for this bot. The regexes read a made-up message shape;
# replace them with ones written against your own bot's messages (see the README).
apiVersion: agentkourier.dev/v1alpha1
kind: Binding
metadata: { name: ci-failures, namespace: agent-kourier }
spec:
agent:
backendRef: { name: ci-agents }
name: ci-triage # on the a2a dialect the backend URL alone selects the agent and these are labels;
namespace: agents # on kagent-v1 they select it. The Binding needs them either way
identity: { userId: agent-kourier-ci }
chat:
connectionRef: { name: slack-ci }
channel: CREPLACEME # the channel the CI bot posts to; replace it with your channel's ID
triggers:
- type: botMessage
from:
botId: B0REPLACEME # the CI bot's bot_id; replace it with your bot's ID
match: # every entry must match. The title entry is there for the dry run: extract.state
# rejects the same titles, but a miss in match is the one the dry run names first
title: '^Pipeline (?:FAILED|PASSED) - '
fallback: '^\[(?:FAILED|PASSED)\] '
exclude: # no entry may match: a failure that will be retried, or that is known flaky
attachmentText: '(?i)\bretry \d+ of \d+ scheduled\b|\bknown flaky\b'
extract: # each regex has a named group with the entry's name
state:
field: title
regex: '^Pipeline (?P<state>FAILED|PASSED) - '
pipeline:
field: title
regex: ' - (?P<pipeline>[A-Za-z0-9_.-]+)$'
key:
field: fallback
regex: '^\[[A-Z]+\] (?P<key>[A-Za-z0-9_.-]+ on [^\s(|]+)'
thread:
by: key # messages with the same key share one thread
cooldown: 2h # no second investigation for a key within 2h of the first starting
maxIdle: 12h # a thread with no message of its key for 12h counts as abandoned
closeWhen: { state: PASSED }
limits: # without limits a trigger is unlimited
rateLimit: { maxRuns: 3, per: 15m, maxConcurrent: 1 }
maxRunsPerDay: 20
promptTemplate: |
A pipeline run is {{ .Fields.state }} in channel {{ .Channel }}: {{ .Fields.pipeline }}.
Find out why it failed with the tools you have. Report the likely cause, the evidence you
checked, and the next steps a person should take.
With no limits, a trigger that has no preset is unlimited, so a storm of failing pipelines would start as many
investigations. The recipe's README has the commands to create the Secret, load the directory into the chart and
replay the channel's history through the trigger first.
Check it¶
go test ./test/recipes loads the three resources with Agent Kourier's own loader and replays made-up posts through
the trigger, each judged alone. The first is a failure that matches, with the values it extracts:
cases:
- name: a failed pipeline
binding: agent-kourier/ci-failures
message:
bot: B0REPLACEME
title: 'Pipeline FAILED - web-build'
titleLink: 'https://ci.example.com/pipelines/web-build/runs/1423'
fallback: '[FAILED] web-build on main (#1423) | https://ci.example.com/pipelines/web-build/runs/1423'
attachmentText: 'Stage test failed after 4m12s. 3 tests failed.'
footer: 'ci-bot'
expect:
outcome: matched
trigger: 0
fields: { state: FAILED, pipeline: web-build, key: web-build on main }
The pipeline and key regexes take only letters, digits and _ . -. A pipeline named team/web-build therefore passes
match, fails an extract, and so matches no trigger. Nothing starts and nothing is posted in Slack: the miss shows
in the metric agentkourier_trigger_unmatched_total, in a debug log line and in a dry run. A case
pins it:
# The pipeline and key regexes take only name-shaped characters. A pipeline name outside them passes match and
# state and then fails those two extracts, so the message matches nothing: no turn, no note in Slack.
- name: a pipeline name with a slash is unmatched
binding: agent-kourier/ci-failures
message:
bot: B0REPLACEME
title: 'Pipeline FAILED - team/web-build'
fallback: '[FAILED] team/web-build on main (#1431) | https://ci.example.com/pipelines/team/web-build/runs/1431'
expect: { outcome: unmatched }
The cases do not evaluate the cooldown, thread keys across messages, limits or closeWhen.