For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

Follow an agent’s Actor across suspend and resume by using the Actor identity that Agent Substrate adds to its logs, traces, and metrics.

Every kagent agent runs as an ActorActorThe sandboxed unit of compute, provided by Agent Substrate, that runs a Session's conversation loop. Every Session is backed by one.Learn more on Agent Substrate. An Actor suspends after each turn, and can resume on a different WorkerWorkerA pre-started, sandboxed pod that hosts at most one Actor at a time. Actors are multiplexed across a pool of Workers over time.Learn more pod the next time, so telemetry that is keyed to a pod loses track of an agent between turns. To keep one agent’s telemetry together, Agent Substrate labels its telemetry with the identity of the Actor rather than the pod that ran it.

What Agent Substrate reports

Agent Substrate reports telemetry from its own components, separately from the telemetry that kagent reports for each agent request.

The Agent Substrate Helm chart sets one OTLP endpoint for traces, metrics, and logs through otel.endpoint, and samples traces at a ratio of 0.01 by default through otel.traces.samplingRatio. The OTel stack and Lightweight OTel stack guides set both.

The router access log and the Actor events are the Agent Substrate logs over OTLP, and the otel.logs settings control them together. To send them to a different backend than traces and metrics, set otel.logs.endpoint. To stop exporting them, set otel.logs.enabled to false. The chart has no setting for the access log alone.

When the Agent Substrate release sets an OTLP endpoint, the component that records an Actor event exports the event and does not write it to stdout. Read the events from your logging backend.

Actor identity

Agent Substrate adds the following labels to Actor logs and to its component logs about an Actor. Each label maps to a kagent object, so you can find an agent’s telemetry from the Session or the Agent that you already know.

Agent Substrate metrics carry template and WorkerPool labels rather than the Actor identity, so that the number of time series does not grow with the number of conversations. To look into one agent, use its logs.

Before you begin

1. Install kagent, including the kubectl-ate plugin that the installation guide describes.
2. Create your first agent and send it at least one message. This setup gives the my-first-agent Agent an Actor with logs to read. The guide also installs the kagent CLI.
3. Install jq, to read the Session ID and filter the JSON log records.

Read an Actor’s logs

An Actor writes to the stdout of whichever Worker pod runs it, and a suspended Actor is assigned to no Worker at all. Find the Actor by its Session ID, then read the Worker pods of its WorkerPool and filter by the Actor’s name.

1. Save the ID of the Session that you want to follow. Use an existing Session ID in place of this command when you already have one.

   export SESSION_ID=$(kagent agent session create --agent my-first-agent -o json | jq -r '.session.id')
   echo $SESSION_ID

   A value of null means that the Session was not created. The CLI lists only the Sessions that you created, so confirm that you are using the identity that created the agent.

2. Confirm that the Actor exists, and check its state. The Actor name is session- followed by the Session ID.

   kubectl ate get actors session-$SESSION_ID --atespace kagent

   Example output:

   ATESPACE   NAME                                      TEMPLATE                                              STATE                   WORKER POD   WORKER IPS   VERSION   AGE
   kagent     session-01a0d409-d249-728e-865c-ee58b55dff04   kagent/my-first-agent-a995d20d30a7   ACTOR_STATE_SUSPENDED   <none>                   5         32m
   

   ACTOR_STATE_SUSPENDED with a WORKER POD of <none> is the resting state between turns. Because a suspended Actor names no Worker, the remaining steps read every Worker in the pool rather than one pod.

3. Find the WorkerPool that the Harness runs on.

   export WORKER_POOL=$(kubectl get harness my-first-harness -n kagent \
     -o jsonpath='{.spec.substrate.workerPoolRef.name}')
   echo $WORKER_POOL

   Example output:

   kagent-default
   

4. Read the Actor’s lines from the Workers in the pool. A Worker writes the output of every Actor that it runs, so the filter selects a single Actor by name. Worker output mixes JSON records with plain text, so the filter also drops any line that is not a JSON object.

   kubectl logs -n kagent -l ate.dev/worker-pool=$WORKER_POOL --tail=-1 \
     | jq -cR --arg actor "session-$SESSION_ID" \
       'fromjson? | objects | select(.labels["ate.actor.name"] == $actor)'

   Example output:

   {"labels":{"ate.actor.name":"session-01a0d409-d249-728e-865c-ee58b55dff04","ate.actor.uid":"c91579a8-675c-474c-9437-4482ac628774","ate.atespace":"kagent","ate.template.atespace":"kagent","ate.template.name":"my-first-agent-a995d20d30a7"},"message":"Actor restoring","span_id":"d81bb304af04c945","time":"2026-09-24T16:02:52.008842377Z","trace_flags":"00","trace_id":"3b8987f98057610937b38639136d32b5"}
   {"labels":{"ate.actor.name":"session-01a0d409-d249-728e-865c-ee58b55dff04","ate.actor.uid":"c91579a8-675c-474c-9437-4482ac628774","ate.atespace":"kagent","ate.template.atespace":"kagent","ate.template.name":"my-first-agent-a995d20d30a7"},"message":"Actor checkpointed","span_id":"4914db77fcc2f59b","time":"2026-09-24T15:30:09.671009219Z","trace_flags":"01","trace_id":"c4971ca2e614428819443f64fe70f81c"}
   

   The filter returns the Actor’s lines from every Worker that currently runs in the pool, including the lines from before the Actor last suspended.

kubectl logs reads only the Worker pods that exist now. When a Worker pod is replaced, the output that it held is gone. To keep an Actor’s history across Worker replacements, collect the stdout of the Worker pods with a log agent that stores structured JSON, and query it by the same labels. The OTel stack and Lightweight OTel stack guides configure their collectors with only the OTLP receiver. Neither collector reads pod stdout.

Agent Substrate can also stream a single Actor’s output directly, without the pool-wide filter.

kubectl ate logs actor session-$SESSION_ID --atespace kagent

The command reads the Worker that the Actor runs on, so it works only while the Actor is running. A kagent agent runs only during a turn, and the command returns an error the rest of the time.

Error: actor kagent/session-01a0d409-d249-728e-865c-ee58b55dff04 is not currently running on any worker pod

Suspend and resume records

The records that Agent Substrate writes for an Actor mark each step of a suspend and resume cycle. Each record carries the Actor labels and a message field, and no container label. To list only these records for an Actor, filter out the lines that have a container label.

kubectl logs -n kagent -l ate.dev/worker-pool=$WORKER_POOL --tail=-1 \
  | jq -cR --arg actor "session-$SESSION_ID" \
    'fromjson? | objects | select(.labels["ate.actor.name"] == $actor and .labels["ate.actor.container.name"] == null) | {time, message}'

Example output:

{"time":"2026-09-24T15:30:08.965994177Z","message":"Actor restoring"}
{"time":"2026-09-24T15:30:09.04801126Z","message":"Actor restored"}
{"time":"2026-09-24T15:30:09.582808844Z","message":"Actor checkpointing"}
{"time":"2026-09-24T15:30:09.671009219Z","message":"Actor checkpointed"}
{"time":"2026-09-24T16:02:52.008842377Z","message":"Actor restoring"}
{"time":"2026-09-24T16:02:52.093494544Z","message":"Actor restored"}
{"time":"2026-09-24T16:02:52.481758336Z","message":"Actor checkpointing"}
{"time":"2026-09-24T16:02:52.588873294Z","message":"Actor checkpointed"}

Each turn produces one cycle of the four records. The time between Actor restoring and Actor restored is how long the agent took to wake up for the turn.

The Actor that builds a pair’s first snapshot, in the ate-golden atespace, writes Actor starting and Actor started instead of the restore records, because it has no snapshot to restore from.

Each record also carries trace_id, span_id, and trace_flags, which join the record to the Agent Substrate trace for the same operation. A trace_flags value of 01 means that Agent Substrate sampled the trace and exported it. A value of 00 means that the trace was not sampled, which is the case for most requests at the default ratio of 0.01. A sampled trace reaches your tracing backend only if your collector keeps the Agent Substrate spans.

Actor state changes

The Agent Substrate API server, ateapi, writes an Actor state changed event each time an Actor moves to a new state. The event includes the Actor identity labels, the operation that caused the change in ate.actor.operation.name, and the new state in ate.actor.state. The last event for an Actor tells you its state, and when the Actor entered it.

Where the events go depends on whether the Agent Substrate release sets an OTLP endpoint. When the release sets an endpoint, ateapi exports each event as an OTLP log with the same attributes. A logging backend then receives the state changes of every Actor without reading pod output. Do not sample this stream. A dropped event leaves the last known state wrong, with nothing to show that an event is missing. In your logging backend, select the records with the body Actor state changed, and filter them by the ate.actor.name attribute.

Read state changes from stdout

When the Agent Substrate release does not set an OTLP endpoint, ateapi writes each event to stdout. ateapi runs more than one replica, and each replica writes only the changes that it handles. Read the logs of every replica.

for pod in $(kubectl get pods -n ate-system -o name | grep ate-api-server); do
  kubectl logs -n ate-system "$pod"
done | grep '"msg":"Actor state changed"' | grep "session-${SESSION_ID}"

Example output:

{"time":"2026-09-24T17:00:43.007925845Z","level":"INFO","msg":"Actor state changed","ate.atespace":"kagent","ate.actor.name":"session-01a0d45b-c729-7d67-85c6-daa053202bff","ate.actor.uid":"831c62c5-b5ac-411f-b15e-7addfef97d73","ate.template.atespace":"kagent","ate.template.name":"my-first-agent-699a5ed2f709","ate.actor.operation.name":"resume","ate.actor.state":"resuming","trace_id":"402701838ffd50932dc0aec8e43a1604","span_id":"1f8f032ff37422c7","trace_flags":"00"}
{"time":"2026-09-24T17:00:43.194765137Z","level":"INFO","msg":"Actor state changed","ate.atespace":"kagent","ate.actor.name":"session-01a0d45b-c729-7d67-85c6-daa053202bff","ate.actor.uid":"831c62c5-b5ac-411f-b15e-7addfef97d73","ate.template.atespace":"kagent","ate.template.name":"my-first-agent-699a5ed2f709","ate.actor.operation.name":"resume","ate.actor.state":"running","trace_id":"402701838ffd50932dc0aec8e43a1604","span_id":"f92aecd47d89ad68","trace_flags":"00"}

Actor usage samples

The Worker records an Actor usage sampled event each time an Actor starts running and again when it stops. Each event includes the Actor identity labels, the WorkerPool, the Actor’s CPU time, and its memory use, peak memory, and working set. The ate.stats.kind attribute is initial for the first sample of a run and final for the last.

Learn more

The Agent Substrate project documents its telemetry in full, including the complete metric registry, in Actor observability. That page describes the newest Agent Substrate, which can be ahead of version 0.5.0-alpha2 that this guide uses. Records that it describes might not exist in your version.

You can also check out the following resources in this documentation set.