Skip to main content

What you will have

A roster agent called Risk Agent that runs on a cron schedule, reads new Radar reviews and velocity alerts since its last run, and posts one review packet per alert to #risk-review. Each packet lists every related payment attempt in the window with Stripe ids, names the shared attribute that links the activity, cites prior cases, and ends with a recommendation of clear, monitor or block. The agent never blocks a customer, never refunds, never edits a Radar rule. A block recommendation waits for an admin to approve it before it is written into the packet. Every run is graded against three categories, and the scorecard shows the hit rate per category. This guide follows the six steps of Build an agent in order. Guardrails come last, after the agent has proven on five historical alerts that it can assemble a correct packet with read-only credentials.

The agent spec

1. Define the job

Read Define the job first. The job here has a clear input (one alert), a clear output (one packet), and a hard boundary (no enforcement).
1

Create the roster agent

Go to Agents and click New Agent. On Profile name it Risk Agent. On Instructions, use this description:
Sweeps new Stripe Radar reviews and velocity alerts, assembles a review packet with related attempts, linked accounts and prior cases, and recommends clear, monitor or block. Never enforces.
And these system instructions:
Skip Triggers for now and click Create agent.
2

Point it at a template

Open the agent’s card. Under Default template pick Risk Review (created in step 3; come back and set it if the template does not exist yet). Leave Coding agent on Claude Code. Click Save profile.

2. Measure success

Read Measure success. The grader routes each run to one category and returns pass or fail with a reason. Because the historical alerts already have analyst decisions, the success criteria can name the decision.
Performance evaluation section with a category, grading tag, human cost and human time

Performance evaluation on the agent's Profile tab. One category per alert type, each with success criteria the grader reads verbatim.

1

Add three evaluation categories

On the agent’s Profile tab, scroll to Performance evaluation and click Add evaluation category three times.On each category add one grading tag, recommendation, with the instruction “The recommendation the packet ends with: clear, monitor or block”. Set Avg human cost (USD) to 45 and Avg human time (minutes) to 40.
2

Set the monthly budget

Under Monthly budget enter 600. This is a flag on the scorecard and an audit event when spend passes it. It does not stop runs; the hard cap lives in org limits, covered in step 6. Click Save profile.

3. Give it tools

Read Give it tools, especially the credential hierarchy. Three systems, all read-only.
1

Create the Risk Review template

Go to Templates, click New Template, choose Blank environment, name it Risk Review, keep Basic and click Create Template. Nothing is cloned; the template exists to carry connections and the skill.
2

Stripe, as an agent-scoped connection

Stripe is not in the built-in catalog, but it has a ready-made static schema (one field, STRIPE_SECRET_KEY). If your org has not created it yet, define it once under Settings > Context > Tools & MCP, New tool or MCP server, Tool, then define a custom provider named stripe with a single secret field materialized as STRIPE_SECRET_KEY.In the Stripe dashboard create a restricted key with read permission on Charges, Payment Intents, Customers, Reviews and Radar early fraud warnings, and nothing else. Back in Runtime, connect the provider and choose the agent scope, picking Risk Agent. An agent-scoped connection is used only by runs attributed to Risk Agent and beats any personal or org-wide Stripe key. It also skips the skill filter, so the support team’s org-wide Stripe key stays theirs and this key stays the risk team’s.The CLI creates only org and personal connections. Agent scope is set in the dashboard, or with scope: "agent" and agent_id on POST /api/cloud/knowledge/integrations.
3

BigQuery, org-wide, one dataset

BigQuery is a built-in provider. Pick it from Add a provider, choose the Service account JSON method, and paste a service account key whose IAM grant is BigQuery Data Viewer on the fraud dataset only plus BigQuery Job User on the project. Scope: Team. The provider installs gcloud and bq into the snapshot and activates the service account at session start.BigQuery is used here because it is a built-in and the fraud warehouse in this example lives in it. If your warehouse is Snowflake, define it as a custom provider with account, user and a read-only role, exactly as Stripe above.
4

Postgres, org-wide, read-only role

Postgres also has a ready-made static schema (one field, a connection string). Create the provider if it does not exist, then connect it with a connection string for a role that can only SELECT from prior_cases and the account views. Scope: Team.
Org-wide and personal connections reach a session only when an attached skill requires the provider. The skill in the next step lists all three, so BigQuery and Postgres arrive through the skill and Stripe arrives through the agent scope.

4. Define how it works

Read Define how it works. The runbook is a skill, the trigger is a schedule, and the prohibitions stay in text until step 6.
1

Write the fraud-review-packet skill

Under Settings > Context > Skills, click New skill and create fraud-review-packet with this SKILL.md:
Add references/packet-template.md:
Set Requires integrations to stripe, bigquery, postgres. Then attach the skill to the Risk Review template.
2

Build the template once

On Templates, open Risk Review and click Build on the Build tab. Wait for ready.
3

Create the schedule, disabled

On Agents, open the Scheduled tab and click New scheduled agent. Fill Name this scheduled agent with Risk sweep. For the prompt, write:
Run the fraud-review-packet skill for every Radar review, early fraud warning and fraud.alerts row created since the last run. Post one packet per alert. If there are none, say so.
Set run-as to Agent and pick Risk Agent, so the run carries the agent’s identity: the agent-scoped Stripe key applies and the run is graded against the agent’s categories. Set the schedule to Interval, Every 30 minutes. Under Post, pick the Slack integration and the #risk-review channel. Choose Team visibility. Create it, then flip the toggle off so it is disabled until step 5 proves it.
New scheduled agent dialog with the prompt field and the harness, template, schedule and post controls

The scheduled agent dialog: prompt first, then coding agent, template, schedule and post target along the bottom.

The cron expression Runtime stores is five fields in UTC. Every 30 minutes is written as minute zero and minute thirty of every hour, not with a slash-step, because the slash form cannot appear inside this page’s hidden CLI comments. The CLI can create the schedule with a harness and template but cannot set run-as Agent; set that in the dialog.
4

Optional: a Slack mention for ad hoc alerts

Analysts sometimes want “look at this one now”. In the agent’s sheet add a Slack trigger so @Risk Agent <review id> in #risk-review runs the same skill on one alert. The mention runs as Risk Agent too, so the same credentials and grading apply.

5. Prove it works and iterate

Read Prove it works and iterate. No guardrail exists yet. The agent is safe because every credential is read-only, the sandbox is destroyed after each run, and the schedule is disabled so nothing runs unless you press Run now.
1

Seed five historical alerts

Pick five alerts from the last quarter whose analyst decisions you know. Write the expected outcome down before running anything.Case E exists to prove the agent can say “nothing links this”. Cases A and D exercise the approval step.
2

Run each case with Run now

For each case, temporarily set the schedule prompt to name the alert (Run the fraud-review-packet skill for Radar review prv_...), then click Run now on the schedule’s row. Run now executes the identical code path a cron tick takes, so a failure here is the failure you would otherwise get at 03:00 UTC.For cases A and D the session will stop in awaiting_approval. Open Sessions, find the run, and click Approve or Reject on its card. Try both across the two cases to see the packet change.
3

Read the grade and the packet

Open each run under Agents > Runs. Read the packet against the template: every attempt with a Stripe id, the shared attribute named, prior cases cited. Then read the grade.A typical failure on the first pass:
That is a runbook gap, not a rubric gap. Add the baseline query to step 3 of the skill, rebuild, re-run case C.
4

Read the scorecard

Open Activity > Agents and filter to Risk Agent over the last 7 days. You want five graded runs, a per-category breakdown under tasks, and a hit rate you can explain case by case.
Activity page, Agents tab, showing per-agent performance and budget

Activity > Agents: graded runs, objective hit rate, value returned and the monthly budget per agent.

5

Tighten one thing, then re-run

Change one input per iteration. If the grader passed a packet you would reject, tighten the success criteria. If the agent skipped a step, add it to the skill. If it could not find data, the credential or the dataset grant is too narrow. Re-run the affected case after every change; grades are not recomputed retroactively.Write down the exact commands and hosts the five passing runs used. That list is the input to step 6. Restore the sweep prompt before moving on.

6. Add guardrails and approvals

Read Add guardrails and approvals. This agent runs unattended, so it gets the full set: allowlist rules, network rules, and the approval step you already exercised.
1

Deny every write, from what you saw

Under Settings > Guardrails > Allowlists, click New rule for each. These patterns match Bash commands the runbook could plausibly attempt.Then set the Default policy to Ask, so any command outside these patterns pauses the run instead of running.
2

Pin the network

Under Network, add one rule per host. As soon as one rule exists, egress is deny-all plus this list. Slack posting is done by Runtime after the run, not by the sandbox, so hooks.slack.com is not needed.
Guardrails page, Network tab, with host rows

Network rules turn egress into deny-all plus the listed hosts.

3

Rebuild, then re-run one case per category

Allowlist rules are materialized into the snapshot, so rebuild the template. Network rules apply at session creation without a rebuild. Then Run now cases A, C and D again. If a case that passed in step 5 now stalls on an ask or fails, the last rule you added is the cause: remove it, re-run, and write it narrower.
4

Set the org hard cap

Under Guardrails > Limits, set the org monthly budget and maximum concurrent sessions. These are enforced at session creation, unlike the agent’s $600 budget, which only flags.
5

Enable the schedule

Flip the toggle on the Risk sweep row. Check next_run_at is in the future. After the first tick, confirm a packet (or “No new alerts”) landed in #risk-review and a graded run appears under Runs.

What stays human

  • Every enforcement action. Blocks, refunds, limit changes and Radar rule edits happen in Stripe by an analyst, using the packet as evidence. The agent’s credential cannot do them and the allowlist denies the commands anyway.
  • Every block recommendation. The runbook requests an approval with --required-role admin and waits. The request appears on the session card under Sessions and in runtm-api session approvals list. It is not delivered to Slack; someone with the admin role resolves it on the board or with the CLI. Rejected means the packet says monitor and quotes the note.
  • The rubric. When the grader and the analyst disagree, the analyst is right and the success criteria change.

Gotchas

  • Run-as Harness instead of Agent. A schedule created in Harness mode runs with no agent identity: the agent-scoped Stripe key never applies, the run is not graded, and the scorecard stays at zero. Check the schedule row says “Runs as Risk Agent”.
  • Skill attached, template not rebuilt. template get shows attachments_changed_since_build: true until you build. The first sweep then runs without the skill and posts an improvised packet.
  • BigQuery works in the terminal but not in the run. The org-wide connection reaches the session only because the skill requires bigquery. Remove it from requires.integrations and the connection is silently filtered out.
  • A network rule set without app.runtm.com. The sandbox loses the runtm CLI, so runtm-approval fails and every block case errors instead of waiting.
  • Enabled before proven. An enabled schedule with last_run_at null and next_run_at in the past never ran. Disable, Run now, read the session, enable.
  • The cron step form. Runtime accepts */30 in the dialog, but the hidden CLI comments on this page use 0,30 * * * * because the slash form ends an MDX comment early. Both mean every 30 minutes.
  • Budget exceeded keeps running. $600 is a scorecard flag. The org limit in step 6 is the cap.

Payment support agent

A Slack and Email agent that reconstructs a charge and drafts a reply, never sends.

Merchant underwriting agent

A Linear-triggered agent that assembles a KYB decision memo and leaves the decision to underwriters.

Build an agent

The six steps this example follows.