Octo Onboarding Series (10): Record & Replay in Practice — Record Your Actions Once, Let octo Replay Them

The last post wired up the browser — fine for a one-off task where octo can just watch and click live. But if you do the same sequence every week, having the model re-observe-and-decide every single time is wasted work. This post covers recording it once so it can replay on its own — and fix its own selectors when they drift.

It records what you do, not what the model does

This is the single most important thing to understand about the mechanism: once you call record_start, octo hands the browser back to you. You perform the steps yourself, say “done,” and only then does it call record_stop <name> to save what happened. octo doesn’t drive the page itself during recording — it watches and takes notes, it doesn’t act.

Each step captures: the action type (click/type/select/upload/navigate, and so on), a selector anchored to the nearest ancestor element with a stable id (rather than a positional selector chain that breaks the moment the page layout shifts), that element’s visible text at the time, and the URL it happened on.

One click in the web UI

You don’t need to remember the record_start/record_stop action names — the “Browser” panel in the web UI has a “● Record” button. Clicking it opens a fresh chat whose first message spells the whole flow out for octo: confirm the browser is connected, ask what to record and what to name it, call record_start, hand control to you, and record_stop once you say you’re done.

Next to every saved recording in that same panel:

“▶ Replay” — calls run_skill by name, through the full agent path (the self-heal mechanism covered below only kicks in on this path — it’s not a server-side replay that bypasses the model).
“✎ Edit” — also conversational: it reads the recording’s YAML, lists the steps for you, and you say what to change (fix a selector, add/remove/reorder a step, set a param’s default), and it writes the file back without replaying it on your behalf.

What a saved recording looks like

The raw recording goes through one model pass — dropping detours that led nowhere, swapping the concrete values you typed for {{param}} placeholders, and writing a description — but this pass can only reorder or rename steps that were actually recorded, never invent a new one: any step whose selector isn’t in the original recording gets rejected outright, falling back to the raw step instead.

The distilled result is saved as plain YAML at ~/.octo/browser-skills/<name>.yaml — readable, hand-editable, diffable in git. Roughly:

name: submit-expense-report
description: Log into the expense system and submit a standard reimbursement form
params:
  - name: amount
    description: Reimbursement amount
    default: "128.00"
  - name: memo
    description: Reason for the expense
outputs:
  - name: receipt_url
    type: string
steps:
  - action: navigate
    url: https://expense.example.com/new
  - action: click
    selector: "#category-travel"
    label: Pick the "Travel" category
  - action: type
    selector: "#amount"
    hint: Reimbursement amount
    value: "{{amount}}"
  - action: type
    selector: "#memo"
    hint: Reason field
    value: "{{memo}}"
  - action: click
    selector: "#submit-btn"
    verify:
      text: Submitted successfully
  - action: extract
    js: document.querySelector('.receipt-link').href
    bind: receipt_url

params are replay-time inputs you can override; {{amount}}/{{memo}} in the steps get substituted with whatever’s passed in, falling back to default when nothing is. hint is that form field’s accessible name (placeholder/name/aria-label/id, or its associated <label> text) — when the positionally-recorded selector drifts, replay tries relocating the field by hint before handing off to self-heal (covered below). outputs declares the values this recording exposes to whoever calls it; a step like extract/download can bind its result to a named output via bind for a downstream step to consume.

What makes replay trustworthy

browser(action: "run_skill", name: "<recording-name>", params: { ... })

In the common case replay is deterministic and involves no model call at all: each step waits for its target to appear before acting, and runs a verify check if one was declared. A few robustness details worth knowing:

A click first looks for the exact selector from the original recording; if the page structure has drifted a bit but an element carrying that same recorded visible text still exists, it clicks that instead of failing outright.
If a step opens a new tab, subsequent steps automatically follow onto it.
If a field comes back empty right after typing into it, replay clears and retries once before actually calling the step a failure.

Replay is all-or-nothing by design — octo is explicitly instructed to only use a recording when the request matches it start to finish; it won’t execute a partial subset, or improvise beyond the declared params. Recordings are also replay-only: there’s no keyword trigger for this path (an earlier version tried one and it was pulled — not reliable enough).

When the selector is truly gone — self-heal

If a step’s selector and its hint both come up empty — and only when a model is configured specifically for this purpose — octo takes a plain-text summary of the page’s currently interactive elements (selectors + visible text, no screenshot), sends it along with the intended action, the expected label, and the selector that just failed, and asks the model to reply with nothing but a corrected CSS selector. The fix gets one retry, and on success it’s written straight back into the recording’s YAML file — so a self-heal is a permanent fix, not a one-time patch; every future replay skips re-healing the same thing.

Wiring it into a workflow

A recording’s outputs can be plugged straight into the workflow scripts from the previous post in this series via skill("browser:<name>", params) — record “log in and export the invoice,” bind the exported file’s path as an output, and hand it to another agent to parse and summarize in the next step, with no manual hand-off in between.

End of the series

Ten posts in: install, Skills, MCP, Loop, Cron, the weekly-report capstone, Workflow, Goal, Browser, and Record & Replay — covering both the “ask once, keep pushing on its own” spectrum and a second, completely different capability: going from “calling a tool” to “actually operating a web page for you.” Which one fits depends on the task in front of you: a one-off question just gets asked directly; something that needs repeated checking gets /loop; something on a schedule gets cron; something that splits into independent chunks gets workflow; something you can’t fully scope up front gets a goal; and anything with no API, only a UI, gets a browser connection you can drive live or record once and replay forever.

Previous in the series: Octo Onboarding Series (9): Browser in Practice — Hand octo Your Own Browser Back to the start: Octo Onboarding Series (1): Install It, Say Your First Word to It