Learn from a demonstration
The user did a task on your screen while you watched. The hub saved what happened in /data/demos/<id>/:
NNNN.png: a screenshot every ~2 seconds, only when the screen changed. File names are in order.log.jsonl: one line per frame,{"t": <ms since start>, "frame": "NNNN.png", "url": "<Chrome's URL or null>"}.
Steps
- Read
log.jsonlfirst. The URL changes tell you which sites and pages the task went through and roughly where each step starts. - Look at a sample of frames with
read_file: the first and last, and the frames around each URL change or long pause. Don't read every frame. A frame over 3 MB doesn't show: shrink it into a separate file first (for example Python/Pillow) and read that one. - Work out the goal (what the user wanted done) and the steps that got there: where they went, what they clicked, what they typed and what they checked before moving on.
- If a step is unclear (why they picked one option, where a value came from, whether a step is always needed), ask the user one short question with what you saw. Don't guess.
- Write the skill so a fresh run can follow it from scratch:
- Steps in order, each with what to look for on the page, not screen coordinates.
- Values that change each time (dates, amounts, names, files) become inputs the user gives or you work out, with how to find them.
- No passwords, codes, card numbers or other secrets, and no one-off values from this recording. Where a login or sensitive field comes up, say to use
request_formorrequest_login. - How to tell it worked.
- Call
propose_skillwith it. The user saves it or not.
Don't redo the task while learning it unless the user asks.