Skip to content
Documentation

Annotate an existing image

Some screens cannot be reached by a script: a sign-in with a hardware key, a browser extension's own interface, a native dialog. This guide annotates a picture you took by hand, using the same callouts and the same publication step as every other recipe.

Step 1: put the image somewhere in the project

Take the screenshot however you like and save it in the repository. PNG, JPEG and WebP are all accepted, and the format is read from the file's contents rather than its name.

Step 2: measure the regions

There is no page to search, so marks are literal rectangles rather than queries. Open the image in any editor that shows pixel coordinates and note, for each region you want to point at, its [x, y, width, height] in image pixels.

Step 3: write the recipe

Set source: file and give file the path to the image, resolved from the configuration file's folder:

screenshots/recipes/billing-settings.yaml 
name: billing-settings
source: file
file: captures/billing-settings.png
install: guide

marks:
  plan: { rect: [866, 874, 150, 50] }
  seats: { rect: [866, 948, 150, 50] }

callouts:
  - { mark: plan, text: The current plan, place: top }
  - { mark: seats, text: How many people it covers, place: bottom }

Run it exactly as you would any other recipe. Callouts, numbered discs, style overrides, install and --check all behave the same way.

Number the regions instead

screenshots/recipes/billing-settings.yaml 
name: billing-settings
source: file
file: captures/billing-settings.png

marks:
  plan: { rect: [866, 874, 150, 50] }
  seats: { rect: [866, 948, 150, 50] }

numbered: [plan, seats]

Cover something in the image

mask also takes literal rectangles here:

screenshots/recipes/billing-settings.yaml 
name: billing-settings
source: file
file: captures/billing-settings.png

mask:
  - { rect: [120, 96, 320, 40] }

marks:
  plan: { rect: [866, 874, 150, 50] }

callouts:
  - { mark: plan, text: The current plan }

Notes

  • Playwright is still required. The callouts are drawn in a browser page, even though no site is opened.
  • retries has no effect on these recipes. There is no live page to be flaky, so a second attempt would reach the same answer.
  • The coordinates are in the image's own pixels. If you re-take the picture at a different size, re-measure.

Related