Trillion

Operate the person's computer screen

Click, press, type, scroll, drag or use keys on the person's computer, finding the target by what it is called and checking afterwards what really changed

One call looks at the screen, finds the target, does the action, looks again and says what changed. action is one the computer keeps (tool:screen-actions lists them; new ones are added as data on the computer). Give expect, a list of { target, min?, max? } or { changed }, and verified says whether the screen shows it; without expect, verified is null and the change seen is reported. dry_run shows the steps without doing them. Press and set_value act through the accessibility tree and do not move the person's pointer; click, type and key do. A target that moved or vanished is refused, never guessed. Uses this Mac's existing Accessibility and Screen Recording permissions. The person can turn on an optional grant policy in the computer's settings; calls cannot change that policy.

What it asks for

  • action (Needed, words): An action id from tool:screen-actions.
  • target (Optional, value): What to find on the screen: a label string, or { match: { label|role|title|description|value|identifier|text: a string or { equals | contains | starts_with | regex } }, source?: ax or ocr, nth? }. Never coordinates.
  • to (Optional, value): What to find on the screen: a label string, or { match: { label|role|title|description|value|identifier|text: a string or { equals | contains | starts_with | regex } }, source?: ax or ocr, nth? }. Never coordinates.
  • text (Optional, words):
  • value (Optional, words):
  • keys (Optional, words): A key or combination, such as return or cmd+s.
  • dx (Optional, number):
  • dy (Optional, number):
  • expect (Optional, list):
  • app (Optional, words): The running app to read (its name or bundle id); the app in front when left out.
  • dry_run (Optional, yes or no):

Use it from your AI

Connect the AI you already use to Trillion once. Then ask in your own words; your AI finds this tool by what it does and runs it.

Say to your AI: “Use Trillion to click, press, type, scroll, drag or use keys on the person's computer, finding the target by what it is called and checking afterwards what really changed”

What your AI calls: run_tool {"tool_id":"tool:drive-the-screen"}