Find a thing on the person's computer screen
Say where a described thing is on the person's screen right now (a button, a field, a line of text) without touching it
Describe the target by what it is called or what kind it is, never by coordinates. Returns the ranked matches with where each is now. Read-only. Uses this Mac's existing Accessibility and Screen Recording permissions. The person can turn on an optional grant policy in the computer's settings; calls cannot change that policy.
What it asks for
- target (Needed, value): What to find on the screen: a label string, or { match: { label|role|title|description|value|identifier|text: a string or { equals | contains | starts_with | regex } }, source?: ax or ocr, nth? }. Never coordinates.
- app (Optional, words): The running app to read (its name or bundle id); the app in front when left out.
- sources (Optional, list): Where to read from: ax (the accessibility tree) and/or ocr (the pixels read as text).
Use it from your AI
Connect the AI you already use to Trillion once. Then ask in your own words; your AI finds this tool by what it does and runs it.
Say to your AI: “Use Trillion to say where a described thing is on the person's screen right now (a button, a field, a line of text) without touching it”
What your AI calls: run_tool {"tool_id":"tool:find-on-the-screen"}