See what is on the person's computer screen
Read the person's computer screen: the app in front (or one named), its controls from the accessibility tree and the text on its windows from the pixels, with where each is
Returns the app, the displays, and the things on the screen (role, label, centre and size, and the actions each offers) in points from the top left of the main display, never coordinates guessed from a picture. Text from the pixels is kept to the app's own windows. Read-only. Uses this Mac's existing Accessibility and Screen Recording permissions. The person can turn on an optional grant policy in the computer's settings; calls cannot change that policy.
What it asks for
- app (Optional, words): The running app to read (its name or bundle id); the app in front when left out.
- sources (Optional, list): Where to read from: ax (the accessibility tree) and/or ocr (the pixels read as text).
Use it from your AI
Connect the AI you already use to Trillion once. Then ask in your own words; your AI finds this tool by what it does and runs it.
Say to your AI: “Use Trillion to read the person's computer screen: the app in front (or one named), its controls from the accessibility tree and the text on its windows from the pixels, with where each is”
What your AI calls: run_tool {"tool_id":"tool:look-at-the-screen"}