Miguel/observe a11y

[miguelg719]

why

Including a new format to get website context using a11y trees. It unlocks a new paradigm in processing context for web agents leveraging the full potential of CDP/playwright. By using the structured, semantic data from a11y trees, this approach aims to improve interaction fidelity, reduce token cost, speed up inference, and optimize contextual awareness when LLMs perform tasks in web-based environments through text.

what changed

Context is now provided optionally with the flag useAccessibilityTree for observe tasks. This changes the way DOM is processed by using a11y trees. The DOM function is still used for backward compatibility with selector maps, but I also include a backendNodeId which is another approach for directly interacting with elements through CDP.

Sample usage:

 const observations = await stagehand.page.observe({
   instruction: "Find all the links on the header section",
   useAccessibilityTree: true
 });

Sample output:

[
  {
    description: 'Sample link',
    selector: 'xpath=/html/body[1]/div[1]...',
    backendNodeId: 60
  },
...

test plan

• Test against existing observe evals
• Generate new evals (new approach) for various observe tasks (currently only 5)
• Test against new evals and quantify improvement in fidelity and cost/speed vs current approaches
• Include act functions for direct compatibility
• Test with act

[changeset-bot]

🦋 Changeset detected

Latest commit: 0861e9c

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@browserbasehq/stagehand	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[kamath]

Can we put raw CDP as a stagehandPage function? i wanna do something like this.stagehandPage.sendCDP(cmd, args) instead.

[kamath]

🚀 🚢