Integration with OS accessibility APIs

[LexiconCode]

@lexxish @alexboche

Is your feature request related to a problem? Please describe.
Put simply re-implementing Dragon's Select-and-Say capability and text navigation.

Describe the solution you'd like
Work to explore tool that use accessibility API.
Key concepts

• Functions consumed by Grammars would be API, Engine, and OS agnostic
• As fast as possible
• The expectations this project would move out of Caster into dtactions Which will support Common OS action and related code from dictation-toolbox projects

Accessibility APIs
There can be overlap between applications with the respective OSs. I have only research tools for exploring the Windows OS

• Windows OS
  Microsoft Active Accessibility (MSAA)
  UI Automation

  • Tools
    Accessibility Insights for Windows (UIAutomation/win32) (best in class)
    aViewer 2.0.3 (MSAA/IA2/ARIA/HTML/UIAutomation)
    au3spy
    Inspect
    spy++ part of visual studio.
    winspy

• Linux
  AT-SPI
  IAccessible2 (IA2)

• Mac
  Accessibility API

Other APIs
Accessible Rich Internet Applications (ARIA)

Scintilla

MFC classes to encapsulate the Scintilla edit control

• naughter
• pywin scintilla control

Java Access Bridge is for Windows OS interface with Java software
https://doc.bccnsoft.com/docs/jdk7-docs/technotes/guides/access/jab/introduction.html
https://docs.oracle.com/javase/7/docs/technotes/guides/access/enable_and_test.html
https://docs.oracle.com/javase/7/docs/api/javax/accessibility/package-summary.html
https://github.com/nvaccess/nvda/blob/defed0ef2317cf3192922692a40cc2eb92d41143/source/JABHandler.py

Describe alternatives you've considered

Additional context

Dragonfly as a Accessibility API which is OS/controller agnostic, Due to the pyia2 that exposes Google Chrome, Firefox and Gitter(electron based on chrome) is limited.

The Dragonfly accessibility if you can be leveraged by this Caster grammar accessibilityapi.py

• Other resources
  https://www.autoitscript.com/forum/topic/153520-iuiautomation-ms-framework-automate-chrome-ff-ie/page/8/
  https://github.com/pywinauto/pywinauto/wiki/How-to-enable-accessibility-(tips-and-tricks)

Accessibility and Document Object Model (DOM) API Comparisons
accessibility.linuxfoundation.org/a11yspecs/atspi/adoc/a11y-dom-apis.html

How to proceed

I've been looking into the following as many of the accessibility API UI automation platforms.

WinAppDriver
robotframework
pywinauto
Python-UIAutomation

pyatspi2 for AT-SPI

The next step would be experiment integrating APIs and frameworks into Dragonflys Accessibility API. Staying in line with its design principles.

---

IssueHunt Summary

Backers (Total: $200.00)

• itsrainingschmen ($200.00)

Become a backer now!

Or submit a pull request to get the deposits!

Tips

• Checkout the Issuehunt explorer to discover more funded issues.
• Need some help from other developers? Add your repositories on IssueHunt to raise funds.

[caspark]

I cobbled this together as an example after @LexiconCode asked me about UI Automation after I mentioned it in the dictationtoolbox general channel: https://gist.github.com/caspark/36b1f2b47e37735e323863b1dda7458f

It's just a quick demo of fetching text from the control under the cursor.

[LexiconCode]

Some notes regarding using uiautomation PIP package.

• Prereqs
  • Python
    • Do notuse with 2.7.x, 3.7.6 or 3.8.1 due to comtypes issues. Recommended 3.8.2
    • python -m pip install uiautomation
• Uses with VS Code
  • Enable Accessibility Support in VS Code
    Accessibility Support typically used for (Screen Reader Optimizations) must be "on" not "auto" for uiautomation to work appropriatelly and then restart VS Code.
• Uses with Electron Based applications
  • Some executables require --force-renderer-accessibility flag which is used for chrome based applications. It is not needed with VS Code.

[LexiconCode]

Application Evaluation

Successful - UI Automation

These are applications that bear minimum of enabled grep text from various fields

• Visual Studio Code - with screenreader support enabled
• Microsoft Word(presumably other products)
• Git Extensions
• Texmaker
• Firefox
• Google Chrome
• Discord
• notepad
• Windows OS Explorer and so on.
• Mozilla Thunderbird

Unsuccessful

These are applications that have been unsuccessful grab text for various reasons.
Some of these applications may respond better to IAccessible2, scintilla or other APIs. Further research is required

• Adom - Electron
• Mark Text - Electron
• Typora - Electron
• Sublime - Electron
• gitter - Electron - crashes when inspected
• JetBrains - qt - has screenreader support but in me only activated when the screen readers present?
• notpad++ - scintilla
• SciTE - scintilla
• LibreOffice - IAccessible2 API on Windows, via GNOME Accessibility API (ATK) on Linux with GNOME, and via NSAccessibility on Apple OS X systems
  • When LibreOffice is launched and an AT is detected in use, accessible events will be detected and rendered as MSAA/IAccessible events without further action by the user. Additional adjustments to AT behavior can be made from the Tools ▸ Options ▸ Accessibility dialog, and a few only in the Tools ▸ Options ▸ Advanced panel's Expert Configuration dialog.

[isidorn]

Hello, VS Code dev here,

Please note for vscode accessibility support on should only be required on Linux. On windows and mac in theory it should work just fine with auto.
Also note this issue microsoft/vscode#98256
Which just fixed that it is possible to set caret selection on vscode on linux and mac. However this is only availalbe in vscode insiders in a couple of days. So in general I always recommend to use vscode insiders.

And let me know if we can help somehow from the VS Code side.

[LexiconCode]

Python-UIAutomation and pywinauto do not properly support functionality to fully grab the necessary information like carrot absolute/relative and so on.

This means were limited to improving 1 of the 2 above projects.
Other less explored are leveraging screenreader technologies. screen readers use the same information needed for text manipulation and are often implemented using OS specific accessibility APIs.

Orca or Nvda
nvda repo on GitHub https://github.com/nvaccess/nvda
Orca repo https://github.com/GNOME/orca

[LexiconCode]

I will update this thread this week there's been some significant changes.

Accessibility and Document Object Model (DOM) API Comparisons
https://accessibility.linuxfoundation.org/a11yspecs/atspi/adoc/a11y-dom-apis.html

[issuehunt-oss]

@itsrainingschmen has funded $200.00 to this issue.

---

• Submit pull request via IssueHunt to receive this reward.
• Want to contribute? Chip in to this issue via IssueHunt.
• Checkout the IssueHunt Issue Explorer to see more funded issues.
• Need help from developers? Add your repository on IssueHunt to raise funds.