whale.automation
Description: | The whale.automation API allows developers to access the automation (accessibility) tree for the browser. The tree resembles the DOM tree, but only exposes the semantic structure of a page. It can be used to programmatically interact with a page by examining names, roles, and states, listening for events, and performing actions on nodes. |
---|---|
Availability: | Dev channel only. Learn more. |
Manifest: | "automation": {...} |
## Types
### EventType
Possible events fired on an [automation.AutomationNode](/extensions/automation#type-AutomationNode).
| Enum |
|---|
| `"activedescendantchanged"`, `"alert"`, `"ariaAttributeChanged"`, `"autocorrectionOccured"`, `"blur"`, `"checkedStateChanged"`, `"childrenChanged"`, `"clicked"`, `"documentSelectionChanged"`, `"expandedChanged"`, `"focus"`, `"imageFrameUpdated"`, `"hide"`, `"hover"`, `"invalidStatusChanged"`, `"layoutComplete"`, `"liveRegionCreated"`, `"liveRegionChanged"`, `"loadComplete"`, `"locationChanged"`, `"mediaStartedPlaying"`, `"mediaStoppedPlaying"`, `"menuEnd"`, `"menuListItemSelected"`, `"menuListValueChanged"`, `"menuPopupEnd"`, `"menuPopupStart"`, `"menuStart"`, `"mouseCanceled"`, `"mouseDragged"`, `"mouseMoved"`, `"mousePressed"`, `"mouseReleased"`, `"rowCollapsed"`, `"rowCountChanged"`, `"rowExpanded"`, `"scrollPositionChanged"`, `"scrolledToAnchor"`, `"selectedChildrenChanged"`, `"selection"`, `"selectionAdd"`, `"selectionRemove"`, `"show"`, `"textChanged"`, `"textSelectionChanged"`, `"treeChanged"`, or `"valueChanged"` |
### RoleType
Describes the purpose of an [automation.AutomationNode](/extensions/automation#type-AutomationNode).
| Enum |
|---|
| `"abbr"`, `"alertDialog"`, `"alert"`, `"anchor"`, `"annotation"`, `"application"`, `"article"`, `"audio"`, `"banner"`, `"blockquote"`, `"button"`, `"buttonDropDown"`, `"canvas"`, `"caption"`, `"caret"`, `"cell"`, `"checkBox"`, `"client"`, `"colorWell"`, `"columnHeader"`, `"column"`, `"comboBox"`, `"complementary"`, `"contentInfo"`, `"date"`, `"dateTime"`, `"definition"`, `"descriptionListDetail"`, `"descriptionList"`, `"descriptionListTerm"`, `"desktop"`, `"details"`, `"dialog"`, `"directory"`, `"disclosureTriangle"`, `"document"`, `"embeddedObject"`, `"feed"`, `"figcaption"`, `"figure"`, `"footer"`, `"form"`, `"genericContainer"`, `"grid"`, `"group"`, `"heading"`, `"iframe"`, `"iframePresentational"`, `"ignored"`, `"imageMap"`, `"image"`, `"inlineTextBox"`, `"inputTime"`, `"labelText"`, `"legend"`, `"lineBreak"`, `"link"`, `"listBoxOption"`, `"listBox"`, `"listItem"`, `"listMarker"`, `"list"`, `"locationBar"`, `"log"`, `"main"`, `"mark"`, `"marquee"`, `"math"`, `"menuBar"`, `"menuButton"`, `"menuItem"`, `"menuItemCheckBox"`, `"menuItemRadio"`, `"menuListOption"`, `"menuListPopup"`, `"menu"`, `"meter"`, `"navigation"`, `"note"`, `"pane"`, `"paragraph"`, `"popUpButton"`, `"pre"`, `"presentational"`, `"progressIndicator"`, `"radioButton"`, `"radioGroup"`, `"region"`, `"rootWebArea"`, `"rowHeader"`, `"row"`, `"ruby"`, `"svgRoot"`, `"scrollBar"`, `"search"`, `"searchBox"`, `"slider"`, `"sliderThumb"`, `"spinButtonPart"`, `"spinButton"`, `"splitter"`, `"staticText"`, `"status"`, `"switch"`, `"tabList"`, `"tabPanel"`, `"tab"`, `"tableHeaderContainer"`, `"table"`, `"term"`, `"textField"`, `"time"`, `"timer"`, `"titleBar"`, `"toggleButton"`, `"toolbar"`, `"treeGrid"`, `"treeItem"`, `"tree"`, `"unknown"`, `"tooltip"`, `"video"`, `"webArea"`, `"webView"`, or `"window"` |
### StateType
Describes characteristics of an [automation.AutomationNode](/extensions/automation#type-AutomationNode).
| Enum |
|---|
| `"collapsed"`, `"default"`, `"editable"`, `"expanded"`, `"focusable"`, `"focused"`, `"haspopup"`, `"horizontal"`, `"hovered"`, `"invisible"`, `"linked"`, `"multiline"`, `"multiselectable"`, `"offscreen"`, `"protected"`, `"required"`, `"richlyEditable"`, `"selectable"`, `"selected"`, `"vertical"`, or `"visited"` |
### Restriction
The input restriction for a object -- even non-controls can be disabled.
| Enum |
|---|
| `"disabled"`, or `"readOnly"` |
### Rect
| properties |
|---|
| integer | left | |
| integer | top | |
| integer | width | |
| integer | height | |
### FindParams
| properties |
|---|
| [RoleType](/extensions/automation#type-RoleType) | (optional) role | |
| object | (optional) state |
A map of [automation.StateType](/extensions/automation#type-StateType) to boolean, indicating for each state whether it should be set or not. For example: `{ StateType.disabled: false }` would only match if `StateType.disabled` was _not_ present in the node's `state` object.
|
| object | (optional) attributes |
A map of attribute name to expected value, for example `{ name: 'Root directory', checkbox_mixed: true }`. String attribute values may be specified as a regex, for example `{ name: /stralia$/` }. Unless specifying a regex, the expected value must be an exact match in type and value for the actual value. Thus, the type of expected value must be one of:
* string
* integer
* float
* boolean
|
### SetDocumentSelectionParams
| properties |
|---|
| AutomationNode | anchorObject |
The node where the selection begins.
|
| integer | anchorOffset |
The offset in the anchor node where the selection begins.
|
| AutomationNode | focusObject |
The node where the selection ends.
|
| integer | focusOffset |
The offset within the focus node where the selection ends.
|
### AutomationEvent
| properties |
|---|
| [AutomationNode](/extensions/automation#type-AutomationNode) | target |
The [automation.AutomationNode](/extensions/automation#type-AutomationNode) to which the event was targeted.
|
| [EventType](/extensions/automation#type-EventType) | type |
The type of the event.
|
| string | eventFrom |
The source of this event.
|
| integer | mouseX | |
| integer | mouseY | |
| function | stopPropagation |
Stops this event from further processing except for any remaining listeners on [AutomationEvent.target](/extensions/automation#property-AutomationEvent-target).
|
### TreeChange
| properties |
|---|
| [AutomationNode](/extensions/automation#type-AutomationNode) | target |
The [automation.AutomationNode](/extensions/automation#type-AutomationNode) that changed.
|
| enum of `"nodeCreated"`, `"subtreeCreated"`, `"nodeChanged"`, `"textChanged"`, or `"nodeRemoved"` | type |
The type of change.
- nodeCreated
- * This node was added to the tree and its parent is new as well, so it's just one node in a new subtree that was added.
- subtreeCreated
- * This node was added to the tree but its parent was already in the tree, so it's possibly the root of a new subtree - it does not mean that it necessarily has children.
- nodeChanged
- * This node changed.
- textChanged
- * This node's text (name) changed.
- nodeRemoved
- * This node was removed.
### AutomationNode
| properties |
|---|
| [AutomationNode](/extensions/automation#type-AutomationNode) | (optional) root |
The root node of the tree containing this AutomationNode.
|
| boolean | isRootNode |
Whether this AutomationNode is a root node.
|
| [RoleType](/extensions/automation#type-RoleType) | (optional) role |
The role of this node.
|
| object | (optional) state |
The [automation.StateType](/extensions/automation#type-StateType)s describing this node.
|
| [Rect](/extensions/automation#type-Rect) | (optional) location |
The rendered location (as a bounding box) of this node in global screen coordinates.
|
| function | boundsForRange |
Computes the bounding box of a subrange of this node in global screen coordinates. Returns the same as |location| if range information is not available. The start and end indices are zero-based offsets into the node's "name" string attribute.
Returns [Rect](/extensions/automation#type-Rect).
| Parameters |
|---|
| integer | startIndex | |
| integer | endIndex | |
|
| string | (optional) description |
The purpose of the node, other than the role, if any.
|
| string | (optional) placeholder |
The placeholder for this text field, if any.
|
| string | (optional) roleDescription |
The role description for this node.
|
| string | (optional) name |
The accessible name for this node, via the [Accessible Name Calculation](http://www.w3.org/TR/wai-aria/roles#namecalculation) process.
|
| enum of `"uninitialized"`, `"attribute"`, `"attributeExplicitlyEmpty"`, `"contents"`, `"placeholder"`, `"relatedElement"`, or `"value"` | (optional) nameFrom |
The source of the name.
|
| string | (optional) value |
The value for this node: for example the `value` attribute of an ` element.`
|
| string | (optional) htmlTag |
The HTML tag for this element, if this node is an HTML element.
|
| integer | (optional) hierarchicalLevel |
The level of a heading or tree item.
|
| array of integer | (optional) wordStarts |
The start and end index of each word in an inline text box.
|
| array of integer | (optional) wordEnds | |
| array of [AutomationNode](/extensions/automation#type-AutomationNode) | (optional) controls |
The nodes, if any, which this node is specified to control via [`aria-controls`](http://www.w3.org/TR/wai-aria/states_and_properties#aria-controls).
|
| array of [AutomationNode](/extensions/automation#type-AutomationNode) | (optional) describedBy |
The nodes, if any, which form a description for this node.
|
| array of [AutomationNode](/extensions/automation#type-AutomationNode) | (optional) flowTo |
The nodes, if any, which may optionally be navigated to after this one. See [`aria-flowto`](http://www.w3.org/TR/wai-aria/states_and_properties#aria-flowto).
|
| array of [AutomationNode](/extensions/automation#type-AutomationNode) | (optional) labelledBy |
The nodes, if any, which form a label for this element. Generally, the text from these elements will also be exposed as the element's accessible name, via the [automation.AutomationNode.name](/extensions/automation#property-AutomationNode-name) attribute.
|
| [AutomationNode](/extensions/automation#type-AutomationNode) | (optional) activeDescendant |
The node referred to by `aria-activedescendant`, where applicable
|
| [AutomationNode](/extensions/automation#type-AutomationNode) | (optional) inPageLinkTarget |
The target of an in-page link.
|
| array of object | (optional) customActions |
An array of custom actions.
#### Properties of each object
## Methods
| integer | id | |
|---|---|---|
| string | description | |
### getTree
`whale.automation.getTree(integer tabId, function callback)`
Get the automation tree for the tab with the given tabId, or the current tab if no tabID is given, enabling automation if necessary. Returns a tree with a placeholder root node; listen for the "loadComplete" event to get a notification that the tree has fully loaded (the previous root node reference will stop working at or before this point).
| Parameters |
|---|
| integer | (optional) tabId | |
| function | (optional) callback |
Called when the `AutomationNode` for the page is available.
If you specify the _callback_ parameter, it should be a function that looks like this:
`function( [AutomationNode](/extensions/automation#type-AutomationNode) rootNode) {...};`
| [AutomationNode](/extensions/automation#type-AutomationNode) | rootNode | |
|---|---|---|
|
### getDesktop
`whale.automation.getDesktop(function callback)`
Get the automation tree for the whole desktop which consists of all on screen views. Note this API is currently only supported on Chrome OS.
| Parameters |
|---|
| function | callback |
Called when the `AutomationNode` for the page is available.
The _callback_ parameter should be a function that looks like this:
`function( [AutomationNode](/extensions/automation#type-AutomationNode) rootNode) {...};`
| [AutomationNode](/extensions/automation#type-AutomationNode) | rootNode | |
|---|---|---|
|
### getFocus
`whale.automation.getFocus(function callback)`
Get the automation node that currently has focus, globally. Will return null if none of the nodes in any loaded trees have focus.
| Parameters |
|---|
| function | callback |
Called with the `AutomationNode` that currently has focus.
The _callback_ parameter should be a function that looks like this:
`function( [AutomationNode](/extensions/automation#type-AutomationNode) focusedNode) {...};`
| [AutomationNode](/extensions/automation#type-AutomationNode) | focusedNode | |
|---|---|---|
|
### addTreeChangeObserver
`whale.automation.addTreeChangeObserver(enum of `"noTreeChanges"`, `"liveRegionTreeChanges"`, `"textMarkerChanges"`, or `"allTreeChanges"` filter, function observer)`
Add a tree change observer. Tree change observers are static/global, they listen to changes across all trees. Pass a filter to determine what specific tree changes to listen to, and note that listnening to all tree changes can be expensive.
| Parameters |
|---|
| enum of `"noTreeChanges"`, `"liveRegionTreeChanges"`, `"textMarkerChanges"`, or `"allTreeChanges"` | filter | |
| function | observer |
A listener for changes on the `AutomationNode` tree.
The _observer_ parameter should be a function that looks like this:
`function( [TreeChange](/extensions/automation#type-TreeChange) treeChange) {...};`
| [TreeChange](/extensions/automation#type-TreeChange) | treeChange | |
|---|---|---|
|
### removeTreeChangeObserver
`whale.automation.removeTreeChangeObserver(function observer)`
Remove a tree change observer.
| Parameters |
|---|
| function | observer |
A listener for changes on the `AutomationNode` tree.
The _observer_ parameter should be a function that looks like this:
`function( [TreeChange](/extensions/automation#type-TreeChange) treeChange) {...};`
| [TreeChange](/extensions/automation#type-TreeChange) | treeChange | |
|---|---|---|
|
### setDocumentSelection
`whale.automation.setDocumentSelection( [SetDocumentSelectionParams](/extensions/automation#type-SetDocumentSelectionParams) params)`
Sets the selection in a tree. This creates a selection in a single tree (anchorObject and focusObject must have the same root). Everything in the tree between the two node/offset pairs gets included in the selection. The anchor is where the user started the selection, while the focus is the point at which the selection gets extended e.g. when dragging with a mouse or using the keyboard. For nodes with the role staticText, the offset gives the character offset within the value where the selection starts or ends, respectively.
| Parameters |
|---|
| [SetDocumentSelectionParams](/extensions/automation#type-SetDocumentSelectionParams) | params | |