Python library is an open-source project for automating desktop GUI applications on Windows. Over the past two years, it has gained several significant features:
- Support for MS UI Automation technology. The interface remains the same, and now supports: WinForms, WPF, Qt5, Windows Store (UWP), and more — almost everything available on Windows.
- A system of backends/plugins (currently, there are two under the hood: the default
"win32"and the new"uia"). We are smoothly moving towards cross-platform solutions. - Win32 hooks for mouse and keyboard (hot keys in the style of pyHook).
We will also provide a brief overview of what's available in open source for desktop automation (without claiming a thorough comparison).
This article is partly a transcription of a presentation from the SQA Days 20 conference in Minsk ( and ), partly the Russian version of for pywinauto.
- Main Approaches
- Key desktop accessibility technologies
Let's start with a brief overview of open source in this area. For desktop GUI applications, things are a bit more complex than for the web, which has Selenium. Here are the main approaches:
Coordinate Method
Hardcode click points, hoping for accurate hits.
[+] Cross-platform, easy to implement.
[+] Easy to create 'record-replay' test recordings.
[-] The most unstable with changes in screen resolution, themes, fonts, window sizes, etc.
[-] Requires huge efforts for support; often easier to regenerate tests from scratch or test manually.
[-] Automates only actions; other methods are needed for verification and data extraction.
Tools (cross-platform): , , and many others. Generally, more complex tools include this functionality (not always cross-platform).
It is worth mentioning that the coordinate method can complement other approaches. For instance, for custom graphics, you can click on relative coordinates (from the top left corner of the window/element, not the entire screen) — this is usually quite reliable, especially if you take into account the length/width of the entire element (then even different screen resolutions won't be an issue).
Another option: select only one machine with stable settings for tests (not cross-platform, but suitable in some cases).
Recognition of Reference Images
[+] Cross-platform
[+-] Relatively reliable (better than the coordinate method), but still requires some tricks.
[-+] Relatively slow, as it requires CPU resources for recognition algorithms.
[-] On text recognition (OCR), usually, there is no mention => you can't extract textual data. As far as I know, existing OCR solutions are not very reliable for this type of tasks and are not widely used (comments welcome if this is no longer the case).
Tools: , (Sikuli-compatible, pure Python), .
Accessibility Technologies
[+] The most reliable method, as it allows searching by text, regardless of how it is rendered by the system or framework.
[+] Allows extracting textual data => easier to verify test results.
[+] Usually the fastest, as it consumes almost no CPU resources.
[-] Difficult to create a cross-platform tool: nearly all open-source libraries support one or two accessibility technologies. No one supports the entire Windows/Linux/MacOS, except for paid options like TestComplete, UFT, or Squish.
[-] Such technology is not always available in principle. For example, testing the boot screen inside VirtualBox — here, image recognition is a must. But in many classic cases, the accessibility approach is still applicable. This will be discussed further.
Tools: in C#, in C# (Selenium compatible), in C# (Appium compatible), , (compatible with LDTP), , in Ruby, (Linux Desktop Testing Project) and its Windows version .
LDTP is perhaps the only cross-platform open-source tool (more accurately, a family of libraries) based on accessibility technologies. However, it is not very popular. I haven't used it myself, but feedback suggests that the interface is not the most user-friendly. If there are positive reviews, please share in the comments.
Test backdoor (a.k.a. internal bicycle)
For cross-platform applications, developers often create an internal mechanism for ensuring testability. For example, they build a service TCP server within the application, to which tests connect and send text commands: what to click, where to get data from, etc. Reliable, but not universal.
Key desktop accessibility technologies
The good old Win32 API
Most Windows applications written before the release of WPF and later Windows Store are built, one way or another, on the Win32 API. Specifically, MFC, WTL, C++ Builder, Delphi, VB6 — all these tools use the Win32 API. Even Windows Forms — largely compatible with the Win32 API.
Tools: (similar to VB) and Python wrapper , (custom language, has IDispatch COM interface), (Python), (Ruby), (Ruby).
Microsoft UI Automation
The main advantage: the MS UI Automation technology supports the overwhelming majority of GUI applications on Windows with few exceptions. The problem: it is not significantly easier to learn than the Win32 API. Otherwise, no one would create wrappers for it.
In fact, this is a set of custom COM interfaces (mainly, UIAutomationCore.dll), and it also has a .NET wrapper in the form of namespace System.Windows.Automation. By the way, it has an introduced bug that may cause some UI elements to be missed. Therefore, it's better to use UIAutomationCore.dll directly (if you've heard of UiaComWrapper on C#, that's it).
Varieties of COM interfaces:
(1) Basic IUknown — "the root of all evil". The lowest-level, not user-friendly at all.
(2) IDispatch and its derivatives (for example, Excel.Application), which can be used in Python with the win32com.client package (part of pyWin32). The most convenient and elegant option.
(3) Custom interfaces that can be worked with via the third-party Python package .
Tools: in C#, 0.6.0+, in C#, (their source code for C wrappers over UIAutomationCore.dll is not disclosed), for Ruby.
AT-SPI
Despite the fact that almost all OSs in the Linux family are built on the X Window System (in Fedora 25, the 'X' was replaced with Wayland), 'X' allows operating only with top-level windows and mouse/keyboard. For detailed analysis of buttons, list boxes, etc. — there is the AT-SPI technology. The most popular window managers have what is called an AT-SPI registry daemon that provides an automatable GUI for applications (at least Qt and GTK are supported).
Tools: .
In my opinion, pyatspi2 contains too many dependencies like PyGObject. The technology itself is available as a regular dynamic library libatspi.so. It has access to For the pywinauto library, we plan to implement support for AT-SPI specifically by loading libatspi.so and using the ctypes module. There is a small issue with using the correct version, as the versions for GTK+ and Qt applications are slightly different. A potential release of pywinauto 0.7.0 with full Linux support can be expected in the first half of 2018.
Apple Accessibility API
On MacOS, there is a native automation language called AppleScript. To implement something similar in Python, you will need to utilize functions from ObjectiveC. Starting from MacOS 10.6, the pre-installed Python includes the pyobjc package. This will also simplify the list of dependencies for future support in pywinauto.
Tools: In addition to the AppleScript language, it's worth noting , also known as pyatom. It is interface-compatible with LDTP but is also a standalone library. There is a , written by my student. There is a known issue: flexible timings (methods waitFor*) do not work. But overall, it’s a decent tool.
How to get started with pywinauto
First, one should equip themselves with a GUI object inspector (commonly referred to as a Spy tool). It will help to examine the application from the inside: how the element hierarchy is structured, which properties are available. The most well-known object inspectors include:
- Spy++ — included in the Visual Studio installation, including Express or Community Edition. It uses the Win32 API. Its clone is also well-known, AutoIt Window Info.
- Inspect.exe — part of the Windows SDK. If it is installed, you can find it in the folder
C:Program Files (x86)Windows Kits<winver>binx64. In the inspector, you need to select the mode UI Automation instead of MS AA (Active Accessibility, the predecessor to UI Automation).
Having thoroughly examined the application, we choose the backend we will use. It is sufficient to specify the backend name when creating the Application object.
- backend="win32" — currently used by default, works well with MFC, WTL, VB6, and other legacy applications.
- backend="uia" — a new backend for MS UI Automation: works perfectly with WPF and WinForms; also suitable for Delphi and Windows Store applications; works with Qt5 and some Java applications. In general, if Inspect.exe can see the elements and their properties, then this backend is appropriate. Most browsers also support UI Automation (Mozilla by default, while Chrome needs to be fed the command line flag
--force-renderer-accessibility, to see elements on pages in Inspect.exe). Of course, competition with Selenium in this area is unlikely. Just another way to work with the browser (which might be useful for cross-product scenarios).
Entry points for automation
The application is well-studied. It's time to create an Application object and launch it or join an already running one. This is not just a clone of the standard class subprocess.Popen, but rather an introductory object that confines all your actions within the boundaries of the process. This is very useful if multiple instances of the application are running, and you don't want to affect the others.
from pywinauto.application import Application
app = Application(backend="uia").start('notepad.exe')
# Describe the window we want to find in the Notepad.exe process
dlg_spec = app.UntitledNotepad
# wait until the window actually appears
actionable_dlg = dlg_spec.wait('visible')If you want to control several applications at once, the class Desktopwill help you. For instance, in the calculator on Win10, the hierarchy of elements is spread across several processes (not just calc.exe). So you need the object Desktop to navigate.
from subprocess import Popen
from pywinauto import Desktop
Popen('calc.exe', shell=True)
dlg = Desktop(backend="uia").Calculator
dlg.wait('visible')The root object (Application or Desktop) is the only place where you need to specify the backend. Everything else transparently fits into the specification->wrapper concept, which we will discuss next.
Window/element specifications
This is the main concept on which the pywinauto interface is built. You can describe a window/element roughly or in more detail, even if it does not yet exist or is already closed. The window specification (object WindowSpecification) holds the criteria by which to search for the real window or element.
Example of a detailed window specification:
>>> dlg_spec = app.window(title='Untitled - Notepad')
>>> dlg_spec
>>> dlg_spec.wrapper_object()The search for the window occurs by calling the method .wrapper_object(). It returns a certain 'wrapper' for the real window/element or throws ElementNotFoundError (sometimes ElementAmbiguousError, if multiple elements are found, meaning the search criteria need to be specified). This 'wrapper' can already perform some actions with the element or retrieve data from it.
Python can hide the call .wrapper_object(), so the final code becomes shorter. We recommend using it only for debugging. The next two lines do exactly the same thing:
dlg_spec.wrapper_object().minimize() # debugging
dlg_spec.minimize() # productionThere are many search criteria for window specifications. Here are just a few examples:
# могут иметь несколько уровней
app.window(title_re='.* - Notepad$').window(class_name='Edit')
# можно комбинировать критерии (как AND) и не ограничиваться одним процессом приложения
dlg = Desktop(backend="uia").Calculator
dlg.window(auto_id='num8Button', control_type='Button')A list of all possible criteria can be found in the function documentation. .
The magic of access by attribute and key
Python simplifies the creation of window specifications and dynamically recognizes object attributes (the method is overridden inside __getattribute__). Of course, the name of the attribute is subject to the same restrictions as the name of any variable (you cannot insert spaces, commas, or other special characters). Fortunately, pywinauto uses a so-called 'best match' search algorithm, which is resilient to typos and slight variations.
app.UntitledNotepad
# same as
app.window(best_match='UntitledNotepad')If Unicode strings (for example, for the Russian language), spaces, etc. are still required, you can access by key (as if it were a regular dictionary):
app['Untitled - Notepad']
# same as
app.window(best_match='Untitled - Notepad')Five rules for magical names
How to find out the canonical magic names? Those assigned to the element before the search. If you specify a name that is similar to the canonical one, the element will be found.
- By title (text, name):
app.Properties.OK.click() - By text and type of element:
app.Properties.OKButton.click() - By type and number:
app.Properties.Button3.click()(namesButton0andButton1are tied to the first found element,Button2— to the second, and so on in order — that's just how it has historically been) - By static text (to the left or above) and by type:
app.OpenDialog.FileNameEdit.set_text("")(useful for elements with dynamic text) - By type and by text inside:
app.Properties.TabControlSharing.select("General")
Usually two or three rules are applied simultaneously, rarely more. To check which specific names are available for each element, you can use the method print_control_identifiers(). It can print the tree of elements both to the screen and to a file. For each element, its canonical magic names are printed. You can also copy and paste more detailed specifications of child elements from there. The result in the script will look like this:
app.Properties.child_window(title="Contains:", auto_id="13087", control_type="Edit")The element tree itself is usually quite a lengthy document.
>>> app.Properties.print_control_identifiers()
Control Identifiers:
Dialog - 'Windows NT Properties' (L688, T518, R1065, B1006)
[u'Windows NT PropertiesDialog', u'Dialog', u'Windows NT Properties']
child_window(title="Windows NT Properties", control_type="Window")
|
| Image - '' (L717, T589, R749, B622)
| [u'', u'0', u'Image1', u'Image0', 'Image', u'1']
| child_window(auto_id="13057", control_type="Image")
|
| Image - '' (L717, T630, R1035, B632)
| ['Image2', u'2']
| child_window(auto_id="13095", control_type="Image")
|
| Edit - 'Folder name:' (L790, T596, R1036, B619)
| [u'3', 'Edit', u'Edit1', u'Edit0']
| child_window(title="Folder name:", auto_id="13156", control_type="Edit")
|
| Static - 'Type:' (L717, T643, R780, B658)
| [u'Type:Static', u'Static', u'Static1', u'Static0', u'Type:']
| child_window(title="Type:", auto_id="13080", control_type="Text")
|
| Edit - 'Type:' (L790, T643, R1036, B666)
| [u'4', 'Edit2', u'Type:Edit']
| child_window(title="Type:", auto_id="13059", control_type="Edit")
|
| Static - 'Location:' (L717, T669, R780, B684)
| [u'Location:Static', u'Location:', u'Static2']
| child_window(title="Location:", auto_id="13089", control_type="Text")
|
| Edit - 'Location:' (L790, T669, R1036, B692)
| ['Edit3', u'Location:Edit', u'5']
| child_window(title="Location:", auto_id="13065", control_type="Edit")
|
| Static - 'Size:' (L717, T695, R780, B710)
| [u'Size:Static', u'Size:', u'Static3']
| child_window(title="Size:", auto_id="13081", control_type="Text")
|
| Edit - 'Size:' (L790, T695, R1036, B718)
| ['Edit4', u'6', u'Size:Edit']
| child_window(title="Size:", auto_id="13064", control_type="Edit")
|
| Static - 'Size on disk:' (L717, T721, R780, B736)
| [u'Size on disk:', u'Size on disk:Static', u'Static4']
| child_window(title="Size on disk:", auto_id="13107", control_type="Text")
|
| Edit - 'Size on disk:' (L790, T721, R1036, B744)
| ['Edit5', u'7', u'Size on disk:Edit']
| child_window(title="Size on disk:", auto_id="13106", control_type="Edit")
|
| Static - 'Contains:' (L717, T747, R780, B762)
| [u'Contains:1', u'Contains:0', u'Contains:Static', u'Static5', u'Contains:']
| child_window(title="Contains:", auto_id="13088", control_type="Text")
|
| Edit - 'Contains:' (L790, T747, R1036, B770)
| [u'8', 'Edit6', u'Contains:Edit']
| child_window(title="Contains:", auto_id="13087", control_type="Edit")
|
| Image - 'Contains:' (L717, T773, R1035, B775)
| [u'Contains:Image', 'Image3', u'Contains:2']
| child_window(title="Contains:", auto_id="13096", control_type="Image")
|
| Static - 'Created:' (L717, T786, R780, B801)
| [u'Created:', u'Created:Static', u'Static6', u'Created:1', u'Created:0']
| child_window(title="Created:", auto_id="13092", control_type="Text")
|
| Edit - 'Created:' (L790, T786, R1036, B809)
| [u'Created:Edit', 'Edit7', u'9']
| child_window(title="Created:", auto_id="13072", control_type="Edit")
|
| Image - 'Created:' (L717, T812, R1035, B814)
| [u'Created:Image', 'Image4', u'Created:2']
| child_window(title="Created:", auto_id="13097", control_type="Image")
|
| Static - 'Attributes:' (L717, T825, R780, B840)
| [u'Attributes:Static', u'Static7', u'Attributes:']
| child_window(title="Attributes:", auto_id="13091", control_type="Text")
|
| CheckBox - 'Read-only (Only applies to files in folder)' (L790, T825, R1035, B841)
| [u'CheckBox0', u'CheckBox1', 'CheckBox', u'Read-only (Only applies to files in folder)CheckBox', u'Read-only (Only applies to files in folder)']
| child_window(title="Read-only (Only applies to files in folder)", auto_id="13075", control_type="CheckBox")
|
| CheckBox - 'Hidden' (L790, T848, R865, B864)
| ['CheckBox2', u'HiddenCheckBox', u'Hidden']
| child_window(title="Hidden", auto_id="13076", control_type="CheckBox")
|
| Button - 'Advanced...' (L930, T845, R1035, B868)
| [u'Advanced...', u'Advanced...Button', 'Button', u'Button1', u'Button0']
| child_window(title="Advanced...", auto_id="13154", control_type="Button")
|
| Button - 'OK' (L814, T968, R889, B991)
| ['Button2', u'OK', u'OKButton']
| child_window(title="OK", auto_id="1", control_type="Button")
|
| Button - 'Cancel' (L895, T968, R970, B991)
| ['Button3', u'CancelButton', u'Cancel']
| child_window(title="Cancel", auto_id="2", control_type="Button")
|
| Button - 'Apply' (L976, T968, R1051, B991)
| ['Button4', u'ApplyButton', u'Apply']
| child_window(title="Apply", auto_id="12321", control_type="Button")
|
| TabControl - '' (L702, T556, R1051, B962)
| [u'10', u'TabControlSharing', u'TabControlPrevious Versions', u'TabControlSecurity', u'TabControl', u'TabControlCustomize']
| child_window(auto_id="12320", control_type="Tab")
| |
| | TabItem - 'General' (L704, T558, R753, B576)
| | [u'GeneralTabItem', 'TabItem', u'General', u'TabItem0', u'TabItem1']
| | child_window(title="General", control_type="TabItem")
| |
| | TabItem - 'Sharing' (L753, T558, R801, B576)
| | [u'Sharing', u'SharingTabItem', 'TabItem2']
| | child_window(title="Sharing", control_type="TabItem")
| |
| | TabItem - 'Security' (L801, T558, R851, B576)
| | [u'Security', 'TabItem3', u'SecurityTabItem']
| | child_window(title="Security", control_type="TabItem")
| |
| | TabItem - 'Previous Versions' (L851, T558, R947, B576)
| | [u'Previous VersionsTabItem', u'Previous Versions', 'TabItem4']
| | child_window(title="Previous Versions", control_type="TabItem")
| |
| | TabItem - 'Customize' (L947, T558, R1007, B576)
| | [u'CustomizeTabItem', 'TabItem5', u'Customize']
| | child_window(title="Customize", control_type="TabItem")
|
| TitleBar - 'None' (L712, T521, R1057, B549)
| ['TitleBar', u'11']
| |
| | Menu - 'System' (L696, T526, R718, B548)
| | [u'System0', u'System', u'System1', u'Menu', u'SystemMenu']
| | child_window(title="System", auto_id="MenuBar", control_type="MenuBar")
| | |
| | | MenuItem - 'System' (L696, T526, R718, B548)
| | | [u'System2', u'MenuItem', u'SystemMenuItem']
| | | child_window(title="System", control_type="MenuItem")
| |
| | Button - 'Close' (L1024, T519, R1058, B549)
| | [u'CloseButton', u'Close', 'Button5']
| | child_window(title="Close", control_type="Button")In some cases, printing the entire tree can be slow (for example, in iTunes, there can be three thousand elements in one tab!), but you can use the parameter depth (depth): depth=1 — the element itself, depth=2 — only immediate children, and so on. It can also be specified in the specifications when creating child_window.
Examples
We are constantly updating . Among the latest, it's worth noting the automation of the network analyzer WireShark (it's a good example of a Qt5 application; although this task can also be solved without a GUI since there is scapy.Sniffer from the Python package ). There is also an example of automating MS Paint with its Ribbon toolbar.
Another great example, written by my student: (it will migrate to the main repository a bit later).
And, of course, an example of subscribing to keyboard (hot keys) and mouse events:
.
Acknowledgments
A separate thank you to those who continuously help develop the project. For me and this is a constant hobby. Two of my students from Nizhny Novgorod State University recently graduated with bachelor's degrees on this topic. made a significant contribution to supporting MS UI Automation and recently started creating an automatic code generator based on the 'record-playback' principle using text properties (this is the most complex feature), currently only for the 'uia' backend. is developing a new backend for Linux based on AT-SPI (modules mouse and keyboard based on — already in releases 0.6.x).
Since I have been teaching a special course on automation in Python for quite some time, some master's students complete homework by implementing small features or automation examples. Some key things during the research stage were also discovered by the students. Although sometimes I have to closely monitor the code quality. Static analyzers (QuantifiedCode, Codacy, and Landscape) and automated tests in the cloud (AppVeyor service) with around 95% code coverage help greatly with this.
Also, thank you to everyone who leaves feedback, reports bugs, and sends pull requests!
Additional resources
We keep track of questions via (a tag recently appeared ) and . There is .
We update In terms of stars on GitHub, only Autohotkey (which has a very large community and a long history) and PyAutoGUI (largely thanks to the popularity of its author Al Sweigart's books, such as "Automate the Boring Stuff with Python" and others) are growing faster.
Source: habr.com
