Python library β is an open source project for automating desktop GUI applications on WindowsOver the past two years, it has added major new features:
- Support for MS UI Automation technology. The interface remains the same, and now supports WinForms, WPF, Qt5, Windows Store (UWP) and so on - almost everything that is on Windows.
- Backend/plugin system (now there are two of them under the hood: default
"win32"and a new"uia"). Then we move smoothly towards cross-platform. - Win32 hooks for mouse and keyboard (hot keys in the spirit of pyHook).
We will also make a small overview of what is available in open source for desktop automation (without pretensions to a serious comparison).
This article is a partial transcript of a report from the SQA Days 20 conference in Minsk ( ΠΈ ), partially Russian version for pywinauto.
- Basic approaches
- Main desktop accessibility technologies
Let's start with a brief overview of open source in this area. For desktop GUI applications, things are somewhat more complicated than for the web, which has Selenium. Here are the main approaches:
coordinate method
Hardcode click points, hope for good hits.
[+] Cross-platform, easy to implement.
[+] It's easy to make a "record-replay" record of tests.
[-] The most unstable to changes in screen resolution, theme, fonts, window sizes, etc.
[-] A lot of effort is needed for support, it is often easier to regenerate tests from scratch or test manually.
[-] Only automates actions, there are other methods for verifying and extracting data.
Tools (cross-platform): , , and many others. As a rule, more complex tools include this functionality (not always cross-platform).
It is worth saying that the coordinate method can complement other approaches. For example, for custom graphics, you can click on relative coordinates (from the upper left corner of the window / element, and not the entire screen) - this is usually quite reliable, especially if you take into account the length / width of the entire element (then different screen resolutions will not hurt).
Another option is to allocate only one machine with stable settings for tests (not cross-platform, but in some cases it is good).
Recognition of reference images
[+] Cross-platform
[+-] Relatively reliable (better than the coordinate method), but still requires tricks.
[-+] Relatively slow, because requires CPU resources for recognition algorithms.
[-] Text recognition (OCR), as a rule, is out of the question => you can't get text data. As far as I know, the existing OCR solutions are not very reliable for this type of task, and are not widely used (welcome in the comments if this is not already the case).
Tools: , (Sikuli-compatible, pure Python), .
accessibility technology
[+] The most reliable method, because allows you to search for text, regardless of how it is rendered by the system or framework.
[+] Allows you to extract text data => easier to verify test results.
[+] As a rule, the fastest, because consumes almost no CPU resources.
[-] It is difficult to make a cross-platform tool: absolutely all open-source libraries support one or two accessibility technologies. Windows/Linux/MacOS is not fully supported by anyone except paid ones like TestComplete, UFT or Squish.
[-] Such technology is not always available in principle. For example, testing the boot screen inside VirtualBox is indispensable without image recognition. But in many classic cases, the accessibility approach is still applicable. About it further and will be discussed.
Tools: in C# in C# (Selenium compatible), in C# (Appium compatible), , (compatible with LDTP), , in Ruby (Linux Desktop Testing Project) and its Windows version .
LDTP is perhaps the only cross-platform open-source tool (more precisely, a family of libraries) based on accessibility technologies. However, he is not very popular. I have not used it myself, but according to reviews, the interface is not the most convenient. If there are positive reviews, please share in the comments.
Test backdoor (aka inside bike)
For cross-platform applications, the developers themselves often make an internal mechanism to ensure testability. For example, they create a service TCP server in the application, tests connect to it and send text commands: what to click on, where to get data from, etc. Reliable, but not universal.
Main desktop accessibility technologies
Good old Win32 API
Most Windows applications written before WPF was released and after Windows Store, are built in one way or another on the Win32 API. Namely, MFC, WTL, C++ Builder, Delphi, VB6βall these tools use the Win32 API. Even Windows Forms are largely Win32 API compatible.
Tools: (similar to VB) and Python wrapper , (own language, there is an IDispatch COM interface), (Python) (Ruby) (Ruby).
Microsoft UI Automation
The main advantage: MS UI Automation technology supports the vast majority of GUI applications on Windows With rare exceptions. The problem: it's not much easier to learn than the Win32 API. Otherwise, no one would be creating wrappers for it.
In fact, this is a set of custom COM interfaces (mainly UIAutomationCore.dll) and also has a .NET wrapper in the form namespace System.Windows.Automation. By the way, it has an introduced bug, due to which some UI elements can be skipped. Therefore, it is better to use UIAutomationCore.dll directly (if you heard about UiaComWrapper in C #, then this is it).
Varieties of COM interfaces:
(1) Base IUknown is "the root of all evil". The most low-level, never user-friendly.
(2) IDispatch and derivatives (for example, Excel.Application) that can be used in Python using the win32com.client package (included with pyWin32). The most convenient and beautiful option.
(3) Custom interfaces that a third-party Python package can work with .
Tools: in C# 0.6.0 + in C# (their source code for sish wrappers over UIAutomationCore.dll is not disclosed), on Ruby.
AT-SPI
Despite the fact that almost all axes of the family Linux Built on the X Window System (in Fedora 25, "X" was replaced by Wayland), "X" only allows you to operate top-level windows and the mouse/keyboard. For detailed analysis of buttons, list boxes, and so on, there's AT-SPI technology. The most popular window managers have a so-called AT-SPI registry daemon, which provides an automated GUI for applications (at least Qt and GTK are supported).
Tools: .
pyatspi2, in my opinion, contains too many dependencies like the same PyGObject. The technology itself is available as a regular dynamic library libatspi.so. She has For the pywinauto library, we plan to implement AT-SPI support this way: by loading libatspi.so and the ctypes module. The only minor issue is using the correct version, as they are slightly different for GTK+ and Qt applications. The likely release of pywinauto 0.7.0 will feature full support. Linux can be expected in the first half of 2018.
Apple Accessibility API
MacOS has its own automation language, AppleScript. To implement something like this in Python, of course, you need to use functions from ObjectiveC. Starting, it seems, even with MacOS 10.6, the pyobjc package is included in the pre-installed python. This will also make it easier to list dependencies for future support in pywinauto.
Tools: In addition to the Apple Script language, you should pay attention to , aka pyatom. It is interface compatible with LDTP, but is also a standalone library. It has written by my student. There is a known issue: flexible timings do not work (methods waitFor*). But, in general, a good thing.
How to get started with pywinauto
The first step is to equip yourself with a GUI object inspector (what is called the Spy tool). It will help to study the application from the inside: how the hierarchy of elements is arranged, what properties are available. The most famous object inspectors are:
- Spy++ - included with Visual Studio, including Express or Community Edition. Uses Win32 API. Also known as a clone AutoIt Window Info.
- Inspect.exe β is included in Windows SDK. If you have it installed, then it's on a 64-bit Windows You can find it in the folder
C:Program Files (x86)Windows Kits<winver>binx64. In the inspector itself, you need to select the mode UI Automation instead of MS AA (Active Accessibility, ancestor of UI Automation).
Having enlightened the application through and through, we select the backend that we will use. It is enough to specify the name of the backend when creating the Application object.
- backend="win32" - while used by default, works well with MFC, WTL, VB6 and other legacy applications.
- backend="uia" β new backend for MS UI Automation: works perfectly with WPF and WinForms; also good for Delphi and Windows Store apps; works with Qt5 and some Java applications. And generally, if Inspect.exe sees elements and their properties, then this backend is suitable. Basically, most browsers also support UI Automation (Mozilla by default, and Chrome requires a command-line switch upon launch).
--force-renderer-accessibilityto see items on pages in Inspect.exe). Of course, competition with Selenium in this area is hardly possible. Just another way to work with the browser (may come in handy for a cross-product scenario).
Entry points for automation
The app has been well researched. It's time to create an Application object and run it, or attach to an already running one. It's not just a clone of the standard class subprocess.Popen, which is an introductory object that limits all your actions to the boundaries of the process. This is very useful if multiple instances of an application are running and you don't want to touch the rest.
from pywinauto.application import Application
app = Application(backend="uia").start('notepad.exe')
# ΠΠΏΠΈΡΠ΅ΠΌ ΠΎΠΊΠ½ΠΎ, ΠΊΠΎΡΠΎΡΠΎΠ΅ Ρ
ΠΎΡΠΈΠΌ Π½Π°ΠΉΡΠΈ Π² ΠΏΡΠΎΡΠ΅ΡΡΠ΅ Notepad.exe
dlg_spec = app.UntitledNotepad
# ΠΆΠ΄Π΅ΠΌ ΠΏΠΎΠΊΠ° ΠΎΠΊΠ½ΠΎ ΡΠ΅Π°Π»ΡΠ½ΠΎ ΠΏΠΎΡΠ²ΠΈΡΡΡ
actionable_dlg = dlg_spec.wait('visible')If you want to manage several applications at once, the class will help you Desktop. For example, in the calculator on Win10, the hierarchy of elements is spread over several processes (not only calc.exe). So no object Desktop can not do.
from subprocess import Popen
from pywinauto import Desktop
Popen('calc.exe', shell=True)
dlg = Desktop(backend="uia").Calculator
dlg.wait('visible')Root object (Application or Desktop) is the only place where you need to specify the backend. Everything else transparently falls into the concept of "specification-> wrapper", which will be discussed later.
Window/element specifications
This is the core concept on which the pywinauto interface is built. You can describe the window/element roughly or in more detail, even if it does not yet exist or is already closed. window specification (object window specification) stores the criteria by which you need to search for a real window or element.
An example of a detailed window specification:
>>> dlg_spec = app.window(title='Untitled - Notepad')
>>> dlg_spec
<pywinauto.application.WindowSpecification object at 0x0568B790>
>>> dlg_spec.wrapper_object()
<pywinauto.controls.win32_controls.DialogWrapper object at 0x05639B70>The search for the window itself occurs by calling the method .wrapper_object(). It returns some "wrapper" for a real window/element or throws ElementNotFoundError (sometimes ElementAmbiguousError, if multiple elements are found, that is, you need to refine the search criteria). This "wrapper" already knows how to do some actions with the element or receive data from it.
Python can hide the call .wrapper_object(), so that the final code becomes shorter. We recommend using it for debugging purposes only. The next two lines do exactly the same thing:
dlg_spec.wrapper_object().minimize() # debugging
dlg_spec.minimize() # productionThere are a variety of search criteria for a window specification. Here are just a few examples:
# ΠΌΠΎΠ³ΡΡ ΠΈΠΌΠ΅ΡΡ Π½Π΅ΡΠΊΠΎΠ»ΡΠΊΠΎ ΡΡΠΎΠ²Π½Π΅ΠΉ
app.window(title_re='.* - Notepad$').window(class_name='Edit')
# ΠΌΠΎΠΆΠ½ΠΎ ΠΊΠΎΠΌΠ±ΠΈΠ½ΠΈΡΠΎΠ²Π°ΡΡ ΠΊΡΠΈΡΠ΅ΡΠΈΠΈ (ΠΊΠ°ΠΊ AND) ΠΈ Π½Π΅ ΠΎΠ³ΡΠ°Π½ΠΈΡΠΈΠ²Π°ΡΡΡΡ ΠΎΠ΄Π½ΠΈΠΌ ΠΏΡΠΎΡΠ΅ΡΡΠΎΠΌ ΠΏΡΠΈΠ»ΠΎΠΆΠ΅Π½ΠΈΡ
dlg = Desktop(backend="uia").Calculator
dlg.window(auto_id='num8Button', control_type='Button')The list of all possible criteria is in the function docks .
The magic of access by attribute and by key
Python makes it easy to create window specifications and recognize object attributes dynamically (internally overridden __getattribute__). Of course, the same restrictions are imposed on the attribute name as on the name of any variable (you cannot insert spaces, commas, and other special characters). Luckily, pywinauto uses a so-called "best match" search algorithm that is resistant to typos and slight variations.
app.UntitledNotepad
# ΡΠΎ ΠΆΠ΅ ΡΠ°ΠΌΠΎΠ΅, ΡΡΠΎ
app.window(best_match='UntitledNotepad')If you still need Unicode strings (for example, for the Russian language), spaces, etc., you can access by key (as if it were a regular dictionary):
app['Untitled - Notepad']
# ΡΠΎ ΠΆΠ΅ ΡΠ°ΠΌΠΎΠ΅, ΡΡΠΎ
app.window(best_match='Untitled - Notepad')Five Rules for Magical Names
How to find out the reference magical names? Those that are assigned to the element before the search. If you have specified a name that is sufficiently similar to the template, then the element will be found.
- By title (text, name):
app.Properties.OK.click() - By text and by element type:
app.Properties.OKButton.click() - By type and by number:
app.Properties.Button3.click()(namesButton0ΠΈButton1bound to the first element found,Button2- to the second, and then in order - it happened historically) - By static text (left or top) and by type:
app.OpenDialog.FileNameEdit.set_text("")(useful for elements with dynamic text) - By type and by text inside:
app.Properties.TabControlSharing.select("General")
Usually two or three rules are applied at the same time, rarely more. To check which specific names are available for each element, you can use the method print_control_identifiers(). It can print a tree of elements both to the screen and to a file. For each element, its reference magic names are printed. You can also copy-paste more detailed specifications of child elements from there. The result in the script will look like this:
app.Properties.child_window(data-gt-translate-attributes='["title"]' title="Contains:", auto_id="13087", control_type="Edit")The tree of elements itself is usually a rather large footcloth.
>>> app.Properties.print_control_identifiers()
Control Identifiers:
Dialog - 'Windows NT Properties' (L688, T518, R1065, B1006)
[u'Windows NT PropertiesDialog', u'Dialog', u'Windows NT Properties']
child_window(data-gt-translate-attributes='["title"]' title="Windows NT Properties", control_type="Window")
|
| Image - '' (L717, T589, R749, B622)
| [u'', u'0', u'Image1', u'Image0', 'Image', u'1']
| child_window(auto_id="13057", control_type="Image")
|
| Image - '' (L717, T630, R1035, B632)
| ['Image2', u'2']
| child_window(auto_id="13095", control_type="Image")
|
| Edit - 'Folder name:' (L790, T596, R1036, B619)
| [u'3', 'Edit', u'Edit1', u'Edit0']
| child_window(data-gt-translate-attributes='["title"]' title="Folder name:", auto_id="13156", control_type="Edit")
|
| Static - 'Type:' (L717, T643, R780, B658)
| [u'Type:Static', u'Static', u'Static1', u'Static0', u'Type:']
| child_window(data-gt-translate-attributes='["title"]' title="Type:", auto_id="13080", control_type="Text")
|
| Edit - 'Type:' (L790, T643, R1036, B666)
| [u'4', 'Edit2', u'Type:Edit']
| child_window(data-gt-translate-attributes='["title"]' title="Type:", auto_id="13059", control_type="Edit")
|
| Static - 'Location:' (L717, T669, R780, B684)
| [u'Location:Static', u'Location:', u'Static2']
| child_window(data-gt-translate-attributes='["title"]' title="Location:", auto_id="13089", control_type="Text")
|
| Edit - 'Location:' (L790, T669, R1036, B692)
| ['Edit3', u'Location:Edit', u'5']
| child_window(data-gt-translate-attributes='["title"]' title="Location:", auto_id="13065", control_type="Edit")
|
| Static - 'Size:' (L717, T695, R780, B710)
| [u'Size:Static', u'Size:', u'Static3']
| child_window(data-gt-translate-attributes='["title"]' title="Size:", auto_id="13081", control_type="Text")
|
| Edit - 'Size:' (L790, T695, R1036, B718)
| ['Edit4', u'6', u'Size:Edit']
| child_window(data-gt-translate-attributes='["title"]' title="Size:", auto_id="13064", control_type="Edit")
|
| Static - 'Size on disk:' (L717, T721, R780, B736)
| [u'Size on disk:', u'Size on disk:Static', u'Static4']
| child_window(data-gt-translate-attributes='["title"]' title="Size on disk:", auto_id="13107", control_type="Text")
|
| Edit - 'Size on disk:' (L790, T721, R1036, B744)
| ['Edit5', u'7', u'Size on disk:Edit']
| child_window(data-gt-translate-attributes='["title"]' title="Size on disk:", auto_id="13106", control_type="Edit")
|
| Static - 'Contains:' (L717, T747, R780, B762)
| [u'Contains:1', u'Contains:0', u'Contains:Static', u'Static5', u'Contains:']
| child_window(data-gt-translate-attributes='["title"]' title="Contains:", auto_id="13088", control_type="Text")
|
| Edit - 'Contains:' (L790, T747, R1036, B770)
| [u'8', 'Edit6', u'Contains:Edit']
| child_window(data-gt-translate-attributes='["title"]' title="Contains:", auto_id="13087", control_type="Edit")
|
| Image - 'Contains:' (L717, T773, R1035, B775)
| [u'Contains:Image', 'Image3', u'Contains:2']
| child_window(data-gt-translate-attributes='["title"]' title="Contains:", auto_id="13096", control_type="Image")
|
| Static - 'Created:' (L717, T786, R780, B801)
| [u'Created:', u'Created:Static', u'Static6', u'Created:1', u'Created:0']
| child_window(data-gt-translate-attributes='["title"]' title="Created:", auto_id="13092", control_type="Text")
|
| Edit - 'Created:' (L790, T786, R1036, B809)
| [u'Created:Edit', 'Edit7', u'9']
| child_window(data-gt-translate-attributes='["title"]' title="Created:", auto_id="13072", control_type="Edit")
|
| Image - 'Created:' (L717, T812, R1035, B814)
| [u'Created:Image', 'Image4', u'Created:2']
| child_window(data-gt-translate-attributes='["title"]' title="Created:", auto_id="13097", control_type="Image")
|
| Static - 'Attributes:' (L717, T825, R780, B840)
| [u'Attributes:Static', u'Static7', u'Attributes:']
| child_window(data-gt-translate-attributes='["title"]' title="Attributes:", auto_id="13091", control_type="Text")
|
| CheckBox - 'Read-only (Only applies to files in folder)' (L790, T825, R1035, B841)
| [u'CheckBox0', u'CheckBox1', 'CheckBox', u'Read-only (Only applies to files in folder)CheckBox', u'Read-only (Only applies to files in folder)']
| child_window(data-gt-translate-attributes='["title"]' title="Read-only (Only applies to files in folder)", auto_id="13075", control_type="CheckBox")
|
| CheckBox - 'Hidden' (L790, T848, R865, B864)
| ['CheckBox2', u'HiddenCheckBox', u'Hidden']
| child_window(data-gt-translate-attributes='["title"]' title="Hidden", auto_id="13076", control_type="CheckBox")
|
| Button - 'Advanced...' (L930, T845, R1035, B868)
| [u'Advanced...', u'Advanced...Button', 'Button', u'Button1', u'Button0']
| child_window(data-gt-translate-attributes='["title"]' title="Advanced...", auto_id="13154", control_type="Button")
|
| Button - 'OK' (L814, T968, R889, B991)
| ['Button2', u'OK', u'OKButton']
| child_window(data-gt-translate-attributes='["title"]' title="OK", auto_id="1", control_type="Button")
|
| Button - 'Cancel' (L895, T968, R970, B991)
| ['Button3', u'CancelButton', u'Cancel']
| child_window(data-gt-translate-attributes='["title"]' title="Cancel", auto_id="2", control_type="Button")
|
| Button - 'Apply' (L976, T968, R1051, B991)
| ['Button4', u'ApplyButton', u'Apply']
| child_window(data-gt-translate-attributes='["title"]' title="Apply", auto_id="12321", control_type="Button")
|
| TabControl - '' (L702, T556, R1051, B962)
| [u'10', u'TabControlSharing', u'TabControlPrevious Versions', u'TabControlSecurity', u'TabControl', u'TabControlCustomize']
| child_window(auto_id="12320", control_type="Tab")
| |
| | TabItem - 'General' (L704, T558, R753, B576)
| | [u'GeneralTabItem', 'TabItem', u'General', u'TabItem0', u'TabItem1']
| | child_window(data-gt-translate-attributes='["title"]' title="General", control_type="TabItem")
| |
| | TabItem - 'Sharing' (L753, T558, R801, B576)
| | [u'Sharing', u'SharingTabItem', 'TabItem2']
| | child_window(data-gt-translate-attributes='["title"]' title="Sharing", control_type="TabItem")
| |
| | TabItem - 'Security' (L801, T558, R851, B576)
| | [u'Security', 'TabItem3', u'SecurityTabItem']
| | child_window(data-gt-translate-attributes='["title"]' title="Security", control_type="TabItem")
| |
| | TabItem - 'Previous Versions' (L851, T558, R947, B576)
| | [u'Previous VersionsTabItem', u'Previous Versions', 'TabItem4']
| | child_window(data-gt-translate-attributes='["title"]' title="Previous Versions", control_type="TabItem")
| |
| | TabItem - 'Customize' (L947, T558, R1007, B576)
| | [u'CustomizeTabItem', 'TabItem5', u'Customize']
| | child_window(data-gt-translate-attributes='["title"]' title="Customize", control_type="TabItem")
|
| TitleBar - 'None' (L712, T521, R1057, B549)
| ['TitleBar', u'11']
| |
| | Menu - 'System' (L696, T526, R718, B548)
| | [u'System0', u'System', u'System1', u'Menu', u'SystemMenu']
| | child_window(data-gt-translate-attributes='["title"]' title="System", auto_id="MenuBar", control_type="MenuBar")
| | |
| | | MenuItem - 'System' (L696, T526, R718, B548)
| | | [u'System2', u'MenuItem', u'SystemMenuItem']
| | | child_window(data-gt-translate-attributes='["title"]' title="System", control_type="MenuItem")
| |
| | Button - 'Close' (L1024, T519, R1058, B549)
| | [u'CloseButton', u'Close', 'Button5']
| | child_window(data-gt-translate-attributes='["title"]' title="Close", control_type="Button")In some cases, printing the entire tree can slow down (for example, in iTunes there are as many as three thousand items on one tab!), But you can use the parameter depth (depth): depth=1 - the element itself depth=2 β only immediate children, and so on. It can also be specified in the specifications when creating child_window.
Examples
We are constantly replenishing . Of the fresh ones, it is worth noting the automation of the WireShark network analyzer (this is a good example of a Qt5 application; although this task can be solved without a GUI, because there are scapy.Sniffer from python package ). There is also an example of MS Paint automation with its Ribbon toolbar.
Another great example written by my student: (it will migrate to the main repository a little later).
And, of course, an example of subscribing to keyboard (hot keys) and mouse events:
.
Acknowledgements
Special thanks to those who constantly help to develop the project. For me and it's an ongoing hobby. Two of my students from UNN recently completed their bachelor's degrees in this topic. made a great contribution to MS UI Automation support and recently started making an automatic code generator based on the "record-play" principle based on text properties (this is the most difficult feature), so far only for the "uia" backend. is developing a new backend for Linux based on AT-SPI (modules mouse ΠΈ keyboard on the basis of - already in 0.6.x releases).
Since I have been teaching a special course on automation in Python for quite some time, some of the master's students do their homework, implementing small features or examples of automation. Some key things at the research stage were also once unearthed by students. Although sometimes you have to strictly monitor the quality of the code. This is greatly helped by static analyzers (QuantifiedCode, Codacy and Landscape) and automated tests in the cloud (AppVeyor service) with code coverage of around 95%.
Also thanks to everyone who leaves feedback, starts bugs and sends pull requests!
Additional resources
We follow questions on (recently appeared ) and . There is .
We update every month . In terms of the number of stars on the github, only Autohotkey (they have a very large community and a long history) and PyAutoGUI (largely due to the popularity of books by its author Al Sweigart: "Automate the Boring Stuff with Python" and others) are growing faster.
Source: habr.com
