Python library β ΡΡΠΎ open source ΠΏΡΠΎΠ΅ΠΊΡ ΠΏΠΎ Π°Π²ΡΠΎΠΌΠ°ΡΠΈΠ·Π°ΡΠΈΠΈ Π΄Π΅ΡΠΊΡΠΎΠΏΠ½ΡΡ GUI ΠΏΡΠΈΠ»ΠΎΠΆΠ΅Π½ΠΈΠΉ Π½Π° Windows. ΠΠ° ΠΏΠΎΡΠ»Π΅Π΄Π½ΠΈΠ΅ Π΄Π²Π° Π³ΠΎΠ΄Π° Π² Π½Π΅ΠΉ ΠΏΠΎΡΠ²ΠΈΠ»ΠΈΡΡ Π½ΠΎΠ²ΡΠ΅ ΠΊΡΡΠΏΠ½ΡΠ΅ ΡΠΈΡΠΈ:
- ΠΠΎΠ΄Π΄Π΅ΡΠΆΠΊΠ° ΡΠ΅Ρ Π½ΠΎΠ»ΠΎΠ³ΠΈΠΈ MS UI Automation. ΠΠ½ΡΠ΅ΡΡΠ΅ΠΉΡ ΠΏΡΠ΅ΠΆΠ½ΠΈΠΉ, ΠΈ ΡΠ΅ΠΏΠ΅ΡΡ ΠΏΠΎΠ΄Π΄Π΅ΡΠΆΠΈΠ²Π°ΡΡΡΡ: WinForms, WPF, Qt5, Windows Store (UWP) ΠΈ ΡΠ°ΠΊ Π΄Π°Π»Π΅Π΅ β ΠΏΠΎΡΡΠΈ Π²ΡΠ΅, ΡΡΠΎ Π΅ΡΡΡ Π½Π° Windows.
- Backend/plugin system (now there are two of them under the hood: default
"win32"and a new"uia"). Then we move smoothly towards cross-platform. - Win32 hooks for mouse and keyboard (hot keys in the spirit of pyHook).
We will also make a small overview of what is available in open source for desktop automation (without pretensions to a serious comparison).
This article is a partial transcript of a report from the SQA Days 20 conference in Minsk ( ΠΈ ), partially Russian version for pywinauto.
- Basic approaches
- Main desktop accessibility technologies
Let's start with a brief overview of open source in this area. For desktop GUI applications, things are somewhat more complicated than for the web, which has Selenium. Here are the main approaches:
coordinate method
Hardcode click points, hope for good hits.
[+] Cross-platform, easy to implement.
[+] It's easy to make a "record-replay" record of tests.
[-] The most unstable to changes in screen resolution, theme, fonts, window sizes, etc.
[-] A lot of effort is needed for support, it is often easier to regenerate tests from scratch or test manually.
[-] Only automates actions, there are other methods for verifying and extracting data.
Tools (cross-platform): , , and many others. As a rule, more complex tools include this functionality (not always cross-platform).
It is worth saying that the coordinate method can complement other approaches. For example, for custom graphics, you can click on relative coordinates (from the upper left corner of the window / element, and not the entire screen) - this is usually quite reliable, especially if you take into account the length / width of the entire element (then different screen resolutions will not hurt).
Another option is to allocate only one machine with stable settings for tests (not cross-platform, but in some cases it is good).
Recognition of reference images
[+] Cross-platform[+-] Relatively reliable (better than the coordinate method), but still requires tricks.
[-+] Relatively slow, because requires CPU resources for recognition algorithms.
[-] Text recognition (OCR), as a rule, is out of the question => you can't get text data. As far as I know, the existing OCR solutions are not very reliable for this type of task, and are not widely used (welcome in the comments if this is not already the case).
Tools: , (Sikuli-compatible, pure Python), .
accessibility technology
[+] The most reliable method, because allows you to search for text, regardless of how it is rendered by the system or framework.[+] Allows you to extract text data => easier to verify test results.
[+] As a rule, the fastest, because consumes almost no CPU resources.
[-] Π’ΡΠΆΠ΅Π»ΠΎ ΡΠ΄Π΅Π»Π°ΡΡ ΠΊΡΠΎΡΡ-ΠΏΠ»Π°ΡΡΠΎΡΠΌΠ΅Π½Π½ΡΠΉ ΠΈΠ½ΡΡΡΡΠΌΠ΅Π½Ρ: Π°Π±ΡΠΎΠ»ΡΡΠ½ΠΎ Π²ΡΠ΅ open-source Π±ΠΈΠ±Π»ΠΈΠΎΡΠ΅ΠΊΠΈ ΠΏΠΎΠ΄Π΄Π΅ΡΠΆΠΈΠ²Π°ΡΡ ΠΎΠ΄Π½Ρ-Π΄Π²Π΅ accessibility ΡΠ΅Ρ Π½ΠΎΠ»ΠΎΠ³ΠΈΠΈ. Windows/Linux/MacOS ΡΠ΅Π»ΠΈΠΊΠΎΠΌ Π½Π΅ ΠΏΠΎΠ΄Π΄Π΅ΡΠΆΠΈΠ²Π°Π΅Ρ Π½ΠΈΠΊΡΠΎ, ΠΊΡΠΎΠΌΠ΅ ΠΏΠ»Π°ΡΠ½ΡΡ ΡΠΈΠΏΠ° TestComplete, UFT ΠΈΠ»ΠΈ Squish.
[-] Such technology is not always available in principle. For example, testing the boot screen inside VirtualBox is indispensable without image recognition. But in many classic cases, the accessibility approach is still applicable. About it further and will be discussed.
Tools: in C# in C# (Selenium compatible), in C# (Appium compatible), , (compatible with LDTP), , in Ruby (Linux Desktop Testing Project) ΠΈ Π΅Π³ΠΎ Windows version .
LDTP is perhaps the only cross-platform open-source tool (more precisely, a family of libraries) based on accessibility technologies. However, he is not very popular. I have not used it myself, but according to reviews, the interface is not the most convenient. If there are positive reviews, please share in the comments.
Test backdoor (aka inside bike)
For cross-platform applications, the developers themselves often make an internal mechanism to ensure testability. For example, they create a service TCP server in the application, tests connect to it and send text commands: what to click on, where to get data from, etc. Reliable, but not universal.
Main desktop accessibility technologies
Good old Win32 API
Most Windows ΠΏΡΠΈΠ»ΠΎΠΆΠ΅Π½ΠΈΠΉ, Π½Π°ΠΏΠΈΡΠ°Π½Π½ΡΡ Π΄ΠΎ Π²ΡΡ ΠΎΠ΄Π° WPF ΠΈ Π·Π°ΡΠ΅ΠΌ Windows Store, ΠΏΠΎΡΡΡΠΎΠ΅Π½Ρ ΡΠ°ΠΊ ΠΈΠ»ΠΈ ΠΈΠ½Π°ΡΠ΅ Π½Π° Win32 API. Π ΠΈΠΌΠ΅Π½Π½ΠΎ, MFC, WTL, C++ Builder, Delphi, VB6 β Π²ΡΠ΅ ΡΡΠΈ ΠΈΠ½ΡΡΡΡΠΌΠ΅Π½ΡΡ ΠΈΡΠΏΠΎΠ»ΡΠ·ΡΡΡ Win32 API. ΠΠ°ΠΆΠ΅ Windows Forms β Π² Π·Π½Π°ΡΠΈΡΠ΅Π»ΡΠ½ΠΎΠΉ ΡΡΠ΅ΠΏΠ΅Π½ΠΈ Win32 API ΡΠΎΠ²ΠΌΠ΅ΡΡΠΈΠΌΡΠ΅.
Tools: (similar to VB) and Python wrapper , (own language, there is an IDispatch COM interface), (Python) (Ruby) (Ruby).
Microsoft UI Automation
ΠΠ»Π°Π²Π½ΡΠΉ ΠΏΠ»ΡΡ: ΡΠ΅Ρ Π½ΠΎΠ»ΠΎΠ³ΠΈΡ MS UI Automation ΠΏΠΎΠ΄Π΄Π΅ΡΠΆΠΈΠ²Π°Π΅Ρ ΠΏΠΎΠ΄Π°Π²Π»ΡΡΡΠ΅Π΅ Π±ΠΎΠ»ΡΡΠΈΠ½ΡΡΠ²ΠΎ GUI ΠΏΡΠΈΠ»ΠΎΠΆΠ΅Π½ΠΈΠΉ Π½Π° Windows Π·Π° ΡΠ΅Π΄ΠΊΠΈΠΌΠΈ ΠΈΡΠΊΠ»ΡΡΠ΅Π½ΠΈΡΠΌΠΈ. ΠΡΠΎΠ±Π»Π΅ΠΌΠ°: ΠΎΠ½Π° Π½Π΅ ΡΠΈΠ»ΡΠ½ΠΎ Π»Π΅Π³ΡΠ΅ Π² ΠΈΠ·ΡΡΠ΅Π½ΠΈΠΈ, ΡΠ΅ΠΌ Win32 API. ΠΠ½Π°ΡΠ΅ Π½ΠΈΠΊΡΠΎ Π±Ρ Π½Π΅ Π΄Π΅Π»Π°Π» ΠΎΠ±Π΅ΡΡΠΎΠΊ Π½Π°Π΄ Π½Π΅ΠΉ.
In fact, this is a set of custom COM interfaces (mainly UIAutomationCore.dll) and also has a .NET wrapper in the form namespace System.Windows.Automation. By the way, it has an introduced bug, due to which some UI elements can be skipped. Therefore, it is better to use UIAutomationCore.dll directly (if you heard about UiaComWrapper in C #, then this is it).
Varieties of COM interfaces:
(1) Base IUknown is "the root of all evil". The most low-level, never user-friendly.
(2) IDispatch and derivatives (for example, Excel.Application) that can be used in Python using the win32com.client package (included with pyWin32). The most convenient and beautiful option.
(3) Custom interfaces that a third-party Python package can work with .
Tools: in C# 0.6.0 + in C# (their source code for sish wrappers over UIAutomationCore.dll is not disclosed), on Ruby.
AT-SPI
ΠΠ΅ΡΠΌΠΎΡΡΡ Π½Π° ΡΠΎ, ΡΡΠΎ ΠΏΠΎΡΡΠΈ Π²ΡΠ΅ ΠΎΡΠΈ ΡΠ΅ΠΌΠ΅ΠΉΡΡΠ²Π° Linux ΠΏΠΎΡΡΡΠΎΠ΅Π½Ρ Π½Π° X Window System (Π² Fedora 25 Β«ΠΈΠΊΡΡΒ» ΠΏΠΎΠΌΠ΅Π½ΡΠ»ΠΈ Π½Π° Wayland), Β«ΠΈΠΊΡΡΒ» ΠΏΠΎΠ·Π²ΠΎΠ»ΡΡΡ ΠΎΠΏΠ΅ΡΠΈΡΠΎΠ²Π°ΡΡ ΡΠΎΠ»ΡΠΊΠΎ ΠΎΠΊΠ½Π°ΠΌΠΈ Π²Π΅ΡΡ Π½Π΅Π³ΠΎ ΡΡΠΎΠ²Π½Ρ ΠΈ ΠΌΡΡΡΡ/ΠΊΠ»Π°Π²ΠΈΠ°ΡΡΡΠΎΠΉ. ΠΠ»Ρ Π΄Π΅ΡΠ°Π»ΡΠ½ΠΎΠ³ΠΎ ΡΠ°Π·Π±ΠΎΡΠ° ΠΏΠΎ ΠΊΠ½ΠΎΠΏΠΊΠ°ΠΌ, Π»ΠΈΡΡ Π±ΠΎΠΊΡΠ°ΠΌ ΠΈ ΡΠ°ΠΊ Π΄Π°Π»Π΅Π΅ β ΡΡΡΠ΅ΡΡΠ²ΡΠ΅Ρ ΡΠ΅Ρ Π½ΠΎΠ»ΠΎΠ³ΠΈΡ AT-SPI. Π£ ΡΠ°ΠΌΡΡ ΠΏΠΎΠΏΡΠ»ΡΡΠ½ΡΡ ΠΎΠΊΠΎΠ½Π½ΡΡ ΠΌΠ΅Π½Π΅Π΄ΠΆΠ΅ΡΠΎΠ² Π΅ΡΡΡ ΡΠ°ΠΊ Π½Π°Π·ΡΠ²Π°Π΅ΠΌΡΠΉ AT-SPI registry Π΄Π΅ΠΌΠΎΠ½, ΠΊΠΎΡΠΎΡΡΠΉ ΠΈ ΠΎΠ±Π΅ΡΠΏΠ΅ΡΠΈΠ²Π°Π΅Ρ Π΄Π»Ρ ΠΏΡΠΈΠ»ΠΎΠΆΠ΅Π½ΠΈΠΉ Π°Π²ΡΠΎΠΌΠ°ΡΠΈΠ·ΠΈΡΡΠ΅ΠΌΡΠΉ GUI (ΠΊΠ°ΠΊ ΠΌΠΈΠ½ΠΈΠΌΡΠΌ ΠΏΠΎΠ΄Π΄Π΅ΡΠΆΠΈΠ²Π°ΡΡΡΡ Qt ΠΈ GTK).
Tools: .
pyatspi2, in my opinion, contains too many dependencies like the same PyGObject. The technology itself is available as a regular dynamic library libatspi.so. She has . ΠΠ»Ρ Π±ΠΈΠ±Π»ΠΈΠΎΡΠ΅ΠΊΠΈ pywinauto ΠΏΠ»Π°Π½ΠΈΡΡΠ΅ΠΌ ΡΠ΅Π°Π»ΠΈΠ·ΠΎΠ²Π°ΡΡ ΠΏΠΎΠ΄Π΄Π΅ΡΠΆΠΊΡ AT-SPI ΠΈΠΌΠ΅Π΅Π½ΠΎ ΡΠ°ΠΊ: ΡΠ΅ΡΠ΅Π· Π·Π°Π³ΡΡΠ·ΠΊΡ libatspi.so ΠΈ ΠΌΠΎΠ΄ΡΠ»Ρ ctypes. ΠΡΡΡ Π½Π΅Π±ΠΎΠ»ΡΡΠ°Ρ ΠΏΡΠΎΠ±Π»Π΅ΠΌΠ° ΡΠΎΠ»ΡΠΊΠΎ Π² ΠΈΡΠΏΠΎΠ»ΡΠ·ΠΎΠ²Π°Π½ΠΈΠΈ Π½ΡΠΆΠ½ΠΎΠΉ Π²Π΅ΡΡΠΈΠΈ, Π²Π΅Π΄Ρ Π΄Π»Ρ GTK+ ΠΈ Qt ΠΏΡΠΈΠ»ΠΎΠΆΠ΅Π½ΠΈΠΉ ΠΎΠ½ΠΈ Π½Π΅ΠΌΠ½ΠΎΠ³ΠΎ ΡΠ°Π·Π½ΡΠ΅. ΠΠ΅ΡΠΎΡΡΠ½ΡΠΉ Π²ΡΠΏΡΡΠΊ pywinauto 0.7.0 Ρ ΠΏΠΎΠ»Π½ΠΎΡΠ΅Π½Π½ΠΎΠΉ ΠΏΠΎΠ΄Π΄Π΅ΡΠΆΠΊΠΎΠΉ Linux ΠΌΠΎΠΆΠ½ΠΎ ΠΎΠΆΠΈΠ΄Π°ΡΡ Π² ΠΏΠ΅ΡΠ²ΠΎΠΉ ΠΏΠΎΠ»ΠΎΠ²ΠΈΠ½Π΅ 2018-Π³ΠΎ.
Apple Accessibility API
MacOS has its own automation language, AppleScript. To implement something like this in Python, of course, you need to use functions from ObjectiveC. Starting, it seems, even with MacOS 10.6, the pyobjc package is included in the pre-installed python. This will also make it easier to list dependencies for future support in pywinauto.
Tools: In addition to the Apple Script language, you should pay attention to , aka pyatom. It is interface compatible with LDTP, but is also a standalone library. It has written by my student. There is a known issue: flexible timings do not work (methods waitFor*). But, in general, a good thing.
How to get started with pywinauto
The first step is to equip yourself with a GUI object inspector (what is called the Spy tool). It will help to study the application from the inside: how the hierarchy of elements is arranged, what properties are available. The most famous object inspectors are:
- Spy++ - included with Visual Studio, including Express or Community Edition. Uses Win32 API. Also known as a clone AutoIt Window Info.
- Inspect.exe β is included in Windows SDK. ΠΡΠ»ΠΈ ΠΎΠ½ Ρ Π²Π°Ρ ΡΡΡΠ°Π½ΠΎΠ²Π»Π΅Π½, ΡΠΎ Π½Π° 64-Π±ΠΈΡΠ½ΠΎΠΉ Windows ΠΌΠΎΠΆΠ½ΠΎ Π½Π°ΠΉΡΠΈ Π΅Π³ΠΎ Π² ΠΏΠ°ΠΏΠΊΠ΅
C:Program Files (x86)Windows Kits<winver>binx64. In the inspector itself, you need to select the mode UI Automation instead of MS AA (Active Accessibility, ancestor of UI Automation).
Having enlightened the application through and through, we select the backend that we will use. It is enough to specify the name of the backend when creating the Application object.
- backend="win32" - while used by default, works well with MFC, WTL, VB6 and other legacy applications.
- backend="uia" β Π½ΠΎΠ²ΡΠΉ Π±ΡΠΊΠ΅Π½Π΄ Π΄Π»Ρ MS UI Automation: ΠΈΠ΄Π΅Π°Π»ΡΠ½ΠΎ ΡΠ°Π±ΠΎΡΠ°Π΅Ρ Ρ WPF ΠΈ WinForms; ΡΠ°ΠΊΠΆΠ΅ Ρ
ΠΎΡΠΎΡ Π΄Π»Ρ Delphi ΠΈ Windows Store ΠΏΡΠΈΠ»ΠΎΠΆΠ΅Π½ΠΈΠΉ; ΡΠ°Π±ΠΎΡΠ°Π΅Ρ Ρ Qt5 ΠΈ Π½Π΅ΠΊΠΎΡΠΎΡΡΠΌΠΈ Java ΠΏΡΠΈΠ»ΠΎΠΆΠ΅Π½ΠΈΡΠΌΠΈ. Π Π²ΠΎΠΎΠ±ΡΠ΅, Π΅ΡΠ»ΠΈ Inspect.exe Π²ΠΈΠ΄ΠΈΡ ΡΠ»Π΅ΠΌΠ΅Π½ΡΡ ΠΈ ΠΈΡ
ΡΠ²ΠΎΠΉΡΡΠ²Π°, Π·Π½Π°ΡΠΈΡ ΡΡΠΎΡ Π±ΡΠΊΠ΅Π½Π΄ ΠΏΠΎΠ΄Ρ
ΠΎΠ΄ΠΈΡ. Π ΠΏΡΠΈΠ½ΡΠΈΠΏΠ΅, Π±ΠΎΠ»ΡΡΠΈΠ½ΡΡΠ²ΠΎ Π±ΡΠ°ΡΠ·Π΅ΡΠΎΠ² ΡΠΎΠΆΠ΅ ΠΏΠΎΠ΄Π΄Π΅ΡΠΆΠΈΠ²Π°Π΅Ρ UI Automation (Mozilla ΠΏΠΎ ΡΠΌΠΎΠ»ΡΠ°Π½ΠΈΡ, Π° Π₯ΡΠΎΠΌΡ ΠΏΡΠΈ Π·Π°ΠΏΡΡΠΊΠ΅ Π½ΡΠΆΠ½ΠΎ ΡΠΊΠΎΡΠΌΠΈΡΡ ΠΊΠ»ΡΡ ΠΊΠΎΠΌΠ°Π½Π΄Π½ΠΎΠΉ ΡΡΡΠΎΠΊΠΈ
--force-renderer-accessibilityto see items on pages in Inspect.exe). Of course, competition with Selenium in this area is hardly possible. Just another way to work with the browser (may come in handy for a cross-product scenario).
Entry points for automation
The app has been well researched. It's time to create an Application object and run it, or attach to an already running one. It's not just a clone of the standard class subprocess.Popen, which is an introductory object that limits all your actions to the boundaries of the process. This is very useful if multiple instances of an application are running and you don't want to touch the rest.
from pywinauto.application import Application
app = Application(backend="uia").start('notepad.exe')
# ΠΠΏΠΈΡΠ΅ΠΌ ΠΎΠΊΠ½ΠΎ, ΠΊΠΎΡΠΎΡΠΎΠ΅ Ρ
ΠΎΡΠΈΠΌ Π½Π°ΠΉΡΠΈ Π² ΠΏΡΠΎΡΠ΅ΡΡΠ΅ Notepad.exe
dlg_spec = app.UntitledNotepad
# ΠΆΠ΄Π΅ΠΌ ΠΏΠΎΠΊΠ° ΠΎΠΊΠ½ΠΎ ΡΠ΅Π°Π»ΡΠ½ΠΎ ΠΏΠΎΡΠ²ΠΈΡΡΡ
actionable_dlg = dlg_spec.wait('visible')If you want to manage several applications at once, the class will help you Desktop. For example, in the calculator on Win10, the hierarchy of elements is spread over several processes (not only calc.exe). So no object Desktop can not do.
from subprocess import Popen
from pywinauto import Desktop
Popen('calc.exe', shell=True)
dlg = Desktop(backend="uia").Calculator
dlg.wait('visible')Root object (Application or Desktop) is the only place where you need to specify the backend. Everything else transparently falls into the concept of "specification-> wrapper", which will be discussed later.
Window/element specifications
This is the core concept on which the pywinauto interface is built. You can describe the window/element roughly or in more detail, even if it does not yet exist or is already closed. window specification (object window specification) stores the criteria by which you need to search for a real window or element.
An example of a detailed window specification:
>>> dlg_spec = app.window(title='Untitled - Notepad')
>>> dlg_spec
<pywinauto.application.WindowSpecification object at 0x0568B790>
>>> dlg_spec.wrapper_object()
<pywinauto.controls.win32_controls.DialogWrapper object at 0x05639B70>The search for the window itself occurs by calling the method .wrapper_object(). It returns some "wrapper" for a real window/element or throws ElementNotFoundError (sometimes ElementAmbiguousError, if multiple elements are found, that is, you need to refine the search criteria). This "wrapper" already knows how to do some actions with the element or receive data from it.
Python can hide the call .wrapper_object(), so that the final code becomes shorter. We recommend using it for debugging purposes only. The next two lines do exactly the same thing:
dlg_spec.wrapper_object().minimize() # debugging
dlg_spec.minimize() # productionThere are a variety of search criteria for a window specification. Here are just a few examples:
# ΠΌΠΎΠ³ΡΡ ΠΈΠΌΠ΅ΡΡ Π½Π΅ΡΠΊΠΎΠ»ΡΠΊΠΎ ΡΡΠΎΠ²Π½Π΅ΠΉ
app.window(title_re='.* - Notepad$').window(class_name='Edit')
# ΠΌΠΎΠΆΠ½ΠΎ ΠΊΠΎΠΌΠ±ΠΈΠ½ΠΈΡΠΎΠ²Π°ΡΡ ΠΊΡΠΈΡΠ΅ΡΠΈΠΈ (ΠΊΠ°ΠΊ AND) ΠΈ Π½Π΅ ΠΎΠ³ΡΠ°Π½ΠΈΡΠΈΠ²Π°ΡΡΡΡ ΠΎΠ΄Π½ΠΈΠΌ ΠΏΡΠΎΡΠ΅ΡΡΠΎΠΌ ΠΏΡΠΈΠ»ΠΎΠΆΠ΅Π½ΠΈΡ
dlg = Desktop(backend="uia").Calculator
dlg.window(auto_id='num8Button', control_type='Button')The list of all possible criteria is in the function docks .
The magic of access by attribute and by key
Python makes it easy to create window specifications and recognize object attributes dynamically (internally overridden __getattribute__). Of course, the same restrictions are imposed on the attribute name as on the name of any variable (you cannot insert spaces, commas, and other special characters). Luckily, pywinauto uses a so-called "best match" search algorithm that is resistant to typos and slight variations.
app.UntitledNotepad
# ΡΠΎ ΠΆΠ΅ ΡΠ°ΠΌΠΎΠ΅, ΡΡΠΎ
app.window(best_match='UntitledNotepad')If you still need Unicode strings (for example, for the Russian language), spaces, etc., you can access by key (as if it were a regular dictionary):
app['Untitled - Notepad']
# ΡΠΎ ΠΆΠ΅ ΡΠ°ΠΌΠΎΠ΅, ΡΡΠΎ
app.window(best_match='Untitled - Notepad')Five Rules for Magical Names
How to find out the reference magical names? Those that are assigned to the element before the search. If you have specified a name that is sufficiently similar to the template, then the element will be found.
- By title (text, name):
app.Properties.OK.click() - By text and by element type:
app.Properties.OKButton.click() - By type and by number:
app.Properties.Button3.click()(namesButton0ΠΈButton1bound to the first element found,Button2- to the second, and then in order - it happened historically) - By static text (left or top) and by type:
app.OpenDialog.FileNameEdit.set_text("")(useful for elements with dynamic text) - By type and by text inside:
app.Properties.TabControlSharing.select("General")
Usually two or three rules are applied at the same time, rarely more. To check which specific names are available for each element, you can use the method print_control_identifiers(). It can print a tree of elements both to the screen and to a file. For each element, its reference magic names are printed. You can also copy-paste more detailed specifications of child elements from there. The result in the script will look like this:
app.Properties.child_window(data-gt-translate-attributes='["title"]' title="Contains:", auto_id="13087", control_type="Edit")The tree of elements itself is usually a rather large footcloth.
>>> app.Properties.print_control_identifiers()
Control Identifiers:
Dialog - 'Windows NT Properties' (L688, T518, R1065, B1006)
[u'Windows NT PropertiesDialog', u'Dialog', u'Windows NT Properties']
child_window(data-gt-translate-attributes='["title"]' title="Windows NT Properties", control_type="Window")
|
| Image - '' (L717, T589, R749, B622)
| [u'', u'0', u'Image1', u'Image0', 'Image', u'1']
| child_window(auto_id="13057", control_type="Image")
|
| Image - '' (L717, T630, R1035, B632)
| ['Image2', u'2']
| child_window(auto_id="13095", control_type="Image")
|
| Edit - 'Folder name:' (L790, T596, R1036, B619)
| [u'3', 'Edit', u'Edit1', u'Edit0']
| child_window(data-gt-translate-attributes='["title"]' title="Folder name:", auto_id="13156", control_type="Edit")
|
| Static - 'Type:' (L717, T643, R780, B658)
| [u'Type:Static', u'Static', u'Static1', u'Static0', u'Type:']
| child_window(data-gt-translate-attributes='["title"]' title="Type:", auto_id="13080", control_type="Text")
|
| Edit - 'Type:' (L790, T643, R1036, B666)
| [u'4', 'Edit2', u'Type:Edit']
| child_window(data-gt-translate-attributes='["title"]' title="Type:", auto_id="13059", control_type="Edit")
|
| Static - 'Location:' (L717, T669, R780, B684)
| [u'Location:Static', u'Location:', u'Static2']
| child_window(data-gt-translate-attributes='["title"]' title="Location:", auto_id="13089", control_type="Text")
|
| Edit - 'Location:' (L790, T669, R1036, B692)
| ['Edit3', u'Location:Edit', u'5']
| child_window(data-gt-translate-attributes='["title"]' title="Location:", auto_id="13065", control_type="Edit")
|
| Static - 'Size:' (L717, T695, R780, B710)
| [u'Size:Static', u'Size:', u'Static3']
| child_window(data-gt-translate-attributes='["title"]' title="Size:", auto_id="13081", control_type="Text")
|
| Edit - 'Size:' (L790, T695, R1036, B718)
| ['Edit4', u'6', u'Size:Edit']
| child_window(data-gt-translate-attributes='["title"]' title="Size:", auto_id="13064", control_type="Edit")
|
| Static - 'Size on disk:' (L717, T721, R780, B736)
| [u'Size on disk:', u'Size on disk:Static', u'Static4']
| child_window(data-gt-translate-attributes='["title"]' title="Size on disk:", auto_id="13107", control_type="Text")
|
| Edit - 'Size on disk:' (L790, T721, R1036, B744)
| ['Edit5', u'7', u'Size on disk:Edit']
| child_window(data-gt-translate-attributes='["title"]' title="Size on disk:", auto_id="13106", control_type="Edit")
|
| Static - 'Contains:' (L717, T747, R780, B762)
| [u'Contains:1', u'Contains:0', u'Contains:Static', u'Static5', u'Contains:']
| child_window(data-gt-translate-attributes='["title"]' title="Contains:", auto_id="13088", control_type="Text")
|
| Edit - 'Contains:' (L790, T747, R1036, B770)
| [u'8', 'Edit6', u'Contains:Edit']
| child_window(data-gt-translate-attributes='["title"]' title="Contains:", auto_id="13087", control_type="Edit")
|
| Image - 'Contains:' (L717, T773, R1035, B775)
| [u'Contains:Image', 'Image3', u'Contains:2']
| child_window(data-gt-translate-attributes='["title"]' title="Contains:", auto_id="13096", control_type="Image")
|
| Static - 'Created:' (L717, T786, R780, B801)
| [u'Created:', u'Created:Static', u'Static6', u'Created:1', u'Created:0']
| child_window(data-gt-translate-attributes='["title"]' title="Created:", auto_id="13092", control_type="Text")
|
| Edit - 'Created:' (L790, T786, R1036, B809)
| [u'Created:Edit', 'Edit7', u'9']
| child_window(data-gt-translate-attributes='["title"]' title="Created:", auto_id="13072", control_type="Edit")
|
| Image - 'Created:' (L717, T812, R1035, B814)
| [u'Created:Image', 'Image4', u'Created:2']
| child_window(data-gt-translate-attributes='["title"]' title="Created:", auto_id="13097", control_type="Image")
|
| Static - 'Attributes:' (L717, T825, R780, B840)
| [u'Attributes:Static', u'Static7', u'Attributes:']
| child_window(data-gt-translate-attributes='["title"]' title="Attributes:", auto_id="13091", control_type="Text")
|
| CheckBox - 'Read-only (Only applies to files in folder)' (L790, T825, R1035, B841)
| [u'CheckBox0', u'CheckBox1', 'CheckBox', u'Read-only (Only applies to files in folder)CheckBox', u'Read-only (Only applies to files in folder)']
| child_window(data-gt-translate-attributes='["title"]' title="Read-only (Only applies to files in folder)", auto_id="13075", control_type="CheckBox")
|
| CheckBox - 'Hidden' (L790, T848, R865, B864)
| ['CheckBox2', u'HiddenCheckBox', u'Hidden']
| child_window(data-gt-translate-attributes='["title"]' title="Hidden", auto_id="13076", control_type="CheckBox")
|
| Button - 'Advanced...' (L930, T845, R1035, B868)
| [u'Advanced...', u'Advanced...Button', 'Button', u'Button1', u'Button0']
| child_window(data-gt-translate-attributes='["title"]' title="Advanced...", auto_id="13154", control_type="Button")
|
| Button - 'OK' (L814, T968, R889, B991)
| ['Button2', u'OK', u'OKButton']
| child_window(data-gt-translate-attributes='["title"]' title="OK", auto_id="1", control_type="Button")
|
| Button - 'Cancel' (L895, T968, R970, B991)
| ['Button3', u'CancelButton', u'Cancel']
| child_window(data-gt-translate-attributes='["title"]' title="Cancel", auto_id="2", control_type="Button")
|
| Button - 'Apply' (L976, T968, R1051, B991)
| ['Button4', u'ApplyButton', u'Apply']
| child_window(data-gt-translate-attributes='["title"]' title="Apply", auto_id="12321", control_type="Button")
|
| TabControl - '' (L702, T556, R1051, B962)
| [u'10', u'TabControlSharing', u'TabControlPrevious Versions', u'TabControlSecurity', u'TabControl', u'TabControlCustomize']
| child_window(auto_id="12320", control_type="Tab")
| |
| | TabItem - 'General' (L704, T558, R753, B576)
| | [u'GeneralTabItem', 'TabItem', u'General', u'TabItem0', u'TabItem1']
| | child_window(data-gt-translate-attributes='["title"]' title="General", control_type="TabItem")
| |
| | TabItem - 'Sharing' (L753, T558, R801, B576)
| | [u'Sharing', u'SharingTabItem', 'TabItem2']
| | child_window(data-gt-translate-attributes='["title"]' title="Sharing", control_type="TabItem")
| |
| | TabItem - 'Security' (L801, T558, R851, B576)
| | [u'Security', 'TabItem3', u'SecurityTabItem']
| | child_window(data-gt-translate-attributes='["title"]' title="Security", control_type="TabItem")
| |
| | TabItem - 'Previous Versions' (L851, T558, R947, B576)
| | [u'Previous VersionsTabItem', u'Previous Versions', 'TabItem4']
| | child_window(data-gt-translate-attributes='["title"]' title="Previous Versions", control_type="TabItem")
| |
| | TabItem - 'Customize' (L947, T558, R1007, B576)
| | [u'CustomizeTabItem', 'TabItem5', u'Customize']
| | child_window(data-gt-translate-attributes='["title"]' title="Customize", control_type="TabItem")
|
| TitleBar - 'None' (L712, T521, R1057, B549)
| ['TitleBar', u'11']
| |
| | Menu - 'System' (L696, T526, R718, B548)
| | [u'System0', u'System', u'System1', u'Menu', u'SystemMenu']
| | child_window(data-gt-translate-attributes='["title"]' title="System", auto_id="MenuBar", control_type="MenuBar")
| | |
| | | MenuItem - 'System' (L696, T526, R718, B548)
| | | [u'System2', u'MenuItem', u'SystemMenuItem']
| | | child_window(data-gt-translate-attributes='["title"]' title="System", control_type="MenuItem")
| |
| | Button - 'Close' (L1024, T519, R1058, B549)
| | [u'CloseButton', u'Close', 'Button5']
| | child_window(data-gt-translate-attributes='["title"]' title="Close", control_type="Button")In some cases, printing the entire tree can slow down (for example, in iTunes there are as many as three thousand items on one tab!), But you can use the parameter depth (depth): depth=1 - the element itself depth=2 β only immediate children, and so on. It can also be specified in the specifications when creating child_window.
Examples
We are constantly replenishing . Of the fresh ones, it is worth noting the automation of the WireShark network analyzer (this is a good example of a Qt5 application; although this task can be solved without a GUI, because there are scapy.Sniffer from python package ). There is also an example of MS Paint automation with its Ribbon toolbar.
Another great example written by my student: (it will migrate to the main repository a little later).
And, of course, an example of subscribing to keyboard (hot keys) and mouse events:
.
Acknowledgements
Special thanks to those who constantly help to develop the project. For me and it's an ongoing hobby. Two of my students from UNN recently completed their bachelor's degrees in this topic. made a great contribution to MS UI Automation support and recently started making an automatic code generator based on the "record-play" principle based on text properties (this is the most difficult feature), so far only for the "uia" backend. ΡΠ°Π·ΡΠ°Π±Π°ΡΡΠ²Π°Π΅Ρ Π½ΠΎΠ²ΡΠΉ Π±ΡΠΊΠ΅Π½Π΄ ΠΏΠΎΠ΄ Linux Π½Π° ΠΎΡΠ½ΠΎΠ²Π΅ AT-SPI (ΠΌΠΎΠ΄ΡΠ»ΠΈ mouse ΠΈ keyboard on the basis of - already in 0.6.x releases).
Since I have been teaching a special course on automation in Python for quite some time, some of the master's students do their homework, implementing small features or examples of automation. Some key things at the research stage were also once unearthed by students. Although sometimes you have to strictly monitor the quality of the code. This is greatly helped by static analyzers (QuantifiedCode, Codacy and Landscape) and automated tests in the cloud (AppVeyor service) with code coverage of around 95%.
Also thanks to everyone who leaves feedback, starts bugs and sends pull requests!
Additional resources
We follow questions on (recently appeared ) and . There is .
We update every month . In terms of the number of stars on the github, only Autohotkey (they have a very large community and a long history) and PyAutoGUI (largely due to the popularity of books by its author Al Sweigart: "Automate the Boring Stuff with Python" and others) are growing faster.
Source: habr.com
