We present to you the third part of the translation of the material about the journey that Dropbox has taken in implementing a type checking system for Python code.
→ Previous parts: and
Achieving 4 million lines of typed code
Another important task (this was the second most popular issue that concerned those participating in internal surveys) was increasing the amount of code in Dropbox covered by type checks. We tried several approaches to solve this problem — from the natural growth of the typed codebase to focusing the efforts of mypy team members on static and dynamic automated type inference. Ultimately, it seemed that there was no simple winning strategy, but we were able to achieve rapid growth in the volume of annotated code by combining multiple approaches.
As a result, in our largest Python repository (with backend code), the number of annotated code lines reached nearly 4 million. The work on static type checking was carried out over about three years. Mypy now supports various types of coverage reports for types that simplify monitoring the progress of typing. In particular, we can generate reports on code with type uncertainties, such as explicit use of types in annotations that cannot be checked, or cases involving imports of third-party libraries that lack type annotations. As part of our project to improve the accuracy of type checking in Dropbox, we contributed to enhancing the type definitions (so-called stub files) for some popular open-source libraries in the centralized Python repository. Any typeshed .
TypeDict , which provides types for JSON-like dictionaries that have a fixed set of string keys, each with its own type value. We will continue to expand the type system. Our next step will likely be to improve support for Python's capabilities related to handling numbers.Number of annotated code lines: server
Number of lines of annotated code: server
Number of lines of annotated code: client
Total number of lines of annotated code
Here is an overview of the main features of the actions we took to increase the volume of annotated code in Dropbox:
Annotation rigor. We gradually raised the standards for annotating new code. We started with linter suggestions that recommended adding annotations in files that already had some annotations. Now we require type annotations in new Python files and in most existing files.
Typing reports. We send weekly reports to teams about their code's typing level and provide advice on what should be annotated first.
Promoting mypy. We discuss mypy at various events and engage with teams to help them start using type annotations.
Surveys. We conduct periodic user surveys to identify key issues. We are willing to go to great lengths to solve these problems (even creating a new language to speed up mypy!).
Performance. We significantly improved mypy's performance by using a daemon and mypyc. This was done to smooth out the inconveniences that arise during annotation and to enable work with large volumes of code.
Integration with editors. We created tools to support running mypy in popular editors at Dropbox. These include PyCharm, Vim, and VS Code. This greatly simplified the process of annotating code and verifying its functionality. Such actions are typically necessary when annotating existing code.
Static analysis. We developed a tool for outputting function signatures using static analysis tools. This tool can only operate in relatively simple situations but has helped us increase type coverage without much effort.
Support for third-party libraries. Many of our projects use the SQLAlchemy toolkit. It employs Python's dynamic capabilities, which PEP 484 types cannot model directly. In accordance with PEP 561, we created a corresponding stub file and wrote a plugin for mypy.), improving SQLAlchemy support.
Challenges we faced
The journey to 4 million lines of typed code was not always easy for us. Along the way, we encountered many pitfalls and made several mistakes. Here are some of the issues we faced. We hope that sharing these experiences will help others avoid similar pitfalls.
Missing files. We started by reviewing only a small number of files. Everything not included in this list was not checked. Files were added to the review list only when the first annotations appeared. If something was imported from a module outside the review scope, it referred to working with values of type Any, which were not checked at all. This resulted in a significant loss of type accuracy, especially in the early stages of migration. This approach worked surprisingly well, although it was typical for the situation where adding files to the review area revealed problems in other parts of the codebase. In the worst case, when two isolated areas of code were merged, both of which had already independently verified types, it turned out that the types in these areas were incompatible with each other. This necessitated making many changes to the annotations. Now, looking back, we realize that we should have added the basic library modules to the type check as early as possible. This would have made our work much more predictable.
Annotating Legacy Code. When we started the project, we had about 4 million lines of existing Python code. It was clear that annotating all of this code would be a daunting task. We created a tool called PyAnnotate, which can gather type information during test execution and add type annotations to the code based on the collected data. However, we did not observe widespread adoption of this tool. Gathering type information was slow, and the automatically generated annotations often required extensive manual adjustments. We considered automatically running this tool with every code review, or collecting type information based on the analysis of a small volume of real network requests, but ultimately decided against it, as either approach was too risky.
In conclusion, it can be noted that the majority of the code was manually annotated by its owners. To guide this process in the right direction, we prepare reports on particularly important modules and functions that need to be annotated. For instance, it is important to provide type annotations for the library module that is used in hundreds of places. However, annotating the old service that is being replaced with a new one is not as critical. Additionally, we are experimenting with using static analysis to generate type annotations for legacy code.
Cyclic imports. Earlier, I discussed cyclic imports (the "dependency knot"), the existence of which complicated the acceleration of mypy. Furthermore, we had to work hard to equip mypy with support for all kinds of idioms that arise from these cyclic imports. Recently, we completed a major redesign project that resolved most of the mypy issues regarding cyclic imports. These issues actually stemmed from the very early days of the project, back in Alore, the educational language that mypy was originally oriented towards. The Alore syntax easily addresses the challenges of cyclic import commands. Modern mypy inherited some limitations from its early straightforward implementation (which worked well for Alore). Python complicates working with cyclic imports mainly due to expression ambiguity. For instance, during an assignment operation, a type alias may actually be defined. Mypy does not always identify such cases until a large part of the import cycle has been processed. In Alore, there were no such ambiguities. Poor decisions made in the early stages of system development can present a programmer with an unpleasant surprise years later.
Outcomes: The Journey to 5 Million Lines of Code and New Horizons
The mypy project has come a long way — from early prototypes to a system that controls types in production code totaling 4 million lines. Throughout the development of mypy, standardization of type hints in Python was achieved. Nowadays, a robust ecosystem has developed around Python code typing. It includes library support, tools for IDEs and editors, and several type-checking systems, each with its own advantages and disadvantages.
Although type checking is already taken for granted at Dropbox, I believe we are still in the dawn of Python code typing. I think that type-checking technologies will continue to evolve and improve.
If you haven't yet used type checks in your large-scale Python project, now is a very appropriate time to start transitioning to static typing. I've spoken with those who have made this transition. None of them regret it. Type checking makes Python a language that is much better suited for developing large projects than "regular Python."
Dear readers! Do you use type checking in your Python projects?
Source: habr.com
