We continue to talk about the projects of the DevDays spring hackathon, which was attended by students of the master's program .
By the way, we would like to invite readers to join . In it we will publish the latest news about recruitment and study. A video from the open day can also be found in the group. We remind you: the event will be held on April 29, details .
Telegram Desktop Voice Message Parser
The author of idea
Khoroshev Artyom
Line-up
Khoroshev Artem - project manager / developer / QA
Eliseev Anton – Business Analyst/Marketing Specialist
Kuklina Maria - UI designer/developer
Bakhvalov Pavel - UI designer/developer/QA
From our point of view, Telegram is a modern and convenient messenger, and its version for PC is popular and open source, which makes it possible to modify it. The client offers quite rich functionality. In addition to standard text messages, it contains voice calls, video messages, voice messages. And it is the latter that sometimes bring inconvenience to their recipient. Often there is no way to listen to a voice message while at a computer or laptop. You may be disturbed by ambient noise, no headphones, or you don't want anyone to hear the contents of the message. Such problems almost never arise if you use Telegram on a smartphone, because you can simply bring it to your ear, unlike a laptop or PC. We tried to solve this problem.
The task of our DevDays project was to add the ability to translate received voice messages into text into the Telegram desktop client (hereinafter referred to as Telegram Desktop).
All analogues at the moment are bots to which you can send an audio message and receive a text in response. This does not suit us very much: it is not very convenient to forward a message to a bot, we would like to have native functionality. In addition, any bot is a third party that acts as an intermediary between the speech recognition API and the user, and this is at least unsafe.
As noted earlier, telegram-desktop has two weight advantages: lightness and speed. And this is no coincidence, because it is completely written in C ++. And since we decided to add new functionality directly to the client, we had to develop in C ++.
There were 4 people in our team. Initially, two people were looking for a suitable library for speech recognition, one person studied the Telegram-desktop source code, another one deployed the build of the project . Later, everyone was engaged in UI fixes and debugging.
It seemed that the implementation of the planned functionality would not be difficult, but, as always happens, difficulties arose.
The solution to the problem consisted of two independent subtasks: choosing a suitable tool for speech recognition and implementing a UI for new functionality.
When choosing a library for voice recognition, we immediately had to abandon all offline APIs, because language models take up a lot of space. But we are talking about only one language. It became clear that we would have to use the online API. Later it turned out that the speech recognition services of such giants as Google, Yandex and Microsoft are not free at all, and we will have to be content with a trial period. As a result, Google Speech-To-Text was chosen, as it allows you to get a token for using the service, which is enough for a whole year.
The second problem that we encountered is related to some of the shortcomings of C ++ - a zoo of various libraries in the absence of a centralized repository. It so happened that Telegram Desktop depends on many other libraries of specific versions. The official repository has for project assembly. As well as a large number of open issues about build problems, for example и . All the problems turned out to be related to the fact that the build script was written for Ubuntu 14.04, and in order to successfully build telegram under Ubuntu 18.04, we had to make changes.
By itself, Telegram Desktop is assembled for a long time: on a laptop with an Intel Core i5-7200U, a complete assembly (flag -j 4) with all dependencies takes about three hours. Of these, linking the client itself takes about 30 minutes (later it turned out that linking takes about 10 minutes in the Debug configuration), but the linking stage has to be repeated every time after making changes.
Despite the problems, we managed to implement the conceived idea, as well as update for Ubuntu 18.04. A demonstration of the work can be seen at . We also add some animations. A button appeared next to all voice messages that allows you to broadcast the message to text. By right-clicking, you can additionally specify the language that will be used for translation. By download client available.
In our opinion, we got a good Proof of Concept functionality that would be convenient for many users. We hope to see it in future releases of Telegram Desktop.
Enhanced natural language support in IntelliJ IDEA
The author of idea
Tankov Vladislav
Line-up
Tankov Vladislav (team leader, work with LanguageTool and IntelliJ IDEA)
Sokolov Nikita (working with LanguageTool and creating UI)
Khvorov Alexander (work with LanguageTool and performance optimization)
Sadovnikov Alexander (support for parsing markup languages and code)
We have developed a plug-in for IntelliJ IDEA that checks various texts (comments and documentation, literal lines in code, text formatted in Markdown or XML markup) for grammatical, spelling and stylistic correctness (in English this is called proofreading).
The idea of the project was to extend the standard IntelliJ IDEA spellcheck to the scale of Grammarly, to make a kind of Grammarly inside IDE.
You can see what happened .
Well, below we will talk in more detail about the capabilities of the plugin, as well as the difficulties that arose during its creation.
Motivation
There are many natural language writing products out there, but documentation and code comments are most often written in development environments. At the same time, IDEs do an excellent job of finding errors in writing code, but are poorly suited for texts in natural languages. This makes it very easy to make mistakes in grammar, punctuation, or style without the development environment pointing them out. It is most critical to make a mistake in writing the user interface, since this will affect not only the understandability of the code, but also the users of the developed application themselves.
One of the most popular and developed development environments is IntelliJ IDEA, as well as IDEs based on the IntelliJ Platform. The IntelliJ Platform already has a built-in spellchecker, but it does not get rid of even the simplest grammatical errors. We decided to integrate one of the popular natural language analysis systems into IntelliJ IDEA.
implementation
We did not set ourselves the task of creating our own text verification system, so we used the existing solution. The most suitable option was . The license allowed us to freely use it for our purposes: it is free, written in Java and published in open-source. In addition, it supports 25 languages and has been in development for over fifteen years. Despite its openness, LanguageTool is a serious competitor to paid text verification solutions, and the fact that it is able to work locally is literally its killer feature.
The plugin code is in . The entire project was written in Kotlin with a little addition of Java for the UI. During the hackathon, we managed to implement support for Markdown, JavaDoc, HTML, and Plain Text. After the hackathon, a major update added support for XML, string literals in Java, Kotlin and Python, as well as spell checking.
Трудности
Pretty quickly, we realized that if we feed all the text to the LanguageTool check every time, then the IDEA interface will hang on any more or less serious texts, since the inspection itself blocks the UI thread. The problem was solved through the `ProgressManager.checkCancelled` check - this function throws an exception if IDEA thinks that the inspection should be interrupted.
This completely got rid of freezes, but it is impossible to use it: the text is processed for a very long time. At the same time, in our case, most often a very small part of the text changes and we want to somehow cache the results. That is exactly what we did. In order not to check everything every time, we deterministically split the text into pieces and checked only those that have changed. Since the texts can be large and we didn’t want to load the cache, we didn’t store the texts themselves, but their hashes. This made it possible to achieve smooth operation of the plugin even on large files.
LanguageTool supports more than 25 languages, but it is unlikely that one user needs them all. I wanted to give the ability to download libraries for a specific language on request (if you ticked it in the UI). We even implemented it, but it turned out to be too complicated and unreliable. In particular, we had to load the LanguageTool with a new set of languages with a separate classloader, and then carefully initialize it. At the same time, all the libraries were in the user .m2 repository, and at each start we had to check their integrity. In the end, we decided that if users had problems with the size of the plugin, then we would supply a separate plugin for several of the most popular languages.
After the hackathon
The hackathon ended, but work on the plugin continued with a narrower staff. I wanted to support strings, comments, and even language constructs such as variable and class names. This is currently only supported for Java, Kotlin, and Python, but we hope this list will grow. We've fixed a lot of small bugs and made it more compatible with Idea's built-in spell checker. In addition, there was support for XML and spell checking. All this can be found in the second version, which we published recently.
What's next?
Such a plugin can be useful not only for developers, but also for technical writers (often working, for example, with XML in the IDE). Every day they have to work with natural language, while not having an assistant in the form of editor's tips about possible errors. Our plugin provides such hints and does so with a high degree of accuracy.
We plan to develop the plugin, both adding new languages and exploring a general approach to organizing text verification. In the near future, we plan to implement style profiles (sets of rules that define the style guide for text, for example, “don’t write eg, but write the full form”), expanding the dictionary and improving the user interface (in particular, we want to give the user the opportunity not just to ignore the word, but to add it into the dictionary, indicating the part of speech).
Source: www.habr.com