People love ratings. How many applications, games and other things have already been made in the name of a person's desire to be on some list a couple of lines higher than someone else. Or than a competitor, for example. People achieve places in the ranking in different ways, depending on motivation and moral character. Someone will try to become better and honestly step over from place #142 to #139, and someone will decide to put money in and happily take #21 (because the top 20 brought even more).
It's the same with companies. Today we will talk about banks and the ratings that these banks strive to get into. In this post, I will talk about the common problems with research that we have in the country, about the practical difference between quantitative and qualitative testing, and how we tried to correct this situation.
There is a surprise at the end of the article.
It all started with the fact that a year ago we started testing five banks for legal entities, choosing a couple of stylish youth ones (Modulbank and Tinkoff Bank) and three classic ones (VTB, Raiffeisenbank and Promsvyazbank). But first, a little materiel.
Bank ratings in the Russian Federation
There are quite a few players on the market who make usability ratings for the banking sector. Namely, as many as two - Markswebb and USABILITYLAB.
And it turned out that MW and UL have now become a kind of KPI. On the one hand, this is good, since the very presence of at least something competitive sets the general movement in a rather clumsy market in this regard. On the other hand, it all comes down mostly to functional analysis. And the motivation here from the bank tops is no longer to make an awesome product that will take off and bring a lot of benefits to users, thanks to which it will take a place in the rating, but simply to be in the rating.
Your bank is in the rating = you have met the KPI = you have received a bonus. Plus, the team seems to love you, helped the bank get into the rating. For some, this sincerely scratches the CSF. In general, who is in what much, but motivation, by and large, is such βbonusesβ of various kinds, and not a movement towards improving the product.
And here, in terms of the significance of such ratings for the market, it is important to understand one more thing. About 98% of users of banking applications do not know about these ratings at all. They frankly don't care. These ratings are for managers and management. The remaining 2% know about the ratings, but consider them a purchasable topic. Once we tested the sites of banks with these dies about the first places.
People do not choose a bank for business on the basis of whether there is a badge with the logo of one or another rating on the bank's website or not. Itβs easier for a person to call out to friends or on FB who uses which bank and what they are satisfied with / not satisfied with, and confine themselves to this in terms of social capital.
Let's start by creating a rating. To create a rating, you need to do research, and here everything is usually limited to researching one specific function, say, testing currency control.
And research costs money, and money is quite tangible. To do this efficiently, you need to invest well - a portrait of an entrepreneur for testing costs more than an average user. Therefore, companies that try to build their revenue only on research as their main and only activity, incur tangible costs. Moreover, our research market is almost empty: this is not taught in universities, it is not taught in schools.
By the way, about money, so that the numbers are clear. Let's say we have 20 banks in the rating. Each one needs to explore the top 7 features and scenarios, spending approximately 1,5 hours of time. It doesnβt make sense to conduct a test on one respondent longer, because an hour and a half is the limit, after which attention is already dissipated, and people get tired and begin to answer anything, just to quickly go to have a bite and finally exhale.
So. It is difficult and long to take people from the bank's base for such research, so there remains a recruit. 5-7 scenarios for 20 banks means that you need to recruit a minimum of 140 respondents. And then, if more than one bank is tested on one person
The cost of one such respondent varies between 5-10 thousand rubles, there is a clear dependence on the portrait, for example, a single entrepreneur will cost quite inexpensively, 5 thousand. But the portrait of an exporting entrepreneur with currency control will cost about 13 thousand.
In total, there are 140 people who need to be paid to participate in the study. Let's estimate the simplest and cheapest scenario, 5000 rubles per respondent, and we will get a non-illusory 700 rubles. At least yes. Usually, this figure approaches 000. It would be time to open your own recruitment agency π
And this is only for the main scenarios for using the bank. In addition to money, there is a more valuable resource - time. It is also spent with such a large hill on top. You can run tests with 30 respondents and not go crazy in 2 weeks. There are usually about 60 meetings per month if you want to maintain the quality of the interview. 140 people = 2,5 man-months.
After all the respondents, you need to spend about 2 more months to bring the information into a digestible form - to transcribe the results, analyze and group, make a beautiful presentation, and not a final Excel file with a bunch of lines.
In general, it turns out about 4 months of work and 2-3 million rubles, taking into account all the costs during this period. And we haven't calculated taxes yet. And given that so far no one has been able to make money on the research itself, such a model, obviously, does not look the most profitable. If you do not earn on the rating itself and places in it instead of research, of course.
Quantitative and qualitative research, functional analysis
MW presentations are about 60% functional analysis and 40% usability. Moreover, the concept of "functional analysis" in the case of such studies is simply a checklist of the presence of certain functions. You sit down, write a list of functions - so, there should be a normal payment, plus payment from a photo, and also from a file, checking a counterparty, recent counterparties or payments, and so on. Then you analyze and check whether there are functions from the list or not. If there is, great, put a tick, plus in the rating. If not, well, you get the idea.
Sounds logical. But, alas, it slides down to the fact that the plus and tick in such testing is simply the presence of a function in the list, and not its quality or even a need for the user. So mobile applications began to slip into stuffing everything into themselves in order to match the rating, and not what the user needs. Well, here's how Yandex.Phone has a dual camera. There is, but they say it doesn't work. But there is. In total, it turns out that 60% of the significance of such a rating is just a tick itself, whether there is a function or not. And not how it is convenient and necessary for the user.
In addition to functional analysis, there are also quantitative and qualitative studies.
Quantitative usability studies will come in handy if you want to streamline your tests. You recruit more respondents, run them through the application interface, give basic tasks, and at the end just ask how it is in general and what problems there were.
A qualitative usability test is much more complicated - you need to draw out the perception of the entire process and literally all the elements in the process using the method . All thoughts and questions that arise in people, all texts and elements that are incomprehensible to them. And all the root causes - why is it unclear, but how do you expect it to be named, and what word do you keep in your head?
Knowing the root causes of perception, you do not just say:
People did not find it - unusual placement.
And you understand how to change:
The user is looking for this element not at the bottom as we have placed it, but in the upper right corner of the screen. It searches for the word "Search", and we have "Enter", it looks for the magnifying glass icon, and we have the "Search" button.
To summarize, after a quantitative usability test, you will end up with a list of problems in its most general form. Let's say, "The user did not manage to find the Search." Why didn't he make it? But he simply did not master it - this test will not give an answer.
And after a qualitative test, you will have both a problem and its root cause. In the case of Search, you will have a script, the user will tell you exactly how the Search was searched, what elements he expected to see and where, what words came to his mind when he did not find the Search, and so on.
When you have the root cause of the problem and its detailed description, you can already fix something, change the interface so that it meets the expectations of users and solves their problems.
Of course, quality ones cost more. Instead of a task and a questionnaire, you need to train a person who will conduct such tests. Take a person with the right background, introduce him to the field that you are researching. This takes about 3-6 months. There are only a few ready-made specialists on the market - that is, there are practically none.
But even if all these tests are carried out normally, we will get the following situation - the country does not know what to do with these studies and reports. In the market, this is still treated as some kind of ephemeral entity, they believe that they are buying just a presentation, and not a solution to the problem.
Because it turns out: the bank ordered testing, received in response some kind of superficial presentation, which is not clear how to apply, or βwe knew all this ourselves.β What's next? And nothing, put it on the table and be glad that it is. Because people don't know what to do with this presentation, how to use it to improve the product, how to turn the conclusions described in it into new interfaces that will no longer be so problematic. If you do not give the depth and root causes of problems, then you will not understand how to work with problems.
Is everything really sad?
In general, it is rather sad, yes, but this does not mean that the situation cannot be corrected. Our goal was to research well those things in which we already had a good expertise. For example, about the work of payments in the application, we had certain statistics on it. We wanted to take the main scenarios and not just check them for "Yes - No", but to understand what kind of problems people have, at what stages, and in general - why they arise.
Distribution by main scenarios of legal entities
This may be a set of barriers that do not depend much on the bank itself, itβs just that the presentation of some function is made for people is not very clear.
And, of course, we wanted to do a voluminous study, and not compare a couple of banks with each other. We thought we could then sell these detailed studies, and at the same time test the overall demand for them.
Of course, our first pancake turned out with a couple of lumps.
We still tried to take all the scenarios and go through them with one respondent. Spoiler alert - he survived. Perhaps now he uses banking applications much less often. But we once again confirmed the thesis that in an hour and a half it is necessary to turn everything off and launch another one. Therefore, we switched from deep testing of all features to seeing how people find certain features, what they pay attention to, how they perceive the structure of the main page.
Distribution by use of platforms by individuals
When you're testing banking apps, you can't just run them in guest mode and draw conclusions. You must at least have a bank account to understand how everything works there. And in the case of a bank, entrepreneurs need a live account, with a history, with a company established there. If you are also testing currency control and other joys, you will need currency accounts and a little afobazole. The balance cannot be empty, the transaction history must be more serious than βI will transfer 200 r from my account to my account, let's see how it isβ.
We thought that it would be a fairly quick task to open accounts in all the banks studied and make money on them.
Sometimes everything dragged on for a couple of weeks. From the banks, yes. And we also tested 5 banks, but would there be 20 of them?
But we were able to understand for ourselves the distribution of the main functions and the number of some single and unpopular ones. Therefore, from the first pancake to the second run, we went with a more refined methodology. And a designer appeared in the team, which brought the presentations themselves to a new level. It's really more important than it seems when you give information like this.
The result of the work was presentations on 100+ slides. When we did a study on four banks for individuals, we did not sell it. But the first study, on banks for entrepreneurs, was sold to see how interesting the market is in principle. They bought it from us 7 times (top 5 banks and several companies that sold development and design to banks), we did not give any advertising, except for Facebook posts.
- But you yourself wrote that this is a sure way to go into the red!
Great way, yes, if you're only doing research. We earn first of all by design and design.
Research for us is an opportunity to shape the market, because, as you can see, it is almost non-existent. We were often asked, they say, guys, why did you place such a thing in free access, does it cost money? But thanks to this, we can show the community what research can be in principle. Now, just to see a sample of such studies, you need to buy them. Or ask the person who bought it.
We publish them just like that. So that the market also understands what research is. So that clients who order research somewhere else can at least compare and validate the quality of what other companies sell to them. In order for a common understanding to arise, research can be of high quality, and one can benefit from them and understand what to do with it further. We are actually a little jarred that the educational part in terms of research in our country is sad. Therefore, while we are trying to change the situation like this - by forming an understanding that you can get the best result
And besides the educational aspect, such studies and their publication are a good opportunity to generate leads. And here is a plus not only in the fact that customers come to us. Recently, according to one of our posts, they began to prototype a bank from the top 3. A few years ago, we would have really thought - damn it, licked our topic and went to do something of our own.
And now we think - cool, they listen to us, and really try to make products better and closer to the user. Therefore, we will continue to do such research, qualitatively testing individual semantic blocks of applications, and not just the entire product as a whole according to some kind of checklist.
Inside the team, this gives us an increase in expertise - not to go in the dark, but to understand how the main scenarios and needs of people are changing (and they change in 1-2 years, imagine). And then, when you study opening a bank account for entrepreneurs 3-4 times in 2 years, you have an ideal process as it can be in the current technical limitations.
And the situation like βI wanted to be in the rating - I paid for the rating - I got into the ratingβ still got fed up. And the need for a new rating based on the quality of the product is already overdue.
And for those who have read to the end of the article, here are two links to ΠΈ
Source: habr.com