How we assessed the quality of documentation

Hey Habr! My name is Lesha, I am a system analyst for one of Alfa-Bank's product teams. Now I am developing a new Internet bank for legal entities and individual entrepreneurs.

And when you are an analyst, especially in a channel like this, without documentation and close work with it, you can’t go anywhere. And documentation is the thing that always raises a lot of questions. Why is the web application not described? Why does the specification say how the service should work, but it doesn't work at all? Why is it that only two people are able to understand the specification, one of whom wrote it?

At the same time, the documentation cannot be ignored for obvious reasons. And to make our life easier, we decided to evaluate the quality of the documentation. How exactly we did it and what conclusions we came to - under the cut.

Documentation quality

In order not to repeat “New Internet Bank” several dozen times in the text, I will write NIB. Now we have more than a dozen teams working on the development of the NIB for entrepreneurs and legal entities. At the same time, each of them either creates its own documentation for a new service or web application from scratch, or makes changes to the current one. Can the documentation in principle be of high quality with this approach?

And so, to determine the quality of documentation, we identified three main characteristics.

1. It must be complete. Sounds pretty captain-like, but it's important to note. It should describe in detail all the elements of the implemented solution.
2. It must be up to date. That is, to correspond to the current implementation of the solution itself.
3. It must be understandable. So that the person using it understands exactly how the solution is implemented.

In summary - complete, up-to-date and understandable documentation.

Interview

To assess the quality of the documentation, we decided to interview those who directly work with it: NIB analysts. Respondents were asked to evaluate 10 statements according to the scheme “On a scale from 1 to 5 (strongly disagree-strongly agree)”.

The statements, however, reflected the characteristics of quality documentation and the opinion of the survey writers regarding the documents of the NIB.

1. The documentation on NIB applications is up-to-date and fully corresponds to their implementation.
2. The implementation of NIB applications is fully documented.
3. Documentation on NIB applications is needed only for functional support.
4. Documentation on NIB applications is relevant at the time of their submission for functional support.
5. NIB application developers use the documentation to understand what they need to implement.
6. The NIB application documentation is sufficient to understand how they are implemented.
7. I update the documentation on NIB projects in a timely manner if they are finalized (by my team).
8. Developers of NIB applications conduct documentation reviews.
9. I have a clear understanding of how to draw up documentation for NIB projects.
10. I understand when to write/update NIB project documentation.

It is clear that simply answering “From 1 to 5” might not reveal the necessary details, so a person could leave a comment on each item.

We did all this through corporate Slack - we simply sent out an invitation to system analysts to take a survey. There were 15 analysts (9 from Moscow and 6 from St. Petersburg). After the survey was completed, we formed an average score for each of the 10 statements, which we then normalized.

It turned out that's what.

The survey showed that although analysts are inclined to believe that the implementation of NIB applications is fully documented, they do not give unambiguous consent (0.2). As a specific example, they pointed out that a number of databases and queues from existing solutions were not covered by the documentation. The developer is able to tell the analyst that not everything is documented. But the thesis that developers review documentation also did not receive unambiguous support (0.33). That is, the risk of incomplete description of the implemented solutions remains.

Relevance is simpler - although there is no unequivocal agreement again (0,13), analysts still tend to consider the documentation relevant. The comments allowed us to understand that more often there are problems with relevance at the front than in the middle. True, nothing was written about the back to us.

As for whether the analysts themselves understand when to write and update the documentation, the agreement was already much more uniform (1,33), including its design (1.07). What was noted here as an inconvenience was the lack of uniform rules for maintaining documentation. Therefore, in order not to turn on the “Who is in the woods, who is for firewood” mode, they have to work on the basis of examples of already existing documentation. From here, a useful Wishlist is to create a standard for maintaining documents, to develop templates for their parts.

Documentation on NIB applications is up-to-date at the time of submission for functional support (0.73). This is understandable, because one of the criteria for submitting a project for functional support is up-to-date documentation. And it is also sufficient to understand the implementation (0.67), although sometimes there are questions.

But what the respondents did not agree with (rather unanimously) was the fact that, in principle, only functional support needs documentation on NIB applications (-1.53). Analysts were mentioned most often as consumers of documentation. The rest of the team (developers) - much less often. Moreover, analysts believe that developers do not use the documentation to understand what they need to implement, albeit not unanimously (-0.06). This, by the way, is also expected in conditions where code development and documentation writing go hand in hand.

What is the result and why do we need these numbers

To improve the quality of documents, we decided to do the following:

1. Ask the developer to review written documents.
2. If possible, update the documentation in a timely manner, the front - in the first place.
3. Create and adopt a standard for documenting NIB projects so that everyone can quickly understand which elements of the systems and how exactly should be described. Well, develop appropriate templates.

All this should help to raise the quality of documents to a new level.

At least I hope so.

Source: habr.com