In a previous article I looked at 7 ways to quickly test the competencies of IT professionals that can be applied before conducting a large, lengthy and time-consuming technical interview. There I looked at the essence of these methods and my practice of using them, as well as the reasons why I like them or dislike them.
In this article, I want to talk about the modern concept of human decision making, how it relates to job skills testing, and what competency testing methods such as interviews and test items actually test.
Some theory
Scientists for many centuries have been preoccupied with the question - how and why does a person make certain decisions? In each era, this question was answered differently - for millennia, faith in the fate and will of the gods dominated, then for a long time the opinion was popular that man is a rational being, which, basically, acts reasonably and prudently. The scientific revolution led to the fact that in the second half of the 20th century, a lot of research was carried out on the behavioral responses of "reasonable man". And at the moment, the most modern and recognized concept in scientific circles is the hybrid model of human behavior, about which the psychologist Daniel Kahneman wrote very well in his scientific articles and popular science books. Daniel received the Nobel Prize in Economics because his work disproved many economic theories based on models of rational human decision making. Daniel Kahneman convincingly showed that human behavior in most situations is due to automatic behavioral responses that are formed on the basis of life experience.
According to the concept of Daniel Kahneman, human behavior is controlled by two interacting decision-making systems. System 1 is fast and automatic, ensures the safety of the body and does not require significant efforts to form a solution. The accuracy of the decisions of this system depends on experience and training, and the speed depends on the characteristics of the nervous system of the individual. System 2 is slow, requires effort and concentration. It provides us with complex reasoning, logical conclusion, reasonable forecasting. The decision-making speed of this system is tens and hundreds of times lower than the speed of System 1. It is during the operation of System 2 that the full potential of human intelligence is manifested. However, during the operation of this system, resources are intensively consumed - both physical (energy) and attention, which is a derivative of many resources. Therefore, most of the decisions are made by System 1.
I think that each of you noticed that you cannot think hard and solve complex problems for more than a certain period of time in a row. This interval is different for every person. Someone can think hard for only half an hour a day, and someone can solve complex problems for 3 hours in a row. This ability can be developed, but it is given by very hard work and efforts on oneself, and all the same, the resource of attention will be limited.
Both systems work together. The information coming from the senses is first processed by the fast System 1, which recognizes dangerous situations and instantly reacts in case of threats. System 1 also recognizes unfamiliar situations and either chooses to ignore them or activates System 2.
Please multiply 65 by 15 and make a mental note of how long these calculations took you.
How it works? Have you ever watched how professional chess players play - how fast do they move at the beginning of the game? For a person who rarely plays chess, it seems impossible to make such difficult decisions so quickly. However, at the same time, you can fully correct the mistakes of the interns when you do the code review. Your System 1 can recognize the common mistakes of novice programmers and automatically correct them, just like a professional chess player reads the situation on the board and knows how to move, with little to no strain on the conscious System 2.
Please multiply 65 by 15 again and make a mental note of how long these calculations took you.
Numerous experiments have shown that in situations familiar to us, decisions are almost always made by automatic System 1, and this is quite rational from the point of view of the organism's survival and energy expenditure. In this regard, we act very rationally and optimally, but not in the sense of deliberation and optimality of the decisions themselves, but in the sense of a balance between the result and the cost of our body's resources. When you drive in the city on your way to work, your cornering and the amount of acceleration and deceleration may not be optimal, but from the point of view of the task of getting you from your home to your work, everything is very good. If you are a race car driver driving a racing car on a race track, your decisions about trajectory, acceleration and deceleration will be much more calculated.
In unfamiliar situations that are interesting to us or that we could not avoid, we are forced to act consciously, involving attention and System 2. After several repetitions of very similar situations, the result of the work of System 2 is stored in memory in the form of signs and reactions, and then you no longer have to expend energy and time for inference - System 1 will already be trained for this task and will provide the solution automatically next time. Some automatic responses are lost over time unless they are periodically called upon. Skills that we do not practice are lost.
Please multiply 65 by 15 again. Have you noticed any progress since your last attempt at this problem?
How does all this relate to work and competency testing?
Numerous experiments have shown that for the first time in a new place of work, an ordinary, mentally healthy person adapts and tries to accept the rules, conditions and work processes of the new place of work. However, after a while, each of us relaxes and begins to work as best he can. Effort and diligence give way to automatic responses and patterns rooted in System 1. Also, even during probation in stressful situations where a quick decision is required, we respond with automatic System 1 and not always in the way we were taught in this new place of work.
In general, we can say that our main value as an employee is largely determined by our experience - that is, the training of our System 1 in solving certain tasks that the employer needs. Therefore, employers very often want an employee not with an outstanding intellect, but with experience in some area. Experience is valued more than intelligence. This is explained by elementary calculations. If there is enough time, then any employee with sufficient intelligence will be able to understand the topic and solve the tasks. However, he will have to spend time on training and gaining experience, and only then will he be able to qualitatively solve the tasks. His System 2 will have to solve many training problems before his System 1 can quickly and accurately solve real problems. This takes time, which the employer is often not willing to pay at the high rate of a professional. Another employee who has already solved similar problems will do the job much faster, because most of the decisions will be given to him by his System 1, trained to solve problems in the right area. An experienced employee will issue high-quality solutions not only faster, but also less stressful at the same time. This means that the unused resources of attention can be directed to solving new complex problems and gaining new experience.
What to choose - experience or intelligence - the employer decides in each case individually. Where a quick reaction to a typical task and a quick solution are required, experience is often chosen. If you have to solve many different problems, but the solution time is still highly valued, then they choose an experienced and smart one. If time is not very critical, then you can give preference to an intellectual without experience. As you understand, in the real world, places of work where time is not critical are few.
Please multiply 65 by 15 again and make a mental note of how long these calculations took you. Did you notice how you got the result?
Ways to test competencies in terms of testing "System 1" and "System 2"
Experience - that is, the training of System 1 - is often an important, perhaps even defining criterion when an employer selects a new employee. How can we evaluate a candidate's experience most effectively and accurately? Let's take a look at popular ways of assessing competencies in terms of what they measure.
Interviews
This format involves a conversation between the candidate and the evaluator. Mostly, the assessor asks questions, but the candidate has the opportunity to read non-verbal signs, ask clarifying questions and, as they say, change his answer on the fly. This is the “oral exam” familiar to all of us. As a rule, the interview takes place according to a standard plan and many questions are also standard, which means you can prepare for them. That is, train your System 1 to successfully pass interviews.
The success of the candidate assessment depends on the communication skills of both participants. A candidate who is experienced enough in interviewing may well make a good impression. However, this result is obtained not due to work experience, but due to the experience of communication and interviews. A well-trained candidate, who answers well to standard questions, influences the examiner and he becomes more loyal to the candidate.
This method basically tests the candidate's System 1, though often not the experience that is needed in the job. It is good for evaluating specialists who will have to communicate a lot in their job duties and quickly adapt, but for evaluating technical skills, in my opinion, this method is not suitable. It is possible to improve the accuracy of the assessment through non-standard questions and interview scenarios, as well as through the participation of several assessors in the interview, which leads to an increase in the cost of this event.
Test items
The candidate receives a problem that he solves on his own and then demonstrates the result of the solution. In fact, this is our usual “written exam”. The candidate has enough time, the opportunity to ask clarifying questions, as well as search for information on the network and even take the help of friends. If the task is difficult and enough time is given, then this method tests System 2 rather than System 1, that is, intelligence, not experience. If you reduce the time to complete the task, then there is a high probability that candidates will refuse to complete a difficult test task. If at the same time we simplify the task, give several tasks and reduce the time, then this method becomes a completely working tool familiar to us from the school bench. It does a good job of testing System 1. However, its disadvantage is that checking the results requires significant effort on the side of the evaluators, since each solution can be unique and the verifiers must understand the essence of the solution.
Live Doing
The candidate receives a simple task, which he solves under the supervision of an evaluating specialist. This method is often used in the interview process - when evaluators first talk, and then offer to solve problems. For introvert candidates who have not been interviewed for a long time, this method is often psychologically uncomfortable, and they do not show very good results. In my opinion, this method should be offered to candidates as an alternative to a test task. That is, either 3-4 hours of independent work, or 1-1,5 hours of interviews and solving problems online. If the candidate is ready, this method allows you to test the basic skills of System 1 on typical tasks that are part of more complex work tasks. That is, as test tasks, it is worth choosing elements of real work tasks. You should not offer abstract tasks that your employee will never meet later in his work.
Multiple Choice Tests
As you probably know, the final exams in Russian schools now take the form of tests (GIA and USE). At the time, this caused heated discussions. Citizens generally negatively assessed this decision of the Ministry of Education. Personally, I think that if you do not take into account the new opportunities for corruption, then replacing written exams with tests is a good solution. Checking test results does not require much time and attention, it is easily automated. This minimizes the subjectivity of knowledge assessment. Tests allow for 1-2 hours to qualitatively test the knowledge and experience gained over several years of study or work. A novice driver learns the rules of the road for several months, and in the exam he must answer 20 questions within 20 minutes. The practice of decades of using this type of exam shows that this is enough if the test questions are written correctly, and there are a lot of them.
In the modern world, most human decisions come down to choosing one of the existing options that is most suitable for the situation. You hardly need a specialist who will reinvent the wheel. But on the other hand, a specialist who knows the advantages and disadvantages of different types of bicycles and similar modes of transport will come in handy for you, who will help you quickly choose the right model and customize it to solve your problems. Solving logistical problems usually has to be done quickly and there is no time to reinvent the wheel. Sometimes (very rarely) there are situations when you still need a new bike, which does not yet exist and needs to be created. However, in this case, a person who is well versed in the design of bicycles will be more useful than a universal inventor.
One more example. If a programmer can implement several sorting algorithms, then of course he is great, but in real life it will be more useful for him to know the basic methods of the basic language class library - there are probably already several sorting options implemented, just call the desired function.
Conclusion
It is important that when choosing how to test competencies, you turn on your System 2 and choose the appropriate method meaningfully, and not traditionally - "because we have always done it." When choosing a way to test competencies, I advise you first of all to decide what will be more important for you as an employer in the daily activities of your employee. Will it be the ability to quickly solve a certain range of typical tasks, or will it be necessary to solve complex, original, non-standard tasks.
In most cases, time-limited tests are a good first test for candidates. I recommend small tests that take no more than 15-20 minutes to complete. During this time, you can ask 30-40 questions and test the knowledge of candidates in sufficient detail. Then you can conduct an interview, during which you can analyze the mistakes made by the candidates. The test can also serve as a reference plan for the interview, during which you can ask the candidate the reasons why he answered the questions of the test the way he did and how he would have answered if the question had been asked differently.
If it is important for you how the future employee works independently on sufficiently large and isolated tasks, then it would be appropriate to start with an interview and then offer to complete a test task. It is worth remembering that only 20-25% of candidates agree to perform test tasks before the interview, and in this case, you greatly reduce the selection funnel.
In my next article, I will analyze in more detail the features of creating tests to test the competencies of candidates.
Source: habr.com
