In the previous article I explored 7 methods for quickly assessing the competencies of IT specialists that can be applied before conducting a large, comprehensive, and labor-intensive technical interview. I discussed the essence of these methods and my practice in applying them, as well as the reasons why I like or dislike them.
In this article, I want to discuss the modern concept of human decision-making, how it relates to verifying job skills, and what competency assessment methods such as interviews and tests actually evaluate.
A Bit of Theory
For centuries, scientists have pondered the question – how and why does a person make certain decisions? Throughout different ages, various answers have prevailed; for millennia, belief in fate and the will of the gods dominated, and for a long time, the view that humans are rational beings who act primarily sensibly and calculatedly was popular. The scientific revolution led to numerous studies on the behavioral reactions of 'homo sapiens' in the second half of the 20th century. Currently, the most modern and recognized concept in the scientific community is the hybrid model of human behavior, which psychologist Daniel Kahneman elaborated on in his scholarly articles and popular science books. Daniel was awarded the Nobel Prize in Economics for his work, which disproved many economic theories based on models of rational decision-making. He convincingly demonstrated that human behavior in most situations is influenced by automatic behavioral reactions shaped by life experience.
According to Daniel Kahneman's concept, human behavior is governed by two interacting decision-making systems. System 1 is fast and automatic, ensuring the organism's safety without requiring significant effort to form a decision. The accuracy of decisions made by this system depends on experience and training, while its speed is influenced by the individual's nervous system characteristics. System 2 is slower, requiring effort and focused attention. It provides us with complex reasoning, logical deduction, and informed forecasting. The decision-making speed of this system is tens to hundreds of times slower than System 1. It is during the operation of System 2 that the full potential of human intelligence is revealed. However, this system heavily consumes resources—both physical (energy) and attention, which is derived from numerous resources. Therefore, the majority of decisions are made by System 1.
I believe that each of you has noticed that you cannot think intensively and solve complex problems for more than a certain period of time in a row. This interval varies from person to person. Some can think intensely for only half an hour a day, while others can tackle complex problems for 3 consecutive hours. This ability can be developed, but it requires considerable hard work and effort, and still, the resource of attention will be limited.
Both systems work together. The information received from the sensory organs is initially processed by the fast System 1, which recognizes dangerous situations and reacts instantly in case of a threat. System 1 also identifies unfamiliar situations and either decides to ignore them or activates System 2.
Please multiply 65 by 15 and note to yourself how much time it took you to perform this calculation.
How does it work? Have you ever watched professional chess players and noticed how quickly they make their moves at the beginning of a game? For someone who rarely plays chess, it seems impossible to make such complex decisions so quickly. However, you can easily correct the mistakes of beginners when you conduct a code review. Your System 1 can recognize common errors made by novice programmers and automatically fix them, just like a professional chess player reads the board and knows their next move without straining their conscious System 2.
Please multiply 65 by 15 again and note how much time it took you to perform these calculations.
Numerous experiments have shown that in familiar situations, decisions are almost always made by the automatic System 1, which makes perfect sense from the perspective of individual survival and energy expenditure. In this regard, we act quite rationally and optimally, not in terms of the thought processes or the quality of the decisions made, but in the balance between results and the energy resources of our bodies. When you drive in the city on your way to work, your route and the number of accelerations and stops may not be optimal, but in terms of getting you from your home to your workplace, it works quite well. However, if you are a race car driver on a racetrack, your decisions about the trajectory, accelerations, and braking will be much more calculated.
In unfamiliar situations that interest us or cannot be avoided, we must act consciously, engaging our attention and System 2. After several repetitions of very similar situations, the results produced by System 2 are stored in memory in the form of cues and responses, and you won't need to expend energy and time on logical deductions—System 1 will already be trained for this task and will provide a solution automatically next time. Some automatic reactions may fade over time if they are not frequently used. Skills that we do not practice will diminish.
Please multiply 65 by 15 again. Did you notice any progress compared to your previous attempt to solve this problem?
How is all this related to work activities and competency verification?
Numerous experiments have shown that at first, a mentally healthy person adapts to a new job and tries to accept the rules, conditions, and work processes of the new workplace. However, after some time, each of us relaxes and starts working as we know how. Effort and diligence give way to automatic reactions and patterns established in System 1. Additionally, even during the probation period in stressful situations where quick decisions are required, we react using the automatic System 1 and not always in accordance with what we were taught at this new place of work.
In general, it can be said that our primary value as an employee is largely determined by our experience — that is, the training of our System 1 in solving specific tasks needed by the employer. Therefore, employers often seek employees not for outstanding intelligence but for experience in a certain field. Experience is valued more than intelligence. This is explained by basic calculations. Given enough time, any employee with sufficient intelligence can understand the subject matter and solve the assigned tasks. However, they will need to invest time in learning and gaining experience, and only then will they be able to effectively solve the assigned tasks. Their System 2 will need to solve many training tasks before their System 1 can quickly and effectively tackle real tasks. This takes time, which employers are often not willing to pay at a high rate for a professional. Another employee, who has already handled similar tasks, will complete the work much faster because most of the solutions will come from their System 1, trained to solve tasks in the required field. An experienced employee will deliver quality solutions not only faster but also with less effort. This means that their unused attention resources can be directed toward solving new complex tasks and gaining new experiences.
When choosing between experience and intelligence, the employer makes an individual decision in each case. Where a quick reaction to a typical task and a rapid solution are required, experience is often preferred. However, when faced with a variety of tasks, but time is still highly valued, both experienced and intelligent candidates are chosen. If time is not critical, preference can be given to an intellectual without experience. As you can understand, in the real world, there are few jobs where time is not critical.
Please multiply 65 by 15 again and note how much time these calculations took you. Did you notice how you got the result?
Ways to assess competencies from the perspective of ‘System 1’ and ‘System 2’ testing
Experience—meaning the training of System 1—often serves as an important, perhaps even the determining criterion for an employer when selecting a new employee. How can we most effectively and accurately assess a candidate's experience? Let's look at popular methods of evaluating competencies with respect to what they assess.
Interviews
This format involves a conversation between the candidate and the evaluator. Primarily, the evaluator asks questions, but the candidate has the opportunity to read non-verbal cues, ask clarifying questions, and as they say, adjust their answers 'on the fly'. This is the familiar 'oral exam' we all know. Typically, the interview follows a standard plan, and many questions are also standard, meaning that one can prepare for them. Essentially, this means training System 1 to successfully pass interviews.
The success of candidate assessment depends on the communication skills of both participants. A candidate who is experienced in interviews can leave a good impression. However, this result is achieved not through work experience but through communication experience and interview practice. A prepared candidate who answers standard questions well influences the evaluator, making them more favorable towards the candidate.
This method mainly tests Candidate System 1, although it often doesn't assess the experience needed for the job. It is quite suitable for evaluating specialists who will need to communicate frequently and adapt quickly, but in my opinion, it is not effective for assessing technical skills. The accuracy of the evaluation can be improved by using unconventional questions and interview scenarios, as well as involving multiple evaluators in the interview, which increases the cost of the process.
Test Tasks
The candidate receives a task that they solve independently and then demonstrate the solution. Essentially, this is our familiar "written exam." The candidate has ample time, the opportunity to ask clarifying questions, and can search for information online or even seek help from friends. If the task is complex and sufficient time is given, this method tests more of System 2 than System 1, meaning it assesses intelligence rather than experience. However, if the time to complete the task is shortened, there's a high likelihood that candidates will opt not to complete a challenging test task. Conversely, if the task is simplified, multiple tasks are given, and time is limited, this method becomes a useful tool familiar to us since school. It tests System 1 well. However, its drawback is that evaluating results requires significant effort from the evaluators, as each solution may be unique, necessitating a deep understanding of the solution's essence.
Live Doing
The candidate is given a simple task to solve under the supervision of an evaluating specialist. This method is often used in the interview process—where evaluators first engage in a conversation and then offer problem-solving tasks. For introverted candidates, especially those who haven't been to interviews in a while, this approach can often be psychologically uncomfortable, resulting in less than favorable outcomes. In my opinion, this method should be offered to candidates as an alternative to a test assignment. That is, either 3-4 hours of independent work or 1-1.5 hours of interviewing and solving tasks online. If the candidate is ready, this method allows for testing basic System 1 skills on typical tasks that are components of more complex job tasks. Therefore, the test tasks should include elements of real work tasks. Abstract problems that the employee will never encounter in their work should not be offered.
Multiple-choice tests
As you probably know, final exams in Russian schools now take the form of tests (GIA and USE). This decision sparked heated discussions at the time. Citizens generally assessed this decision by the Ministry of Education negatively. Personally, I believe that, aside from the new opportunities for corruption, replacing written exams with tests is a good solution. Evaluating test results does not require much time and attention, and it is easily automatable. At the same time, it minimizes the subjectivity of knowledge assessments. Tests allow for a thorough evaluation of knowledge and experience gained over several years of study or work within 1-2 hours. A novice driver learns the rules of the road for several months, and during the exam, they must answer 20 questions in 20 minutes. Decades of applying this type of exam show that this is sufficient, provided the test questions are well-crafted and sufficiently numerous.
In today's world, most human decisions come down to choosing one of the existing options that best fits the situation. You are unlikely to need a specialist who will reinvent the wheel. However, you will benefit from a specialist who knows the strengths and weaknesses of different types of bicycles and similar modes of transport, helping you quickly select the right model and tailor it to your needs. Logistics problems often need to be solved quickly, and there is no time to invent an innovative bicycle. Sometimes (very rarely), there are situations where you actually need a new bicycle that doesn't yet exist and must be created. However, even in this case, a person who has a deep understanding of bicycle designs will be more useful than a universal inventor.
Another example. If a programmer can implement several sorting algorithms, that’s certainly commendable, but in real life, it is more beneficial for them to know the basic methods of the language's standard class library—there are surely already several sorting options implemented there, and they just need to call the appropriate function.
Conclusion
It is important that when choosing a method for assessing competencies, you engage your System 2 and select an appropriate method thoughtfully, rather than by tradition – 'because we've always done it this way.' In selecting a method for assessing competencies, I recommend first determining what will be more important for you as an employer in your employee's daily activities. Will it be the ability to quickly solve a range of standard tasks, or will they need to address complex, original, non-standard problems?
In most cases, timed tests are an excellent first assessment for candidates. I recommend short tests that take no more than 15-20 minutes to complete. In that time, you can ask 30-40 questions and adequately evaluate the candidates' knowledge. Then, a follow-up interview can address the mistakes made by candidates. The test can also serve as a framework for the interview, where you can clarify with the candidate the reasons for their answers and how they might respond if the questions were posed differently.
If it is important to you how a future employee works independently on relatively large and isolated tasks, it would be appropriate to start with an interview and then suggest a practical assignment. Keep in mind that only 20-25% of candidates agree to complete practical tasks before an interview, which significantly narrows your selection process.
In my next article, I will elaborate on the nuances of creating tests to assess candidates' competencies.
Source: habr.com
