Before enrolling in the new Machine Learning Advanced course, we assess prospective students to determine their readiness level and understand what they need to prepare for the course. However, a dilemma arises: on one hand, we must check knowledge of Data Science, and on the other hand, we cannot conduct a full four-hour exam.
To solve this issue, we established a TestDev team directly within the Data Science course development team (and it seems this is just the beginning). Here’s a list of 10 pitfalls that one encounters when developing tests for knowledge assessment. We hope that the world of online education will improve a bit because of this.
Pitfall 1: Failing to clearly define the testing objectives
To properly define objectives and create a test that considers them, during the planning phase, we must answer several questions:
- What exactly are we checking?
- In what environment will the testing take place, and what mechanics will be used? What limitations exist in this environment? This point will also help us understand the technical requirements for the device on which the testing will occur, as well as the content (if tests are being taken on phones, images must be readable even on small screens, and there should be a possibility to zoom in, etc.).
- How long will the testing last? We need to think about the conditions under which the user will take the test. Is there a chance that they will need to interrupt the testing process and then continue later?
- Will there be feedback? How is it formed and delivered? What is needed to receive it? Is there a time gap between completing the test and getting feedback?
In our case, by answering these questions, we determined the following list of objectives for the test:
- The test should show whether prospective students are ready for the course and if they have enough knowledge and skills.
- The test should provide us with material for feedback, indicating the topic where students made mistakes, so they can strengthen their knowledge. We will explain how to create it further.
Pitfall 2: Not creating specifications for the expert — the test compiler.
To create test assignments, it is crucial to involve an expert in the relevant field of knowledge being assessed. Additionally, the expert needs a well-crafted specification (description) that includes the test topics, the knowledge/skills being evaluated, and their level.
An expert will not create such a specification for themselves, as their job is to devise assignments, not the test structure. Moreover, few people are developing tests professionally, even in the teaching process. This is taught as a separate specialty—psychometrics.
If you want to quickly get acquainted with psychometrics, there is a for everyone interested. For a deeper study, there is a and a graduate school at the Institute of Education.
When preparing the specification, we gather a detailed description of the test for the expert (or better yet, together with them): the topics of the assignments, the type of assignments, and their quantity.
How to choose the type of assignments: once the topics are determined, we decide how best to assess them with assignments. Classic options include open-ended questions, multiple-choice or single-choice questions, matching, etc. (don't forget about the technical limitations of the environment in which the testing takes place!). After defining and specifying the types of assignments, we have a ready specification for the expert. We can call it the test specification.
Pitfall 3: Not Involving the Expert in Test Development
When involving the expert in the test development process, it is important not just to outline the 'scope of work' but to engage them in the actual development procedure.
How to make working with the expert as effective as possible:
- Pre-adjust them and spend some time discussing the science of test development and psychometrics.
- Focus the expert's attention on creating a valid and reliable assessment tool, rather than just a list of questions.
- Explain that their work includes a preparatory phase, not just the creation of the assignments themselves.
Some experts (due to their character) may perceive this as a check of their own work, and we clarify that even when creating excellent assignments, they may simply not meet specific testing objectives.
To ensure the process goes smoothly, we prepare with an expert a coverage table of topics (knowledge and skills), which is part of the test specification. This table allows for precise question development and defines what we will measure. In each specific case, it may be slightly different. Our goal is to assess how well a person understands the knowledge and skills from previous, foundational courses to determine their readiness for the new course.
Pitfall 4: Assuming that the expert "knows better"
Knows the subject better. But doesn't always explain clearly. It's crucial to check the wording of the tasks. Write clear instructions, such as "Select 1 correct option." In 90% of cases, experts prepare questions in a way that makes sense to them. And that's okay. However, before administering the test to those who will take it, everything needs to be checked and polished so that the test-takers clearly understand what is required of them and do not make mistakes simply due to misinterpreting the task instructions.
To avoid double interpretations of tasks, we conduct "cognitive labs." We ask people from the target audience to take the test while verbalizing their thoughts, and we document this in detail. In the "cognitive labs," we can identify unclear questions, poor wording, and get initial feedback on the test.
Pitfall 5: Not considering the time to complete the test
sarcasm mode: on
Of course, our test is the best, and everyone dreams of passing it! Yes, all 4 hours.
sarcasm mode: off
When there's a list of everything that can be tested, the key is not to test everything (sounds odd at first, doesn't it?). You need to ruthlessly cut down, highlighting with the expert the key knowledge and skills (yes, certain skills can also be tested). We look at the types of tasks and estimate the target time for completion: if it still exceeds reasonable limits — we cut!
To reduce the volume, you can also try (carefully) to test two skills with one task. In this case, it's difficult to understand why someone made a mistake, but with the correct completion, both skills can be accounted for. It's important to ensure that these two skills belong to the same area of knowledge.
Pitfall 6: Not planning the scoring system
Often when creating assessment tests, a classic point system is used, for example, 1 point for easy tasks and 2 points for difficult ones. However, this is not a universal approach. The mere sum of points after testing tells us little: we don't know which tasks earned these points and can only determine the number of correct answers. We need a clear understanding of what specific skills the test participants demonstrate. Furthermore, we want to provide them with feedback on which topics need further development.
After all, we are creating a test that will categorize individuals as ready or not ready to undertake the program; we will recommend some to prepare for the course through free training. It is important for us that only those who truly need it and are ready make it into this group.
What we do in our situation: we determine within the working group of test developers which groups of people need to be distinguished (for instance, those ready for training, partially ready) and create a characteristics table for these groups, specifying the skills and knowledge that will be relevant for those ready to learn. This helps us formulate the 'difficulty' of tasks for similar tests.
Pitfall 7: Evaluating results solely through automation
Certainly, the evaluation must be as objective as possible, so part of the students' materials is evaluated automatically, 'by keys'—comparing to correct answers. Even without a specialized testing system, there are plenty of free solutions available. And if there's an understanding of script writing principles, then with Google Forms and results in spreadsheets, anything can be done. If some tasks are judged by experts, we need to think about delivering responses to the experts without revealing the identity of the test takers. We should also consider how to integrate the experts' evaluation results into the final assessment.
Initially, we intended to create several open-ended coding tasks where experts evaluate solutions based on predetermined criteria. We even prepared a system that exports individual participants' responses to a special table for experts and then imports the results into a grading table. However, after discussions with target audience representatives, the product manager, and the instructional designer, we concluded that conducting a technical interview with instant expert feedback and code discussion, along with specific questions, would be much more effective and beneficial for the participants themselves.
Now the expert verifies the completion of the test by clarifying some questions. For this, we have prepared a guideline of questions and grading criteria for the technical interview. Before the technical interview, the expert receives a map of the participant's responses to choose which questions to ask.
Pitfall 8: Not explaining the test results
Providing feedback to participants is a separate issue. We need not only to inform them of their test score but also to help them understand the test results.
These may include:
- Tasks in which the participant made mistakes, contrasted with those they completed correctly.
- Topics where the participant made errors.
- His ranking among those taking the exam.
- Description of the participant's level in accordance with, for example, the level descriptions based on job vacancy descriptions.
During the pilot launch of our test, we showed those who wanted to enroll in the program, along with their results, a list of topics that needed improvement. However, this is certainly not ideal; we will continue to improve and enhance the feedback.
Pitfall 9: Not discussing the test with developers
Perhaps the sharpest pitfalls, stepping on which is particularly unpleasant, is sending the test, description, and scoring scale to developers in a 'as is' state.
What exactly needs discussion:
- The appearance of questions, structure, placement of graphics, how the choice of the correct answer looks.
- How the score is calculated (if needed), whether there are additional conditions.
- How feedback is generated, where to obtain texts from, if there are any automatically generated sections.
- What additional information you need to gather and at what point (like contact details).
To avoid misunderstandings, we ask our developers to encode 2 or 3 different questions to see how they look before programming the test itself.
Trap 10: Deploying to production without testing
Guys, every test must be checked 3 times by different people, or better — each one 3 times. This truth has been proven through blood, sweat, and lines of code.
Our test checks the following trio:
- Product Owner — checks the test for functionality, appearance, and mechanics.
- Test Developer — checks the wording of tasks, their order, the way to work with the test, types of tasks, correct answers, readability, and normal graphics display.
- Task Author (Expert) — checks the test for correctness from an expert standpoint.
Example from practice: only on the third run did the task author notice that one task remained in the old wording. All previous ones were actively edited as well. But when the test was coded, it looked different than originally imagined. It is highly likely that some adjustments will be needed. This must be taken into account.
Summary
Carefully navigating all these 'traps', we created a special , for checking the knowledge of applicants. Anyone interested can test it while we prepare the next material, in which we will explain what happened inside the bot and how it all transformed later.
Acquire a sought-after profession from scratch or Level Up your skills and salary by taking online courses from SkillFactory:
- (12 weeks)
- (20 weeks)
- (20 weeks)
- (12 months)
More Courses
- (12 months)
- (14 weeks)
- (5 weeks)
- (18 months)
- (6 months)
- (12 months)
- (8 months)
- (9 months)
- (12 months)
- (18 months)
- (18 months)
- (12 months)
- (9 months)
- (7 months)
Source: habr.com
