{"id":34958,"date":"2019-10-31T22:01:31","date_gmt":"2019-10-31T19:01:31","guid":{"rendered":"https:\/\/prohoster.info\/blog\/kogda-stoit-proveryat-gipotezu-o-ne-menshej-effektivnosti\/"},"modified":"2019-10-31T22:01:31","modified_gmt":"2019-10-31T19:01:31","slug":"kogda-stoit-proveryat-gipotezu-o-ne-menshej-effektivnosti","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/news\/kogda-stoit-proveryat-gipotezu-o-ne-menshej-effektivnosti","title":{"rendered":"When should you test the hypothesis of equal effectiveness?","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

\"When
\nAn article from the Stitch Fix team suggests using a clinical trial approach of no less effectiveness (non-inferiority trials) in marketing and product A\/B tests. This approach is indeed applicable when we are testing a new solution that has advantages not measurable by tests.<\/p>\n

The simplest example is cost reduction. For instance, we automate the process of scheduling the first lesson, but we don't want to significantly drop the overall conversion rate. Or we test changes aimed at a specific user segment while ensuring that conversions for other segments do not decline drastically (when testing multiple hypotheses, remember about adjustments).<\/p>\n

Choosing the right non-inferiority margin adds additional challenges in the test design phase. The question of how to select \u0394 is not very well covered in the article. It seems that this choice is not entirely transparent, even in clinical trials. Overview<\/a><\/noindex> Medical publications on non-inferiority indicate that only half of the publications justify the choice of margin, and often these justifications are ambiguous or not detailed.<\/p>\n

In any case, this approach seems interesting since it may increase the speed of testing by reducing the required sample size, and thus the speed of decision-making. Darya Mukhina, product analyst for the Skyeng mobile application.<\/i>
\n<\/a><\/noindex>
\nThe Stitch Fix team loves to test different things. The entire tech community generally enjoys conducting tests. Which version of the website attracts more users \u2013 A or B? Does version A of the recommendation model generate more revenue than version B? Almost always, to validate hypotheses, we use the simplest approach from basic statistics: <\/p>\n

\"When
\n
\nAlthough we rarely use this term, such a form of testing is called 'testing the superiority hypothesis.' In this approach, we assume that there is no difference between the two options. We hold onto this idea and only abandon it if the obtained data is convincing enough to do so \u2013 that is, demonstrating that one option (A or B) is better than the other.<\/p>\n

Testing the hypothesis of superiority is suitable for solving various problems. We release the B-version of the recommendation model only if it is clearly better than the currently used version A. However, in some cases, this approach does not work as well. Let's consider a few examples.<\/p>\n

1) We use a third-party service<\/b>, which helps identify counterfeit credit cards. We found another service that costs significantly less. If the cheaper service performs as well as the one we are currently using, we will choose it. It does not necessarily have to be better than the service we use. <\/p>\n

2) We want to abandon data source<\/b> A and replace it with data source B. We could postpone the abandonment of A if B produces very poor results, but continuing to use A is not feasible. <\/p>\n

3) We would like to shift from modeling approach<\/b>A to approach B not because we expect better results from B, but because it gives us greater operational flexibility. We have no reason to believe that B will perform worse, but we will not make the transition if it does.<\/p>\n

4) We have made several qualitative changes<\/b> to the website design (version B) and believe that this version surpasses version A. We do not expect changes in conversion rates or any key performance indicators that we usually evaluate the website on. However, we believe there are advantages in metrics that are either unmeasurable or for which our technologies are insufficient to measure.<\/p>\n

In all these cases, exploring superiority is not the most suitable solution. But most specialists in such situations use it by default. We conduct the experiment carefully to accurately determine the magnitude of the effect. If it were true that versions A and B function very similarly, there is a chance we would not be able to reject the null hypothesis. Do we conclude that A and B generally perform the same? No! The inability to reject the null hypothesis and accepting the null hypothesis are not the same.<\/p>\n

Sample size calculations (which you surely conducted) are usually performed with stricter bounds for Type I error (the probability of incorrectly rejecting the null hypothesis, often referred to as alpha) than for Type II error (the probability of failing to reject the null hypothesis when it is false, often referred to as beta). A typical value for alpha is 0.05, while a common value for beta is 0.20, corresponding to a statistical power of 0.80. This means we may fail to detect the true effect size specified in our power calculations with a probability of 20%, which is quite a significant information gap. For example, let\u2019s consider the following hypotheses:<\/p>\n

\"When
\n
\nH0: my backpack is NOT in my room (3)
\nH1: my backpack is in my room (4)<\/i><\/p>\n

If I searched my room and found my backpack \u2014 great, I can reject the null hypothesis. But if I looked through the room and couldn\u2019t find my backpack (Figure 1), what conclusion should I draw? Am I sure it\u2019s not there? Did I search thoroughly enough? What if I only searched 80% of the room? Concluding that the backpack is definitely not in the room would be a rash decision. It\u2019s no surprise that we cannot \"accept the null hypothesis.\"
\n\"When
\nThe area we searched
\nWe did not find the backpack \u2014 should we accept the null hypothesis?<\/i><\/p>\n

Figure 1. Searching 80% of the room is approximately equivalent to conducting a study with 80% power. If you did not find the backpack after searching 80% of the room, can you conclude that it is not there?<\/i> <\/p>\n

So what should a data specialist do in this situation? You can significantly increase the power of the study, but then you will need a much larger sample size, and the result may still be unsatisfactory.<\/p>\n

Fortunately, such issues have been studied extensively in the world of clinical research. Drug B is cheaper than Drug A; Drug B is expected to cause fewer side effects than Drug A; Drug B is easier to transport because it does not need to be stored in a refrigerator, while Drug A does. Let's test the hypothesis of non-inferiority. This is necessary to demonstrate that version B is as good as version A\u2014at least within some predefined non-inferiority margin, \u0394. We will discuss in more detail later how to establish this margin. But for now, let's assume that this is the minimal difference that is practically significant (in the context of clinical trials, this is usually referred to as clinical significance).<\/p>\n

Non-inferiority hypotheses turn everything upside down:<\/p>\n

\"When
\n
\nNow instead of assuming that there is no difference, we assume that version B is worse than version A, and we will stick to this assumption until we demonstrate otherwise. This is exactly when it makes sense to use a one-tailed hypothesis test! In practice, this can be done by constructing a confidence interval and determining whether the interval is indeed greater than \u0394 (see Figure 2).
\n\"When
\n<\/p>\n

Choosing \u0394<\/h3>\n

\nHow should we choose \u0394? The process of choosing \u0394 involves statistical justification and subject matter assessment. In the world of clinical research, there are regulatory guidelines indicating that delta should represent the smallest clinically significant difference\u2014one that will matter in practice. Here is a quote from the European guidelines to help check yourself: 'If the difference has been chosen correctly, a confidence interval that is entirely between \u2013\u2206 and 0... is still sufficient to demonstrate non-inferiority. If this result does not seem acceptable, it means that \u2206 was not chosen appropriately.'<\/p>\n

The delta should definitely not exceed the effect size of version A relative to the true control (placebo \/ no treatment), as this would imply that version B is worse than the true control while simultaneously demonstrating 'no less efficacy'. Assume that when version A was presented, there was a version 0 or the function did not exist at all (see Figure 3). <\/p>\n

According to the results of the superiority hypothesis test, an effect size E was identified (i.e., presumably \u03bc^A\u2212\u03bc^0=E). Now A is our new standard, and we want to ensure that B is not inferior to A. Another way to write \u03bcB\u2212\u03bcA\u2264\u2212\u0394 (the null hypothesis) is \u03bcB\u2264\u03bcA\u2212\u0394. If we assume that it is equal to or greater than E, then \u03bcB \u2264 \u03bcA\u2212E \u2264 placebo. Now we see that our estimate for \u03bcB completely exceeds \u03bcA\u2212E, thus completely refuting the null hypothesis and allowing us to conclude that B is not inferior to A, yet at the same time \u03bcB may be \u2264 \u03bc placebo, which is not what we want. (Figure 3).<\/p>\n

\"When
\nFigure 3. Demonstration of the risks of choosing a non-inferiority margin. If the margin is too large, one might conclude that B is not inferior to A, yet is indistinguishable from placebo. We would not want to switch to a treatment that is clearly less effective than placebo (A) for one that has the same efficacy as placebo. <\/i><\/p>\n

Choosing \u03b1 <\/h3>\n

\nLet\u2019s move on to choosing \u03b1. The standard value of \u03b1 = 0.05 can be used, but this is not entirely fair. For instance, when you purchase something online and use multiple discount codes simultaneously, which should not stack\u2014it's just that the developer made a mistake, and you got away with it. According to the rules, the value of \u03b1 should be half of the \u03b1 used in the superiority hypothesis test, i.e., 0.05 \/ 2 = 0.025.<\/p>\n

Sample Size <\/h3>\n

\nHow do you estimate the sample size? If you believe that the true difference in means between A and B is 0, then the sample size calculation would be the same as for testing the superiority hypothesis, except that you replace the effect size with the non-inferiority margin, provided that you use \u03b1non-inferiority = 1\/2 \u03b1superiority <\/b>(\u03b1non-inferiority=1\/2\u03b1superiority). If you have reason to believe that variant B might be slightly worse than variant A, but you want to prove that it is not worse by more than \u0394, then you are in luck! In fact, this reduces your sample size because it is easier to demonstrate that B is worse than A if you actually believe it is slightly worse and not equivalent.<\/p>\n

Example with a decision <\/h3>\n

\nSuppose you want to switch to version B on the condition that it is not worse than version A by more than 0.1 points on a 5-point customer satisfaction scale... Let's approach this task using the superiority hypothesis.<\/p>\n

To test the superiority hypothesis, we would calculate the sample size as follows:<\/p>\n

\"When
\n
\nThat is, if you have 2103 observations in your group, you can be 90% confident that you will detect an effect size of 0.10 or greater. But if the value of 0.10 is too large for you, it may not be worth testing the superiority hypothesis for it. Perhaps for reliability, you decide to conduct a study for a smaller effect size, for example, 0.05. In this case, you would need 8407 observations, which means the sample size increases nearly 4 times. But what if we stick to our original sample size but increase the power to 0.99, so we are not unsure if we get a positive result? In that case, n for one group would be 3676, which is already better, but it increases the sample size by more than 50%. And as a result, we still just won't be able to refute the null hypothesis, not to get an answer to our question.<\/p>\n

What if instead we test the hypothesis of non-inferiority? <\/p>\n

\"When
\n
\nThe sample size will be calculated using the same formula except for the denominator.
\nThe differences from the formula used for testing the superiority hypothesis are as follows:<\/p>\n

\u2014 Z1\u2212\u03b1\/2 is replaced with Z1\u2212\u03b1, but if you do everything by the rules, you replace \u03b1 = 0.05 with \u03b1 = 0.025, meaning that it's the same number (1.96)<\/p>\n

\u2014 (\u03bcB\u2212\u03bcA) appears in the denominator<\/p>\n

\u2014 \u03b8 (effect size) is replaced with \u0394 (non-inferiority margin)<\/p>\n

If we assume that \u00b5B = \u00b5A, then (\u00b5B \u2212 \u00b5A) = 0 and the calculation of sample size for the non-inferiority limit is exactly what we would obtain when calculating superiority for an effect size of 0.1, great! We can conduct a study of the same scale with different hypotheses and a different approach to conclusions, and we will get the answer to the question we really want to answer.<\/p>\n

Now let\u2019s assume that we actually do not believe that \u00b5B = \u00b5A and
\nwe think that \u00b5B is slightly worse, perhaps by 0.01 units. This increases our denominator, reducing the sample size for the group to 1737.<\/p>\n

What happens if version B is actually better than version A? We reject the null hypothesis that B is worse than A by more than \u0394 and accept the alternative hypothesis that B, if worse, is not worse than \u0394, and may be better. Try to incorporate this conclusion into a cross-functional presentation and see what comes of it (seriously, give it a try). In situations where we need to orient to perspective, no one wants to agree to 'not worse than \u0394 and possibly better.'<\/p>\n

In this case, we can conduct a study that is very briefly called 'hypothesis testing of whether one option outperforms or underperforms another.' It uses two sets of hypotheses:<\/p>\n

The first set (the same as in testing for non-inferiority): <\/p>\n

\"When
\n
\nThe second set (the same as in testing for superiority):<\/p>\n

\"When
\n
\nWe only test the second hypothesis if the first is rejected. Through sequential testing, we maintain the overall type I error rate (\u03b1). In practice, this can be achieved by constructing a 95% confidence interval for the difference between the means and testing to see if the entire interval exceeds -\u0394. If the interval does not exceed -\u0394, we cannot reject the null hypothesis and stop. If the entire interval does exceed -\u0394, we will continue and check if the interval contains 0. <\/p>\n

There is another type of research that we have not discussed \u2013 equivalence studies. <\/p>\n

This type of research can be replaced by studies aimed at testing the hypothesis of non-inferiority and vice versa, but they have a significant difference. The hypothesis test for non-inferiority aims to demonstrate that option B is at least as good as A. In contrast, an equivalence study aims to show that option B is at least as good as A, and option A is at least as good as B, which is more complex. Essentially, we are trying to determine whether the entire confidence interval for the difference in means lies between \u2212\u0394 and \u0394. Such studies require a larger sample size and are conducted less frequently. Therefore, the next time you conduct a study where your main goal is to ensure that the new version is not worse, do not settle for \u201cfailing to reject the null hypothesis.\u201d If you want to test a truly important hypothesis, consider various options.
\n
Source: habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"

\u0421\u0442\u0430\u0442\u044c\u044f \u043e\u0442 \u043a\u043e\u043c\u0430\u043d\u0434\u044b Stitch Fix \u043f\u0440\u0435\u0434\u043b\u0430\u0433\u0430\u0435\u0442 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u044c \u043f\u043e\u0434\u0445\u043e\u0434 \u043a\u043b\u0438\u043d\u0438\u0447\u0435\u0441\u043a\u0438\u0445 \u0438\u0441\u0441\u043b\u0435\u0434\u043e\u0432\u0430\u043d\u0438\u0439 \u043d\u0435 \u043c\u0435\u043d\u044c\u0448\u0435\u0439 \u044d\u0444\u0444\u0435\u043a\u0442\u0438\u0432\u043d\u043e\u0441\u0442\u0438 (non-inferiority trials) \u0432 \u043c\u0430\u0440\u043a\u0435\u0442\u0438\u043d\u0433\u043e\u0432\u044b\u0445 \u0438 \u043f\u0440\u043e\u0434\u0443\u043a\u0442\u043e\u0432\u044b\u0445 A\/B \u0442\u0435\u0441\u0442\u0430\u0445. \u0422\u0430\u043a\u043e\u0439 \u043f\u043e\u0434\u0445\u043e\u0434 \u0434\u0435\u0439\u0441\u0442\u0432\u0438\u0442\u0435\u043b\u044c\u043d\u043e \u043f\u0440\u0438\u043c\u0435\u043d\u0438\u043c, \u043a\u043e\u0433\u0434\u0430 \u043c\u044b \u0442\u0435\u0441\u0442\u0438\u0440\u0443\u0435\u043c \u043d\u043e\u0432\u043e\u0435 \u0440\u0435\u0448\u0435\u043d\u0438\u0435, \u0438\u043c\u0435\u044e\u0449\u0435\u0435 \u043f\u0440\u0435\u0438\u043c\u0443\u0449\u0435\u0441\u0442\u0432\u0430, \u043d\u0435\u0438\u0437\u043c\u0435\u0440\u044f\u0435\u043c\u044b\u0435 \u0442\u0435\u0441\u0442\u0430\u043c\u0438. \u0421\u0430\u043c\u044b\u0439 \u043f\u0440\u043e\u0441\u0442\u043e\u0439 \u043f\u0440\u0438\u043c\u0435\u0440 \u2014 \u0441\u043d\u0438\u0436\u0435\u043d\u0438\u0435 \u043a\u043e\u0441\u0442\u043e\u0432. \u041d\u0430\u043f\u0440\u0438\u043c\u0435\u0440, \u0430\u0432\u0442\u043e\u043c\u0430\u0442\u0438\u0437\u0438\u0440\u0443\u0435\u043c \u043f\u0440\u043e\u0446\u0435\u0441\u0441 \u043d\u0430\u0437\u043d\u0430\u0447\u0435\u043d\u0438\u044f \u043f\u0435\u0440\u0432\u043e\u0433\u043e \u0443\u0440\u043e\u043a\u0430, \u043d\u043e \u043d\u0435 \u0445\u043e\u0442\u0438\u043c \u0441\u0438\u043b\u044c\u043d\u043e \u0443\u0440\u043e\u043d\u0438\u0442\u044c \u0441\u043a\u0432\u043e\u0437\u043d\u0443\u044e \u043a\u043e\u043d\u0432\u0435\u0440\u0441\u0438\u044e. \u0418\u043b\u0438 \u0442\u0435\u0441\u0442\u0438\u0440\u0443\u0435\u043c […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":26311,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[702],"tags":[],"class_list":["post-34958","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-news"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\n","aioseo_head_json":{"title":"\ud83e\udd47When should you test the non-inferiority hypothesis? | ProHoster","description":"Article from the Stitch Fix team.","canonical_url":"https:\/\/prohoster.info\/en\/blog\/news\/kogda-stoit-proveryat-gipotezu-o-ne-menshej-effektivnosti","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"en_US","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041a\u043e\u0433\u0434\u0430 \u0441\u0442\u043e\u0438\u0442 \u043f\u0440\u043e\u0432\u0435\u0440\u044f\u0442\u044c \u0433\u0438\u043f\u043e\u0442\u0435\u0437\u0443 \u043e \u043d\u0435 \u043c\u0435\u043d\u044c\u0448\u0435\u0439 \u044d\u0444\u0444\u0435\u043a\u0442\u0438\u0432\u043d\u043e\u0441\u0442\u0438? | ProHoster","og:description":"\u0421\u0442\u0430\u0442\u044c\u044f \u043e\u0442 \u043a\u043e\u043c\u0430\u043d\u0434\u044b Stitch Fix.","og:url":"https:\/\/prohoster.info\/en\/blog\/news\/kogda-stoit-proveryat-gipotezu-o-ne-menshej-effektivnosti","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-10-31T19:01:31+00:00","article:modified_time":"2019-10-31T19:01:31+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"34958","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-01-21 21:15:19","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-03-01 02:12:27","updated":"2026-01-21 21:15:19","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/34958","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/comments?post=34958"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/34958\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media\/26311"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media?parent=34958"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/categories?post=34958"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/tags?post=34958"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}