Quality of tests

FS1-8 Quality of tests — coverage pack

1 specification leaves · notes, questions, answers and worked methods

FS1-8.1 · Type I and Type II errors. Size and Power of Test. The power function.

A Type I error rejects $H_0$ when it is true; a Type II error does not reject $H_0$ when a specified alternative is true.
The size is $P(\text{reject }H_0\mid H_0\text{ true})$ , while the power function is $\pi(\theta)=P_{\theta}(\text{reject }H_0)$ .
At a specified alternative parameter, power equals $1-P(\text{Type II error})$ ; higher power means the test is more likely to detect that departure from $H_0$ .
A common error is to call the nominal significance level the size without calculating the attainable probability of the discrete critical region.

Tier 1 · Easy

1. In a test of $H_0:p=0.4$ against $H_1:p>0.4$ , describe Type I and Type II errors in terms of $p$ , and define the size of the test.[4 marks]
Answer
- Type I: conclude that $p>0.4$ when $p=0.4$
- Type II: fail to conclude that $p>0.4$ when a specified value $p>0.4$ is true
- Size: $P(\text{reject }H_0\mid p=0.4)$
Method: Translate the two decision errors using the null value and the direction of the alternative. The size is the probability of the Type I error evaluated at the null parameter.

Tier 2 · Standard

1. Let $X\sim\operatorname{Bin}(12,p)$ . A test of $H_0:p=0.3$ against $H_1:p>0.3$ rejects $H_0$ when $X\geq7$ . Find the size, the power at $p=0.5$ , and the probability of a Type II error at $p=0.5$ .[6 marks]
Answer
- Size $=0.03860$
- Power at $p=0.5$ is $0.38721$
- Type II probability $=0.61279$
Method: The size is $P_{0.3}(X\geq7)=\sum_{x=7}^{12}\binom{12}{x}(0.3)^x(0.7)^{12-x}=0.0386008\ldots$ . At $p=0.5$ , power is $P_{0.5}(X\geq7)=0.3872070\ldots$ . The Type II probability is its complement, $1-0.3872070=0.6127930\ldots$ .

Tier 3 · Hard

1. For $X\sim\operatorname{Bin}(12,p)$ , a test of $H_0:p=0.4$ against $H_1:p>0.4$ uses critical region $X\geq8$ . Write its power function. Calculate its size, its power and Type II error probability when $p=0.65$ , and the expected number of rejections in $40$ independent repetitions at $p=0.65$ . Comment on whether it is a $5\%$ test.[9 marks]
Answer
- $\pi(p)=\sum_{x=8}^{12}\binom{12}{x}p^x(1-p)^{12-x}$
- Size $=0.05731$
- Power at $p=0.65$ is $0.58335$
- Type II probability $=0.41665$
- Expected rejections $=23.3$
- It is not a $5\%$ test because its size exceeds $0.05$
Method: The power function is the critical-region probability at a general $p$ . At the null value, $\pi(0.4)=0.0573099\ldots$ , which is the size. At $p=0.65$ , $\pi(0.65)=0.5833451\ldots$ , so the Type II probability is $1-0.5833451=0.4166549\ldots$ . Across $40$ repetitions, the expected number rejected is $40(0.5833451)=23.3338\ldots$ . Since $0.05731>0.05$ , the test does not have size at most $5\%$ .