in class activity 10_24 - xenon garcia
pdf
keyboard_arrow_up
School
Norco College *
*We aren’t endorsed by this school
Course
70A
Subject
Mathematics
Date
Jan 9, 2024
Type
Pages
3
Uploaded by HighnessButterflyPerson542
Xenon Garcia
Mat 70A
Fall 2023
Lab #7 A/B Testing
One special kind of hypothesis test we do in this class is called an A/B test. The steps
used to run an A/B test are the same as a general hypothesis test, but A/B tests have a
specific null hypothesis (that two samples were drawn from the same distribution). We
carry out this test by performing a permutation of our data.
Mid-Semester Check In:
What has been your favorite
topic/assignment/lecture/anything so far?
A/B Testing and Error Probabilities
Kevin, a museum curator, has recently been given specimens of caddisflies collected
from various parts of Northern California. The scientists who collected the caddisflies
think that caddisflies collected at higher altitudes tend to be bigger. They tell him that the
average length of the 560 caddisflies collected at high elevation is 14mm, while the
average length of the 450 caddisflies collected from a slightly lower elevation is 12mm.
He’s not sure that this difference really matters, and thinks that this could just be the
result of chance in sampling.
1.
Warmup:
When should you use an A/B test versus another kind of
hypothesis test?
You should use an AB test when determining whether two samples, also
known as an A group and a B group, were sampled from the same underlying
distribution/population
2.
What’s an appropriate null hypothesis that Kevin can simulate under?
The distribution of specimen lengths is the same for caddisflies sampled
from high elevation as those sampled from low elevation.
3.
How could you test the null hypothesis in the A/B test from above? What
assumption would you make to test the hypothesis, and how would you
simulate under that assumption?
The caddisflies shouldn't have come from different areas if the null hypothesis
is true. This means that it shouldn't matter whether the samples were labeled
as high elevation or low elevation. Based on this idea, you could change the
caddisflies' labels and use this "relabelled" data to find your test statistic.
4.
What would be a useful test statistic for the A/B test? Remember that the
direction of your test statistic should come from the initial setting.
Difference in mean lengths between the two groups.
Xenon Garcia
Mat 70A
Fall 2023
5.
Assume flies refers to the following table:
Elevation
Specimen Length
High
12.3
Low
13.1
High
12.0
(1007 rows omitted)
Fill in the blanks in this code to generate one value of the test statistic under the null
hypothesis.
def one_simulation():
shuffled_labels = flies.
sample(with_replacement = False)
.column(‘Elevation’)
shuffled_flies = flies.drop(‘Elevation’).with_columns(
‘Elevation’, shuffled_labels
)
grouped = shuffled_flies.
group
(
‘Elevation’, np.mean
)
means = grouped.column(‘Specimen length mean’)
statistic =
means.item(0) - means.item(1)
return statistic
6.
Fill in the code below to simulate 10000 trials of our permutation test.
test_stats =
make_array()
repetitions =
10000
for i in np.arange(
repetitions
):
one_stat =
one_simulation()
test_stats =
np.append(test_stats, one_stat)
test_stats
Xenon Garcia
Mat 70A
Fall 2023
7.
The histogram of test_stats is plotted below with a vertical red line indicating
the observed value of our test statistic. If the p-value cutoff we use is 5%,
what is the conclusion of our test?
If we look at the histogram, we can see that the
p-value
is greater than 5% in
the area to the
right
of the observed value. Our p-value is higher than the
p-value cutoff, so we can't say that the null hypothesis is false. Instead, we
say that the data tend to support the null hypothesis.
8.
Suppose that the null hypothesis is true. If we ran this same hypothesis test
1000 times, each time drawing a new random sample from the population
and with a p-value cutoff of 5%, how many times would we expect to
incorrectly reject the null hypothesis?
There is a 5% chance that a test will reject the null in the wrong way. It's likely
that we'll get it wrong 50 times out of 1000 tests (1000 * 0.05).
9.
What effect does decreasing our p-value cutoff have on the number of times
we expect to incorrectly reject the null hypothesis?
If we
decrease
our p-value cutoff, we are reducing the expected number of
times we’ll incorrectly reject the null.
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
- Access to all documents
- Unlimited textbook solutions
- 24/7 expert homework help