106.x Questions on Significance Testing mission

Q&A DQ Courses
#1

Hi there,

For the first time, following the Data Analysis path at Dataquest, I had to consider an outside source to consolidate new concepts.

It might be the case that learning hypothesis testing is indeed challenging, but the fact is that I finished mission 106 feeling a little confusing.

I then tried some outside sources (mainly Khan Acad) and also went through mission #106 again, and I think now I feel more comfortable to share the below comments & doubts:

  1. Everywhere I checked about hypothesis testing, they state that when the p-value is significant (larger than the threshold) the outcome is that you just cannot reject the null hypothesis (and might keep going in putting it to test to new alternative hypothesis), which is different from accepting that the null hypothesis is true (and the latter is clearly stated in 5/10 & 8/10 within this mission).
    It might be the case that strictly speaking, in theory, one “cannot reject” the null hypothesis in such a case, and that in the real world they just take it as if the null hypothesis is true (something like “feet on street” wisdom from experienced DA/DS professionals). If that is the case, OK, but I suggest this to be clearer in the mission. Other than that, I think concepts should be corrected in 5/10 & 8/10.

  2. When the permutation test is put to work on 5/10 I just did not get why weights from the initial groups A and B are joined in the all_values list so as to start randomnly sampling it to build the sampling distribution. Wouldn’t it make more sense to keep groups A and B separated (each with 50 samples) and sample from each group to build the sampling distribution on top of each mean difference computed over separated A and B randomized groups?
    If you’re mixing up volunteers from both groups and sampling from this joined group for the permutation test, each sample will probably include people who took the pill as well as people who took the placebo and it seems you’re losing the control group at this stage of the process…I just don’t get it.

By the way, I tried to do things this way (randomly sampling with 10 and 5 weights from each separated group of 50) and got a totally different outcome with a very significant p-value (which would in turn not reject the null hypothesis).

In time, I’m very happy with Dataquest but found interesting to add up my comments/doubts.

Thanks!

#2

Obrigado, Celso.

This is an old course that we are planning to rewrite. It’s likely that Alex will be the one rewriting this course.

I’ll tag him here so that he’s aware of your feedback. CC @alex

#3

Eu é que agradeço.
Tks. Abraço.

1 Like
#4

I was about to ask the same question, why they mixed the two groups together and choose random samples from each. It confused me and doesn’t make any sense for me. Thanks to clarify this point.

#5

Hi Bruno, I wonder if this course has been updated? I too have been confused, starting on Lesson 5 of the significance testing mission: https://app.dataquest.io/m/106/significance-testing/5/permutation-test

I’m not too sure how the np.random.rand() function helps with assigning values to specific groups. For example, if assignment chance is >= .5, append to group_a? Not sure how this all plays out.

And to answer the question from OP, I believe that mixing the control and test audience is supposed to help us determine the variation in our results.

#6

Not yet.

Regarding your question concerning screen 5, can you please ask it in a separate post? Feel free to ping me.

Thanks.

#7

Is this solved yet? I also have similar question and confusion with this thread, but cannot find the answer. Why are we using np.random.rand to randomize between 0 to 1 and grouping the weight loss values to 2 groups? What is the basis/reasoning behind this separation methods, is this the way to sample without replacement?
Please let me know if I have the concept right:
Permutation test is assuming that these 2 groups has no significant difference, and therefore we put them back into one group and sample 2 groups again randomly without replacement. We then see if our original mean difference is on the extreme compared to the mean difference distribution of the resampled groups.

#8

@mariastudybelajar Hey.

Please ask your question in a separate topic. The current topic isn’t supposed to encompass all questions concerning mission 106.

Thanks.