(no subject)

[ny_quant]

Добавление ко вчерашнему:



the data in Figure 2 shows a decrease in infection rates after countries eased national lockdowns with >99% statistical significance""

Comments

[misha-b.livejournal.com]

Btw, I was thinking about why the 99% thing is so annoying to me. The issue is the following -- we need to separate our hypothesis from the null hypothesis. 99% is the probability that the data does not come from the null under some probabilistic model.

Without describing the model for the null 99% makes no sense. And the definition of the null is non-trivial -- for example it does not make sense to compare this to iid data, since the rates in different states or countries are obviously correlated. Thus, giving a number like 99% seems manipulative, designed to give some scientific veneer to potentially problematic inferences.

[ny-quant.livejournal.com]

OK, I see your point. But what is the practical outcome? To throw the baby with the bathwater?

[misha-b.livejournal.com]

First we have to make sure that there is a baby. For that we need to know something about their methodology, which they seem to be hiding.

I actually find it hard to believe that those JPM guys do not understand statistics or how to properly deal with data. They seem to be solid quantitative types. Therefore it feels like they have some agenda.

[ny-quant.livejournal.com]

I meant it in general. If you are generally annoyed by the hypothesis testing methodology then why stop with this paper? Why not throw out everything that statisticians have done this way?

But yes, the guy is very smart and silly mistakes are not to be expected.

[ny-quant.livejournal.com]

Я согласен, что p-values могут быть misused and abused. Но это ещё не повод видеть желтый флаг в significance level.

[misha-b.livejournal.com]

Without an explicitly stated model it definitely is. If there is a model, we have to consider its validity.

[ny-quant.livejournal.com]

// 99% is the probability that the data does not come from the null under some probabilistic model.

Could you remind me: if we're testing a linear regression, what is the null hypothesis? I don't recall this being discussed in the last class I took almost 30 years ago, nor in anything I've read later. I may be wrong but I believe it is not necessary b/c we're hiding behind the CLT.

[misha-b.livejournal.com]

It might be something like this. Say we are in a fixed design setting, i.e. we have pairs (x_i, y_i), where x_i are assumed to be fixed. We write a linear model y = w*x + e, and put some distribution on the noise e , e.g. e is a Gaussian. Then we can work out various probabilities in terms of that.

Take this with a grain of salt -- I have not looked carefully at such problems in the testing context.

[ny-quant.livejournal.com]

Можно рассмотреть разные распределения e, и это может иметь последствия в терминах MLE. Но какАя же альтернативная гипотеза vs. y = w*x + e?

[misha-b.livejournal.com]

Well, the null hypothesis is that the data comes from the model of that form. The alternative is that it does not -- given enough data I may be able to reject the null hypothesis with some probability, 99%, say.

[ny-quant.livejournal.com]

I may be wrong but I think it is the other way around.

We test the alternative (that it does not - w/o specifying the nature of the alternative), and if we reject it then we choose to live with the opposite, i.e. the one that we want to accept in the first place. We know we can't prove it but we rejected the alternative, so we deem it good to go.

[misha-b.livejournal.com]

Your description is correct, but you reject the null hypothesis, that's how it is typically formulated, I think.

More precisely there are two ways to do hypothesis testing: 1. you can reject the null (i.e. prove that the null does not hold with high probability in your model). This is called Fisher testing.
2. You choose between two hypotheses A and B. This is called Neyman-Pearson hypothesis testing.

There is a nice discussion on wikipedia: https://en.wikipedia.org/wiki/Statistical_hypothesis_testing

It is pretty subtle stuff actually.

[ny-quant.livejournal.com]

An alternative process is commonly used:

1. Compute from the observations the observed value t_obs of the test statistic T.
2. Calculate the p-value. This is the probability, under the null hypothesis, of sampling a test statistic at least as extreme as that which was observed.
3. Reject the null hypothesis, in favor of the alternative hypothesis, if and only if the p-value is less than (or equal to) the significance level (the selected probability) threshold

What exactly is null hypothesis? That we got our "good fit" by accident, right? No distributional assumptions are required.

Случайно наткнулся на экономиста, который решил написать серию заметок типа введения в статистику.

https://hroniki-paisano.livejournal.com/118867.html
https://hroniki-paisano.livejournal.com/119264.html

Интересно, что он тоже не заморачивается спецификацией нулевой гипотезы.

[misha-b.livejournal.com]

We need a model for point two above: "This is the probability, under the null hypothesis, of sampling a test statistic at least as extreme as that which was observed."

For example what is 99% in your original post refers to? I have no idea, but here is one possibility: assume that the point has equal probability to be above and below the line and assume that the points are sampled iid. Under this model the picture shown in the graph is very unlikely and we can reject this hypothesis. However, iid is an key assumption here. If these data are not iid, this conclusion in general cannot be made.

For example, imagine that the rates are perfectly correlated between the states. Then either all of them will be below the line or all of them will be above the line.

[ny-quant.livejournal.com]

I certainly agree that one needs to state what hypothesis we are testing, i.e. what we want to accept in the end, or in the worst case reject. I'm sure the author of this note did have a very well-defined hypothesis but this little detail didn't get into condensed summary that was available to me.

I thought your original complaint was that the model that we hope to reject based on low p-value is not stated.

[misha-b.livejournal.com]

That's right, the issue I had was using a very suggestive number, like 99% without explaining the model. It is quite possible that the author had a very reasonable model. But I have seen this misused or abused many times, so I am by default skeptical.

[misha-b.livejournal.com]

99% is not the worst actually -- sometimes people have 99.9999% or something like that and you immediately suspect something is fishy.

[ny-quant.livejournal.com]

Я прочитал вашу статью. Во-первых, я, конечно очень слабо разбираюсь в теме: Decision Trees and Ensemble Methods единственное что я хоть как-то понимаю. Во-вторых, мне очень трудно привязать ваши результаты к какой-либо конкретной задаче. В моём маленьком мире очень мало полезных фич (по крайней мере я так думаю) и очень много шума. Было бы хорошо иметь перед глазами какой-то простой конкретный пример когда действительно получается омега-shaped curve.

[misha-b.livejournal.com]

Sorry, forgot to answer this. We did not want to call the curve omega-shaped or W-shaped because the second part typically does not rise. It is "cursive-u" shaped in most examples we looked at.

You can take a look at a simple synthetic model with linear regression, where the features of this curve can be seen: https://arxiv.org/abs/1903.07571