Undesratnding the usefullness of the ranks transformation

[guramaB]

Hi Jakob,

I'm trying to understand your CMI calculation.
Specifically, I have troubles with this line of your code (cmiknn.py, line 179):
array = array.argsort(axis=1).argsort(axis=1).astype(np.float64)

1. I do not get the purpose of the second argsort():
   You have observations in columns. The first argsort(axis=1) transforms the actual physical values of the observations into, let's say, relative ones. Instead of knowing the value of x for the first observation, you now have the rank, i.e. how large this x value is in comparison to all other x values. If you however, now sort the rows for the second time, you are mixing up the original observations, aren't you?

2. On a general level, I do not understand how changing the data into ranks supports your calculations, since the knn-search is based on distances and these are blurred after the transformation. Especially problematic I see it in several dimensions, where the distance "1" between two neigbouring ranks in one dimension can be much smaller/larger than the same distance "1" in another dimension.
   I'm afraid I'm missing something crucial here, but I couldn't find any clue in your 2018 paper (Frenzel and Pompe(2007) do not seem to mention the transformation). Do you have an explanation, or a reference that explains the rational behind the ranks transformation?

Many thanks for a reply.

Best wishes,
Barbora

[jakobrunge]

Dear Barbora

ad 1) Check it yourself, you need to run argsort twice to get ranks:

a=np.array([0.4, 0.8, 0.1])
print(np.argsort(a))
print(np.argsort(np.argsort(a)))
[2 0 1]
[1 2 0]

ad 2) Honestly, this is just a very empirical finding that ranks work better and it would be great if someone could study this in detail. What you describe as problematic is exactly the idea: rescaling all dimensions to ranks avoids the problem of very different scales in different dimensions.

Jakob

[guramaB]

Hi Jakob,

thanks for the quick reply!
Oh, yes, with argsoft() I got confused with ranks and sorting indices when thinking about it. 😳
As to the second point, I think it makes a "click" now. I'll write you more to your work email.

Barbora