Add python CLI utility to get near countries codes.

[Nyanraltotlapun]

Add python CLI utility to get near countries codes. As preparation for ability to select near countries automatically.
This is only "get_near_countries.py" file addition. No changes to reflector-simple itself.

[Nyanraltotlapun]

Ok so, now I added all other bits.
My testing shows no problems, tell me what you think please.

[killajoe]

https://github.com/endeavouros-team/PKGBUILDS/pull/370/files#diff-8cf984f922b598af4b84aa237d628f6732f638d6d75de94b9bce289e916de045

for i in range(1, recursion_depth, 1)

This loop does not do what your help text says ?

you already populate direct neighbors before the loop:

result = set(NEAR_COUNTRIES.get(search_country,[])) ? The starting country can appear in results (logic flaw) ---> Example:

DE → FR

FR → DE

So DE ends up in its own output. anbd yea.. case sensitive? If the user runs:

--country de ? --->

if recursion_depth > 12

There is no recursion here (it’s iterative)? also not coding but: Some entries are politically or geographically debatable:

CN includes HK, MO (special regions)

FR doesn’t include overseas borders

EG–PS border is disputed

[Nyanraltotlapun]

Ok. So, I add line to remove input country from output. And made input case insensitive.

[Nyanraltotlapun]

for i in range(1, recursion_depth, 1)

This loop does not do what your help text says ?

you already populate direct neighbors before the loop

Yes. So rec depth starts from 1. And in this case, loop will not execute. I can refactor this a bit.

[Nyanraltotlapun]

There is no recursion here (it’s iterative)?

Its recursion mapped into loop :)
So in code it looks like loop, but it does recursive search.
Python Idiomatic do not like recursive code.

[Nyanraltotlapun]

also not coding but: Some entries are politically or geographically debatable:

It can be, its data automatically parsed from db that automatically parsed from geodata source.
Most of it should be geographically correct - based on distance.
Because "borders" in context of this utility is not political term.
In future I may use some other source of data. But main purpose of utility is to determine geographical closeness. In a simple but useful way.
Thus, I think it is logical to change DB records if they found to be invalid in some way (country is actually not close, or country not exist anymore).

[Nyanraltotlapun]

for i in range(1, recursion_depth, 1)

This loop does not do what your help text says ?

I refactored code a little so it looks more clean, hopefully.

[dalto8]

Can you better explain the point of getting nearby countries?

Geographic proximity is not a very good indicator of network proximity or performance so I am not sure why it would be beneficial to pull in countries nearby.

[Nyanraltotlapun]

Can you better explain the point of getting nearby countries?

Geographic proximity is not a very good indicator of network proximity or performance so I am not sure why it would be beneficial to pull in countries nearby.

Not very good, but some kind of indicator.
It adds more options for reflector to test what leads to better choices.
For example, I am currently in Hungary, and local mirrors almost never a better choice, at-least for Budapest and my internet provider.
I think also in forum post person mentioned that it always need to choose nearby countries.
And my experience is similar - I always need to check additional countries.

So making this behavior as default, as I see it, removes necessity for many users to make additional tweaking, so basically better defaults.

But I probably need to mention that it works better when default ranking parameter is "rate" and not "age".
I want to propose to change this, because "age" ranking often leads (for me) to overloaded and painfully slow mirrors.

[dalto8]

I understand the idea of needing to check to more countries. However, it is rare that geographically close countries are actually the fastest in that scenario. The network connections don't work that way in most cases.

If you want to search by proximity, it is better to explore network proximity like rate-mirrors does.

[Nyanraltotlapun]

However, it is rare that geographically close countries are actually the fastest in that scenario

Not necessary fastest - but candidates for determining the fastest. Reflector searching for the fastest, and we only provide him with reasonable list of candidates to search.
Also consider situation when some country have only few or have no mirrors at all.

I am also pretty sure that there is positive correlation between geographical closeness and internet connection quality - on average it is better the closest points are. If you think it is unreasonable assumption, I will search for some data on this regard.

[dalto8]

My point is that network proximity is a better indicator. Since that data exists, why wouldn't we use that?

Also, at least for the US, I can pretty much guarantee that Canada and Mexico will not make sense. I use two different ISPs and I have never seen a Canadian or Mexican mirror. The connectivity to Europe is better. I often get European mirrors in my top 10 even though there are many mirrors in the US. From where I am, I have better connectivity to another continent separated by an ocean than I do to parts of my own country.

[Nyanraltotlapun]

My point is that network proximity is a better indicator. Since that data exists, why wouldn't we use that?

Can you point this data to me?
I assume, the problem with it can be - frequent change.

[dalto8]

I would take a look at rate-mirrors to see how he is storing the data.

That being said, I think we should step and think about what we are trying to accomplish. At least to me, the primary purpose of selecting a list of countries in reflector-simple is because you want to limit your results to only those countries.

If you don't care about that, why select a country at all?

[Nyanraltotlapun]

I would take a look at rate-mirrors to see how he is storing the data.

Ok. Ill look to it a bit later. But this also can rise a question - why not just use it all together instead?

At least to me, the primary purpose of selecting a list of countries in reflector-simple is because you want to limit your results to only those countries.

Yes. But not limit too much, just enough.

why select a country at all?

Because rating large number of mirrors is time consuming, and also creates unnecessary network load. But from user perspective mostly time consuming.

It seems to me, that a better way will be to form list on network(hops) distance, then select n freshest from it, and rate them by download speed.
Because we can get hops for all mirrors simultaneously, it should add only small delay to whole process.

But this also leaves question of countries choice limiting and it default behavior. Most people, will not want to limit servers to their country by default in the case of algorithm above. So default need to be different ether way.

[killajoe]

Personal i do always use rate-mirrors on the installer livesession. But reflector-simple is very helpful for users having issues in areas like india with no good mirrors in the country at all. And it is almost everytime helping them to add nearby countries but also Geman and US mirrors as fallback .. In the end it is always true that one good working mirror would be all needed to get OS installed properly.

Its not mainly nearby countries that will provide the best connections its about the infrastructure, the knots and the quality of lines too. Reflector itself is somehow based on Countries? so is reflector-simple may not the very best tool compared to what we have today for arch mirror selection tools. But a valid and logical easy to use and understand option for a lot of users. And yea users.. . its the first in the welcome app and because of that people using it. we could put rate-mirrors first.

And on the india example it is an issue to only have india in the preselection by default it should always add some other country as fallback.

[dalto8]

why not just use it all together instead?

We offer both reflector-simple and rate-mirrors in the welcome app so people can choose which they prefer.

That being said, before you spend too much more time on this, I think we should get some input from @manuel-192 as well. He is the maintainer of reflector-simple so it would be good to understand if this is a direction he wants to take the application.

Also, in case we haven't mentioned it yet, we do appreciate people wanting to jump in and contribute so thank you for that.

[killajoe]

Also, in case we haven't mentioned it yet, we do appreciate people wanting to jump in and contribute so thank you for that.

100% @Nyanraltotlapun very welcome !

And for sure it will need @manuel-192 to decide if he want to merge this or not.. i do only commented here.

[Nyanraltotlapun]

We offer both reflector-simple and rate-mirrors in the welcome app so people can choose which they prefer.

Right now, it looks like this actually:

[dalto8]

Because you probably don't have rate-mirrors installed.

[Nyanraltotlapun]

Because you probably don't have rate-mirrors installed.

Yes. I am very default man 😅

[manuel-192]

@Nyanraltotlapun
Sorry for the delay.

The idea of using neighbor countries might work in some cases but not all.
And there are reasons why using neighbor countries for this is simply not possible.
Reflector-simple (and reflector) already provides an easy way to choose countries,
and rate-mirrors does a good job too.

That's why I think this near-countries idea is not something we can use for reflector-simple.

Of course you are free to, for example, create a package and publish it in the AUR. Then users can decide if it suits their needs.

Thanks for the interest anyway!

[Nyanraltotlapun]

@Nyanraltotlapun Sorry for the delay.

The idea of using neighbor countries might work in some cases but not all. And there are reasons why using neighbor countries for this is simply not possible. Reflector-simple (and reflector) already provides an easy way to choose countries, and rate-mirrors does a good job too.

That's why I think this near-countries idea is not something we can use for reflector-simple.

Of course you are free to, for example, create a package and publish it in the AUR. Then users can decide if it suits their needs.

Thanks for the interest anyway!

Can you give me permission to distribute your original code with my changes under GPL v3.0 license?

[Nyanraltotlapun]

Reflector-simple (and reflector) already provides an easy way to choose countries

But this selection wont last to next app launch...