Give fed admins the ability to adjust server weights on the fly

[jhiemstrawisc]

It'd be nice if fed admins could adjust server ordering through configuration while debugging.

This idea was originally presented as "give the admin the ability to adjust the server weights." I don't think this will have the intended effect because small changes to server weights could put the weight completely out of distribution for the sorting list, causing the small change to induce wide swings in server ordering. For example, you don't know what halving a server weight will do to the ordering unless you know all the other server weights because these weights are not uniformly distributed. This is partly why server ordering in the Director was such a pain to get right.

Rather, we should think about how an admin might specify information about where the server should appear in the sorted list. For example, maybe they can say explicitly "When this server is up for consideration to serve a request, I want it to be XXX in the list." Maybe they can say "I want there to be a XXX% lower probability this cache is picked in the top 3". Or maybe they can say "wherever this server appears in the sorted list, move it back XXX spaces".

As is, this issue isn't meant to suggest one approach over the other, only to checkpoint a discussion with some ideas.

[jhiemstrawisc]

I'd be interested to hear @williamnswanson's and @brianhlin's plain english descriptions of what you hoped to accomplish through whatever knob this gets turned into, and how you think that will help you debug/diagnose problems.

[williamnswanson]

What we're really looking for here is a way to increase or decrease the proportion of transfers that go to a given cache. Currently Pelican assumes that all caches are of "average" capability, but since most caches are either above or below average, this leads to small caches getting blasted and larger caches going under utilized.

I think we'd like Server operators tell Pelican / the federation about the capabilities of their server and the director weights it's ~6 top endpoints according to their relative abilities, but for now maybe being able to note in a Registry entry that a server should have it's random sorting component be given more or less weight in the Director's sorting algorithm? Perhaps only after getting the list of the ~6 closest, non-degraded endpoints, so that we don't start sending people across the world to a beefy cache?