Otherwise you end up with changing colors, which doesn't convey the fact that it is indeed the same user:

This is how nheko does it: hash user ID and distribute it on a color wheel, ensure the contrast is good enough.
Element uses a simpler approach with a LUT and 8 colors, configurable with a theme.