Token key has changed in our dev env causing 401 errors

[box-turtle]

We have an issue in our dev environment that is blocking development. It does not occur in our prod environment.

In our dev environment with tenant id 391c8c4c-6a2a-40fd-ab98-226b6baa5155 we suddenly get 401 errors trying to open or create an item.

The key used by the token changed overnight - it used to be:

{
  "typ": "JWT",
  "alg": "RS256",
  "x5t": "zxeg2WONpTkwN5GmeYcuTdtC6J0",
  "kid": "zxeg2WONpTkwN5GmeYcuTdtC6J0"
}

but today it is

{
  "typ": "JWT",
  "alg": "RS256",
  "x5t": "z1rsYHHJ9-8mggt4HsZu8BKkBPw",
  "kid": "z1rsYHHJ9-8mggt4HsZu8BKkBPw"
}

this key doesn't match any of the ones listed here so Authentication fails.

I don't think we changed anything to cause this. The error we get trying to authenticate is this:

AuthController Exception
      Microsoft.IdentityModel.Tokens.SecurityTokenSignatureKeyNotFoundException: IDX10503: Signature validation failed. The token's kid is: 'z1rsYHHJ9-8mggt4HsZu8BKkBPw', but did not match any keys in TokenValidationParameters or Configuration. Keys tried: 'Microsoft.IdentityModel.Tokens.X509SecurityKey, KeyId: '3PaK4EfyBNQu3CtjYsa3YmhQ5E0', InternalId: '3PaK4EfyBNQu3CtjYsa3YmhQ5E0'. , KeyId: 3PaK4EfyBNQu3CtjYsa3YmhQ5E0
      Microsoft.IdentityModel.Tokens.RsaSecurityKey, KeyId: '3PaK4EfyBNQu3CtjYsa3YmhQ5E0', InternalId: 'G36ByiBJf4FJyQI0G2iTJebA9eVA_l0U0CWxNSjOgGg'. , KeyId: 3PaK4EfyBNQu3CtjYsa3YmhQ5E0
      Microsoft.IdentityModel.Tokens.X509SecurityKey, KeyId: 'k8kbXTgyy1QKJ54n61Qg3OldE_E', InternalId: 'k8kbXTgyy1QKJ54n61Qg3OldE_E'. , KeyId: k8kbXTgyy1QKJ54n61Qg3OldE_E
      Microsoft.IdentityModel.Tokens.RsaSecurityKey, KeyId: 'k8kbXTgyy1QKJ54n61Qg3OldE_E', InternalId: 'kT_njOCa_L5Lq_Zj2gYSarCBfpX11yr_uP_wGNVFaRc'. , KeyId: k8kbXTgyy1QKJ54n61Qg3OldE_E
      Microsoft.IdentityModel.Tokens.X509SecurityKey, KeyId: 'zxeg2WONpTkwN5GmeYcuTdtC6J0', InternalId: 'zxeg2WONpTkwN5GmeYcuTdtC6J0'. , KeyId: zxeg2WONpTkwN5GmeYcuTdtC6J0
      Microsoft.IdentityModel.Tokens.RsaSecurityKey, KeyId: 'zxeg2WONpTkwN5GmeYcuTdtC6J0', InternalId: 'hZKRXjUdItl0lQbxMJqZ_JDfVMWo8sg6rt7IAk5OP8c'. , KeyId: zxeg2WONpTkwN5GmeYcuTdtC6J0
      Microsoft.IdentityModel.Tokens.X509SecurityKey, KeyId: 'JBcrsrKJkcQB9HQF5wBfXN_pYcE', InternalId: 'JBcrsrKJkcQB9HQF5wBfXN_pYcE'. , KeyId: JBcrsrKJkcQB9HQF5wBfXN_pYcE
      Microsoft.IdentityModel.Tokens.RsaSecurityKey, KeyId: 'JBcrsrKJkcQB9HQF5wBfXN_pYcE', InternalId: 'W1XSqecv-7CxFAhWyiLFcDijTtoy_VjZbLYh77quvF4'. , KeyId: JBcrsrKJkcQB9HQF5wBfXN_pYcE
      Microsoft.IdentityModel.Tokens.X509SecurityKey, KeyId: 'Fdkuxg4kKVKOWUdliHLR1HArDmg', InternalId: 'Fdkuxg4kKVKOWUdliHLR1HArDmg'. , KeyId: Fdkuxg4kKVKOWUdliHLR1HArDmg
      Microsoft.IdentityModel.Tokens.RsaSecurityKey, KeyId: 'Fdkuxg4kKVKOWUdliHLR1HArDmg', InternalId: 'ZVKL1JFNa2LFZm_JscKjuuzEoNYDtA69hOygjjOZVrM'. , KeyId: Fdkuxg4kKVKOWUdliHLR1HArDmg
      Microsoft.IdentityModel.Tokens.X509SecurityKey, KeyId: 'RIT15NfQKEtO52Tt3FFujrvMbBk', InternalId: 'RIT15NfQKEtO52Tt3FFujrvMbBk'. , KeyId: RIT15NfQKEtO52Tt3FFujrvMbBk
      Microsoft.IdentityModel.Tokens.RsaSecurityKey, KeyId: 'RIT15NfQKEtO52Tt3FFujrvMbBk', InternalId: 'hlIFdczkXIjdBVkf6gy7HevbJOT7gyTI9z3NkulmGAU'. , KeyId: RIT15NfQKEtO52Tt3FFujrvMbBk
      Microsoft.IdentityModel.Tokens.X509SecurityKey, KeyId: 'MBAgzA7d8bs20uMGX8YEUdaQbbg', InternalId: 'MBAgzA7d8bs20uMGX8YEUdaQbbg'. , KeyId: MBAgzA7d8bs20uMGX8YEUdaQbbg
      Microsoft.IdentityModel.Tokens.RsaSecurityKey, KeyId: 'MBAgzA7d8bs20uMGX8YEUdaQbbg', InternalId: 'IfKLEbvfc6HckfB3Yg-IzkSktrJRiFKktnlUlu5c57w'. , KeyId: MBAgzA7d8bs20uMGX8YEUdaQbbg
      Microsoft.IdentityModel.Tokens.X509SecurityKey, KeyId: 'Mcx1jN8-euQbJmFKYWHZnyqxQNE', InternalId: 'Mcx1jN8-euQbJmFKYWHZnyqxQNE'. , KeyId: Mcx1jN8-euQbJmFKYWHZnyqxQNE
      Microsoft.IdentityModel.Tokens.RsaSecurityKey, KeyId: 'Mcx1jN8-euQbJmFKYWHZnyqxQNE', InternalId: 'S30TA8-qRESUErwR6PWiVxsHJKPtvUW_ETxpODOmx9A'. , KeyId: Mcx1jN8-euQbJmFKYWHZnyqxQNE
      Microsoft.IdentityModel.Tokens.X509SecurityKey, KeyId: '8IIDH6M8c3cXoeojKme8Jvqsr2Y', InternalId: '8IIDH6M8c3cXoeojKme8Jvqsr2Y'. , KeyId: 8IIDH6M8c3cXoeojKme8Jvqsr2Y
      Microsoft.IdentityModel.Tokens.RsaSecurityKey, KeyId: '8IIDH6M8c3cXoeojKme8Jvqsr2Y', InternalId: 'wWOCfK_lR0VkPkkdVrusLpeO-nvovgRiitiNWez0Bag'. , KeyId: 8IIDH6M8c3cXoeojKme8Jvqsr2Y
      '. Number of keys in TokenValidationParameters: '18'. 
      Number of keys in Configuration: '0'.  

[box-turtle]

Hmm, restarting our Auth server solved the issue.