observability | registered schema count != max(schemaId)

[nhomble]

Hi everyone,

Before reading into the investigation so far, my question is: is the registered schema count metric the right metric monitor the total number of schemas served?

Before digging into the code, I was expecting the value of this metric to equal the maximum assigned schemaId, but in practice I see the metric value is higher than the max schemaId I can query against SR. (To query, I do /schemas/ids/:schemaId).

---

The behavior that I am seeing is it tracks the number of schema records in the _schemas topic but if a schema was soft deleted, then the registered value still increments.

    if (schemaValue != null) {
      // Update the maximum id seen so far
      idGenerator.schemaRegistered(schemaKey, schemaValue);

      if (schemaValue.isDeleted()) {
        lookupCache.schemaDeleted(schemaKey, schemaValue, oldSchemaValue);
        updateMetrics(metricsContainer.getSchemasDeleted(),
                      metricsContainer.getSchemasDeleted(getSchemaType(schemaValue)));
      } else {
        lookupCache.schemaRegistered(schemaKey, schemaValue, oldSchemaValue);
        updateMetrics(metricsContainer.getSchemasCreated(),
                      metricsContainer.getSchemasCreated(getSchemaType(schemaValue)));
      }
    } else {
      lookupCache.schemaTombstoned(schemaKey, oldSchemaValue);
    }

  // Update the maximum id seen so far

What seems like a bug is that the records in _schemas for the same subject might have multiple records for the same id (this seems possible when the actual schema value is the same across subjects).

To add to the confusion, when the topic is compacted the metric can also go down which begs the question why the registered count went up in the first place.