Metrics format

[maliberty]

OR adopted Metrics 2.1 as its format (see https://vlsicad.ucsd.edu/Publications/Conferences/388/c388.pdf). It is an easy format to write but is harder to parse. I'm opening this discussion to get input about improvements to the metrics format. The result should be useful to humans and machines.

As a starting point I think stage and category could naturally be mapped to json objects. Less clear is how name modifiers and classification modifiers would be represented.

A metric like finish__design_powergrid__drop__average__net:VDD__corner:default is a bit tricky to parse with stage=finish, category=design_powergrid, name=drop, name_modifier=average, classification modifiers net:VDD and corner:default. The 2.1 standard is a bit vague on whether this is conforming. Giving a reasonable representation for such a metric is a good test for any new scheme.

@gadfort @QuantamHD @rovinski @donn thoughts?

[maliberty]

I do think we should stick with json as the underlying format

[QuantamHD]

I do think the flattened nature of the metrics file isn't always the easiest to work with. In my experience there's a big drive to turn existing TCL reports into a structured machine readable format. Things like report_cell_usage and report_power are good examples where having a more structured JSON object representing these reports would be helpful.

In general a more 1:1 correspondence between the structure of the TCL reports would I think be welcome from our group.

[maliberty]

It would be helpful to make a specific recommendation as "hierarchy of key-value pairs" is rather vague. What do you suggest for the example given in the description?

[bangqixu]

It depends on how these would be aggregated and help readability. For example, if we implement a struct in C++ that stores this particular value (along with the information needed to describe it), one possible answer could be

struct VoltageDrop {
  enum type;   // average
  enum corner; // corner
  string net;  // VDD
  double drop; // value of drop
}

struct DesignPowerGrid {
  string stage_name; // final
  vector<VoltageDrop> voltage_drop;
  ... // E.g., layer usage, etc.
}

I still think starting from a use case (or a few) would help us better understand the actual requirement(s) and those requirements will evolve over time.

Assuming we just translate the above struct to JSON format, if one wants to know the average VDD drop before and after an optimization step, they can first filter by the stage name (by specifying the stage before and after the optimization step) at DesignPowerGrid level and then further filter the voltage_drops by type, corner and net to get what they want.

My point is that, to keep the cognitive load needed to use METRICS under control, more specific each key (which is the variable name here) is, the better. And abstraction can be introduced when it helps (obviously it is not always good to introduce more levels of abstraction).

[maliberty]

I think we should start with a discussion of the resulting format desired.

The C++ data structures will vary across the program and no assumptions should be made. In the current style the applications have no knowledge of how the metrics will be presented. The changes to write json directly from the application code are rather polluting in my view.

[bangqixu]

I am not saying that applications should store such information following the exact format. The example above was meant to give an idea of how a JSON-like C++ struct would look like for the metric we were discussing. To go from the in-memory data format of an application to the JSON file, someone needs to do the translation and the writing.

Sure, it is an option to ask the application to write JSON directly, but I think it would be better to decouple the data generation and JSON writing (hence applications just need to populate the JSON-like struct (or equivalently let a callback do it) and they don't need to worry about the writing part as this can be done by boost (plus some utilities).

I would say if we have some targeted use case in mind, that would help the discussion of desired format.

[maliberty]

The use case is the existing metrics file written by OR. The existing mechanism is already decoupled through Logger::metric. I would like to expand on it to support the format defined.

[gadfort]

I think the modifiers just add a level of complexity to the parsing that seems un-necessary.
I would suggest something like:

{
  "stage name": {
    "metric name": [
        {
          "modifiers": { <- or meta data or ...
            "<key>": <value>
          },
          "value": <value>
        }
    ]
  }
}

So the example provided would be:

{
  "finish": {
    "design_powergrid__drop__average": [ <- if you want to make this another level that is fine
       {
         "modifiers": {
            "net": "VDD",
            "corner": "default"
          },
          "value": 0.2
       }
    ]
  }
}

Writing the json would be trivial (if we just implement a proper json writer in utl from boost) and adding this to the database would also allow us to access these values from the tool (or inspect in the GUI).
To me this makes processing the metrics easier since we don't have to worry about the modifier names when looking up a specific set of them and can handle the modifiers separately.

[maliberty]

In the paper they also have category and a distinction between metric name modifier and classification modifier. The former seems useful but I'm less certain about the importance of the distinction between the modifiers. It appears the main difference is that name modifiers don't take a value (presumably the metric name is implicitly what is being modified).

{
  "stage name": {
    "category": {
      "metric_name": [
         {
            "modifiers": { 
              "<key>": <value>  // classification or modifier
            },
            "value": <value>
          }
      ]
    }
  }
}

perhaps

{
  "finish": {
    "powergrid": {
      "ir_drop": [
         {
           "modifiers": {
              "measure": "average",
              "net": "VDD",
              "corner": "default"
            },
            "value": 0.2
          }
       ]
    }
  }
}

Fwiw this was an early list of all proposed metrics.

[gadfort]

I think that would be fine

[donn]

Sorry for the late reply, had quite a bit going on...

I'm ambivalent. On the one hand, yes, it's pretty unwieldy, but we've already built a pretty solid system parsing and processing metrics with the 2.1 format in mind: https://github.com/efabless/openlane2/blob/b89f7866fd3d19da470220baf89d0e7804962941/openlane/common/metrics/util.py#L48

We can adjust for this by simply translating structured metrics emitted from OpenROAD to METRICS2.1, which it looks like there's more or less a 1:1 mapping anyway based on your proposed structure.