Constellation: Messages arrive out-of-order if I slam the NoC with lots of messages

[JL102]

Hi @jerryz123,

While working on my message queue implementation, based on rocc-to-noc (#1380), I noticed that if I send a ton of messages through the NoC at once (i.e., faster than the one core can process it), the messages from a given core stop arriving in order. To make sure it wasn't an issue with my own config, I went back to the RoCC-over-NoC example to see if it's still present there. Here are the two modifications I made to the customnoc branch in order to run this test:

1. To make it easier to look at the vcd, I reduced the # of cores in RoCCOverNoCConfig from 8 to 4
2. I changed the test to this:

#include "rocc.h"
#include <stdio.h>

void __main(void)
{
  unsigned long hartid = 0;
  asm volatile ("csrr %0, mhartid" : "=r"(hartid));
  unsigned long hart0 = 0;
  for (int i = 0; i < 10; i++) {
    ROCC_INSTRUCTION_SS(0, 100*hartid+i, hart0, 0);
  }

  while(1);
}


// Only hart0 gets here, harts > 0 should will execute the __main code
int main(void) {
  while (1) {
    uint64_t ron_reg;
    ROCC_INSTRUCTION_D(0, ron_reg, 1); // read from hart0's register
    printf("Hart0's RoCCOverNoC register is %ld\n", ron_reg);
  }
}

which is pretty similar to the test I was writing that led me to notice this issue.

In gtkwave, we can see on the bottom that the messages enter the NoC in order, but on the top, there are some cases where messages from a given core come out of order.

Here's the order that messages came in for each core. The messages from core 1 happened to arrive in order, but the same can't be said for cores 2 and 3.

100 200 300
101 201 301
102 203 302
103 202 303
104 204 304
105 205 305
106 206 306
107 207 309
108 208 308
109 209 307

I don't care that the global order stays consistent, e.g. if core 3 sends a message to 0 before core 1, but core 1 arrives first because it's physically closer, that's fine. I understand that in that case, it'd be a huge challenge to keep things in sync, and I think most multi-thread programming workloads assume that the threads will execute at different speeds. But I was hoping that the per-core order stays consistent with the order in which messages were sent, since I imagine that can cause issues when you start to make complex workloads.

Is there a way to rectify this? For example, is there a way to detect inside the module if the NoC is currently overloaded, and then wait before sending the message?

Thanks!

[jerryz123]

You can enable the fifo flow feature in constellation. If you specify FIFO FlowParams, and the routing relation implements a non-physically-divergent routing algorithm, then packets of the same flow will not be reordered in the network.

[JL102]

@jerryz123 Thank you for pointing that out, I just tested and it works!

I see the documentation doesn't have many details on FIFO yet. Are there any disadvantages to consider when enabling FIFO on flow params, such as worse performance or requiring extra hardware? I see the code changes in ucb-bar/constellation@cde8503 when you added FIFO support, but it goes a bit over my head.