How DDIO interacts with CAT?

[xl883109]

Hello,

I'm trying to figure out the relationship between DDIO and CAT.

By default, DDIO will use 2 ways of LLC:

$ rdmsr 0xc8b
c00

And all my CAT configuration is left unchanged by default:

$ pqos -s
NOTE:  Mixed use of MSR and kernel interfaces to manage
       CAT or CMT & MBM may lead to unexpected behavior.
L3CA/MBA COS definitions for Socket 0:
    L3CA COS0 => MASK 0xfff
    L3CA COS1 => MASK 0xfff
    L3CA COS2 => MASK 0xfff
    L3CA COS3 => MASK 0xfff
    L3CA COS4 => MASK 0xfff
    L3CA COS5 => MASK 0xfff
    L3CA COS6 => MASK 0xfff
    L3CA COS7 => MASK 0xfff
    L3CA COS8 => MASK 0xfff
    L3CA COS9 => MASK 0xfff
    L3CA COS10 => MASK 0xfff
    L3CA COS11 => MASK 0xfff
    L3CA COS12 => MASK 0xfff
    L3CA COS13 => MASK 0xfff
    L3CA COS14 => MASK 0xfff
...

My question is:

1. According to the manual, I can increase the ways of LLC that DDIO uses by the command like wrmsr -p 32 0xc8b 0xff0 (which will allow core 32 to use 8 ways of LLC, implied by 0xff0). Will it work as expected to access 8 ways of LLC? (I think so, because all the CAT cache masks are 0xfff, it won't conflict with the DDIO configuration 0xff0)

2. Regarding the CAT configuration, if core 32 is associated with COS1 and the COS1 cache mask is 0x00f, what will happen though? In this case, the cache mask of DDIO of the core 32 is 0xff0; it surely has no overlaps with the COS1 cache mask. Does this mean processes running on core 32 will not benefit from DDIO, because of no available ways of LLC?

3. About the low-level mechanism, how do DDIO and CAT work together? Is there any priority between DDIO and CAT？ In my humble opinion, the DDIO working process is like (please correct me if wrong):

• The NICs send DMA requests via PCIe to the CPU, with the target address located in pre-allocated in-memory buffers.
• The integrated I/O controller in the CPU will redirect the address to the LLC, instead of the address in memory.
  During this process, which stage will CAT participate in? And how does DDIO respect CAT?

Thanks.