Skip to content

prov/efa, common: Fix collisions in QKEY generator #11874

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
alekswn wants to merge 3 commits into
base: main
Choose a base branch
from
+39 −18
Open

prov/efa, common: Fix collisions in QKEY generator #11874

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
5734693
prov/efa: Use domain-scoped RNG state for QKEY generation
alekswn Feb 7, 2026
82ca8ed
common: Add 31-bit LFSR random number generator
alekswn Feb 7, 2026
d96d919
prov/efa: Use LFSR31 for connection ID generation
alekswn Feb 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions include/ofi.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -535,6 +535,29 @@ static inline uint32_t ofi_xorshift_random_r(uint32_t *seed)
return *seed = ofi_xorshift_random(*seed);
}

/**
* ofi_lfsr31_r - 31-bit Fibonacci Linear Feedback Shift Register
* @val: pointer to LFSR state (must be non-zero, range 1 to 0x7FFFFFFF)
*
* Generates a maximal-length pseudo-random sequence with period 2^31-1.
* Uses primitive polynomial x^31 + x^3 + 1 (trinomial for efficiency).
*
* Each value from 1 to 2,147,483,647 appears exactly once before the
* sequence repeats, ensuring no duplicates within the full period.
*
* Primitive polynomial from:
* "Error Correction Coding: Mathematical Methods and Algorithms"
* by Todd K. Moon (Wiley, 2005), Table of Primitive Polynomials
* https://web.eecs.utk.edu/~jplank/plank/papers/CS-07-593/primitive-polynomial-table.txt
*
* Returns: current state value (range 1 to 0x7FFFFFFF)
*/
static inline uint32_t ofi_lfsr31_r(uint32_t *val) {
uint32_t bit = ((*val >> 30) ^ (*val >> 2)) & 1;
*val = ((*val << 1) | bit) & 0x7FFFFFFF;
return *val;
}

uint32_t ofi_generate_seed(void);

size_t ofi_vrb_speed(uint8_t speed, uint8_t width);
Expand Down
25 changes: 7 additions & 18 deletions prov/efa/src/efa_base_ep.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,27 +148,16 @@ int efa_base_ep_destruct(struct efa_base_ep *base_ep)
return err;
}

static int efa_generate_rdm_connid(void)
static int efa_generate_rdm_connid(struct efa_domain* domain)
{
struct timeval tv;
uint32_t val;
int err;

err = gettimeofday(&tv, NULL);
if (err) {
EFA_WARN(FI_LOG_EP_CTRL, "Cannot gettimeofday, err=%d.\n", err);
return 0;
}

/* tv_usec is in range [0,1,000,000), shift it by 12 to [0,4,096,000,000 */
val = (tv.tv_usec << 12) + tv.tv_sec;

val = ofi_xorshift_random(val);
int ret;

ofi_genlock_lock(&domain->util_domain.lock);
/* 0x80000000 and up is privileged Q Key range. */
val &= 0x7fffffff;
ret = (int)ofi_lfsr31_r(&domain->connid_random_state);
ofi_genlock_unlock(&domain->util_domain.lock);

return val;
return ret;
}

static int efa_base_ep_modify_qp_state(struct efa_base_ep *base_ep,
Expand Down Expand Up @@ -409,7 +398,7 @@ int efa_base_ep_enable_qp(struct efa_base_ep *base_ep, struct efa_qp *qp)

qp->qkey = (base_ep->util_ep.type == FI_EP_DGRAM) ?
EFA_DGRAM_CONNID :
efa_generate_rdm_connid();
efa_generate_rdm_connid(base_ep->domain);
err = efa_base_ep_modify_qp_rst2rts(base_ep, qp);
if (err)
return err;
Expand Down
7 changes: 7 additions & 0 deletions prov/efa/src/efa_domain.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -306,6 +306,13 @@ int efa_domain_open(struct fid_fabric *fabric_fid, struct fi_info *info,
efa_domain->util_domain.domain_fid.ops = &efa_domain_ops;
}

/* generate inital seed using common function */
efa_domain->connid_random_state = ofi_generate_seed();
/* make sure each domain has unique seed */
efa_domain->connid_random_state ^= (uint32_t)(uintptr_t)efa_domain;
/* QKEY is 31 bit wide */
efa_domain->connid_random_state &= 0x7fffffff;

#ifndef _WIN32
err = efa_fork_support_install_fork_handler();
if (err) {
Expand Down
2 changes: 2 additions & 0 deletions prov/efa/src/efa_domain.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,8 @@ struct efa_domain {
size_t qp_table_sz_m1;
size_t mtu_size;
size_t addrlen;
/* Random state to generate QKEY */
uint32_t connid_random_state;
Copy link
Contributor

@sunkuamzn sunkuamzn Feb 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the random state needs to be stored at the IBV device level because QPs in different PDs (Libfabric domains) can still get packets meant for each other if the QKEY matches

That's still local to each process

bool mr_local;
struct dlist_entry list_entry; /* linked to g_efa_domain_list */
struct ofi_genlock srx_lock; /* shared among peer providers */
Expand Down