The difference between the if and else cases around line 600 in srp.c differ in whether they do modular arithmetic, but the 2002 Wu paper (SRP6) says "All values are computed modulo N" giving at most sizeof(N) bits. This behavior is not specific to RFC 5054.
With that flag unset, you get > sizeof(N) outputs for B from srp_verifier_new().