poly2nb only works when the polygons are just touching, not crossing into each other, which is not common in cell segmentations. It would be nice to use any spatial predicate to make the graph, whether it's st_intersects (often after st_buffer) or st_within_distance (would it be faster than st_intersects + st_buffer)?