Generating rays and save them to .dat file

[drperpen]

Hi,
@JoanCharmant pointed me the way to be able to use different camera instrinsics in the same project, by using a ray_.dat file with the pre-generated rays. I was trying to learn how to do it reading here:

https://github.com/NVlabs/instant-ngp/blob/master/src/nerf_loader.cu
https://github.com/NVlabs/instant-ngp/blob/master/include/neural-graphics-primitives/nerf_loader.h
https://github.com/bmild/nerf/blob/master/run_nerf_helpers.py

I am still confused as in where should I save the .dat file, and if it should be a single .dat file or one per image? I tried to find one of these .dat files to see the format, but could not find one.
Thanks!

[leventt]

Hi :)

It looks like you need to generate a dat file per image with the same name next to the image like : ray_some_name.dat

and this dat binary file needs to match the number of pixels in this image to provide this structure per pixel:

struct Ray {
	Eigen::Vector3f o;
	Eigen::Vector3f d;
};

[drperpen]

Thanks for the reply. I managed to load the rays if I use rays_some_name.dat (not ray_some_name.dat):

20:32:42 INFO     Loading NeRF dataset from
20:32:42 INFO       ../projects/fox/transforms.json
20:32:42 SUCCESS  Loaded 50 images of size 1080x1920 after 0s
20:32:42 INFO       cam_aabb=[min=[0.5,0.5,0.5], max=[0.5,0.5,0.5]]
20:32:42 SUCCESS  Loaded per-pixel rays.

However, the training renders just black, so I am guessing the .dat files are not properly formatted. I am flattening a numpy array that interleaves camera origin (the same vector3 value per image) and ray direction (a vector3 different value per pixel). Then I save it using numpy.tofile

Before flattening:

[[ 3.16835952 -5.4794898  -0.97916609]
 [ 0.00033885  0.0001233   0.00054072]
[ 3.16835952 -5.4794898  -0.97916609]
[ 0.00033894  0.00012326  0.00054171]]

After flattening:

[ 3.16835952 -5.4794898  -0.97916609  0.00033885  0.0001233   0.00054072  3.16835952 -5.4794898  -0.97916609  0.00033894  0.00012326  0.00054171]

Then:

with open(file_path, 'wb') as fp:
        rays.tofile(fp)

[JoanCharmant]

Put a breakpoint to see if it's reading the values you expect. Maybe it's saving them as doubles?

[drperpen]

Thanks, Joan. I tried converting the flatten array to numpy.float32 (it was numpy.float64) but no luck either. Regarding the breakpoint, I am not sure how to achieve that when running ./testbed

[drperpen]

Well, it took a while :) I found the way to generate the origin and direction rays in the proper (nerf) way. I followed the procedure in the original nerf code

def get_rays_np(H, W, focal, c2w): #c2w = transform matrix
    """Get ray origins, directions from a pinhole camera."""
    i, j = np.meshgrid(np.arange(W, dtype=np.float32),
                       np.arange(H, dtype=np.float32), indexing='xy')
    dirs = np.stack([(i-W*.5)/focal, -(j-H*.5)/focal, -np.ones_like(i)], -1)
    rays_d = np.sum(dirs[..., np.newaxis, :] * c2w[:3, :3], -1)
    rays_o = np.broadcast_to(c2w[:3, -1], np.shape(rays_d))
    return rays_o, rays_d

As @JoanCharmant pointed out, make sure you use float32, not float64