contents of volume raw files ?

[KLXMan]

Thanks a lot to all people involved in instant-ngp. It's really a great job !
I am trying to use instant-ngp with my one picture sets, in order to build printable 3D models. I was used to standard photogrammetric tools (Metashape, MicMac), but I wanted to try NeRF, which is supposed to deal better with specular reflection issues.
I used the suggested scheme: ColMap -->colmap2nerf.py--> instant-ngp (version v1.0dev on Windows10) with default settings.

Rendering results (3D views in the GUI) are very impressive after a few minutes, but the meshes built with "Mesh it" fonction are very disappointing (unusable !).
So I tried to investigate exported data (I am developing in C++).
Among them, density PNG structure is easy to understand. I started by observing the density values along a 3D ray tracing (using an interpolation fonction for the density cube). I expected to get a very sharp transition between the model surface and the outside, but it's not, so that it is difficult to fix a density value to locate the model surface.
Now, I am wondering about using the raw volume output. Presently, the "Save raw volume" function gives me a set of 5 files (256x256x256_0.bin to 256x256x256_4.bin), each one containing 256x256x256x16 bytes (probably 4 floating values for each cube point ?).
I did not find any information about these binary data files. Could any one give some clues ? (and ideally the way to get the exact model surface from them...). .
Thanks in advance

[arnabdeypolimi]

As far as I know, the shape of the raw file is 256X256X256X4 where 256 is the cube resolution and 4 channels are raw radiance and sigma values at that 3D location of the cube. Basically, it creates sample points(XYZ) inside a cube of resolution 256 and returns the radiance and sigma for each of the points.

I have a question do you know how can call the "save raw volume" function using python API?

I got the raw volume using the GUI but I want to get it using python API.

[franz96521]

I did not find the way to do it so I made my own function copying what was in GUI mode and recompiled the project
in testbet.cu after Testbed::save_snapshot add

void Testbed::save_raw_volumes()
{
	static bool flip_y_and_z_axes = false;
	BoundingBox aabb = (m_testbed_mode == ETestbedMode::Nerf) ? m_render_aabb : m_aabb;

	auto res3d = get_marching_cubes_res(m_mesh.res, aabb);
	auto effective_view_dir = flip_y_and_z_axes ? vec3{0.0f, 1.0f, 0.0f} : vec3{0.0f, 0.0f, 1.0f};
	auto old_local = m_render_aabb_to_local;
	auto old_aabb = m_render_aabb;
	m_render_aabb_to_local = mat3(1.0f);
	auto dir = m_data_path / "volume_raw";
	if (!dir.exists())
	{
		fs::create_directory(dir);
	}
	for (int cascade = 0; (1 << cascade) <= m_aabb.diag().x + 0.5f; ++cascade)
	{
		float radius = (1 << cascade) * 0.5f;
		m_render_aabb = BoundingBox(vec3(0.5f - radius), vec3(0.5f + radius));
		GPUMemory<vec4> rgba = get_rgba_on_grid(res3d, effective_view_dir, true, 0.0f, true);
		save_rgba_grid_to_raw_file(rgba, dir.str().c_str(), res3d, flip_y_and_z_axes, cascade);
	}
	m_render_aabb_to_local = old_local;
	m_render_aabb = old_aabb;
}

in testbed.h after void save_snapshot() add
void save_raw_volumes();

and finaly in python_api.cu after .def("save_snapshot" ) add
.def("save_raw_volumes", &Testbed::save_raw_volumes)

[arnabdeypolimi]

thanks a lot.

[KLXMan]

Thank you for your contributions.
In the meantime, I have more or less found the answer to my first question.
I confirm that every file is organized as a 256x256x256 cube of float[4] values (4 bytes each, Intel endianness)
Among these 4 float values, the last one is much higher than the others and seems to be sigma.

The number of files depends on the aabb value. With aabb=16, there are 5 files, each one containing a 256x256x256 cube, with respective spatial resolutions reduced by a factor 1,2,4,8,16. It is a very smart organisation, which means that even with aabb=16, the best resolution remains available in the central unitary cube (first file). This is a very good thing, because it is often necessary to have a high aabb value as soon as there is a background around the central object, to avoid "cloudy" artefacts around this central object in the Nerf result.
A curiosity: I observed that in each cube (except the first one), the data corresponding to the previous one (a central cube 128x128x128) are set to zero (black)!
It does not provide any memory saving, so I guess that it is due to the data building process itself.

Sorry, I did not try to save the raw volume without the GUI, so I have no answer to your question. I save it with a GUI, and then read it in C++.

Some other remarks:

• even using sigma values from the raw volume (insted of density PNG), transitions at the edges of the detected object are not as sharp as I expected...
• I did not deal with RGB values, but there is something that I do not understand in the principle. As I understand from the Nerf theory, these values depend on the view direction (network inputs are 5D: (x,y,z) + direction). So how can they be reported with 3D cubes (raw volume cubes or rgba_slices folder) ?

Regards

[prasadvagdargi]

A quick matlab snippet to read the volume:

fid = fopen(".\256x256x256_0.bin",'r'); % Set the file ID
outmat = fread(fid,'single'); % Read as 4 byte floats
assert(length(outmat)/(256^3) == 4); % Assert shape - this is valid for 256 voxel grid only. Change as needed
vol = reshape(outmat,4,256,256,256); % Reshape to the volume format

% Write into nifti files or others as needed
niftiwrite(squeeze(vol(4,:,:,:)),".\Density.nii");
niftiwrite(squeeze(vol(1,:,:,:)),".\R.nii");
niftiwrite(squeeze(vol(2,:,:,:)),".\G.nii");
niftiwrite(squeeze(vol(3,:,:,:)),".\B.nii");