Multi-GPU Particle Tracking & Immersed Boundary

[jasonxauat]

• Why doesn’t FluidX3D support multi-GPU passive particle tracking? What’s the biggest technical roadblock here? (e.g., data transfer between GPUs? Syncing particle positions?)
• Any ideas for implementing multi-GPU support? Would splitting the domain across GPUs work?

Also, I’m trying to implement an immersed boundary method by adapting the existing Particle code. It has seemed to work.

Thanks in advance!

Regards,
Jason

[ProjectPhysX]

Hi @jasonxauat,

particles on multi-GPU require a lot of extra logic to determine which particles move between GPU domains and have to be copied over in between time steps. There is good solutions for this in literature - especially immersed-boundary codes for blood flow simulations, e.g. Ames et al., Kotsalos et al.. I was just too lazy to implement it.

Kind regards,
Moritz

[jasonxauat]

Hi, Dr. Moritz, Based on your suggestion, I've developed a simpler modification plan to enable multi-GPU support for particles in FluidX3D. Here's a summary of the approach:

Proposed Multi-GPU Particle Support Solution

Instead of implementing complex particle migration logic between GPU domains, we propose a simpler approach where each GPU stores the complete particle list, but only computes interactions for particles within its domain.

Key Modifications:

1. Full Particle Replication: Each GPU domain stores the complete particle list rather than partitioning particles across domains.

2. Domain-aware Computation: Modify the kernel to check if each particle is within the current GPU's domain before computing forces:

   // Only compute forces for particles in current domain
   bool in_domain = (p.x >= min_x && p.x < max_x && 
                     p.y >= min_y && p.y < max_y && 
                     p.z >= min_z && p.z < max_z);
   
   if(in_domain) {
       // Compute forces and spread to fluid grid
   }

3. Synchronization: Add a new communicate_particles() function to synchronize particle positions across all GPU domains after each time step.

4. Transfer Kernels: Implement transfer_extract_particles and transfer_insert_particles kernels to handle full particle list transfers between domains.

5. Memory Management: Modify memory allocation so each LBM_Domain allocates space for the complete particle set rather than a partitioned subset.

6. Remove Limitation: Eliminate the check that prevents particles from running in multi-GPU mode.

Advantages:

• Simpler implementation without complex particle migration logic
• Ensures all GPUs have complete particle information
• Minimal changes to existing code structure
• Maintains correctness of particle-fluid interactions

Trade-offs:

• Increased memory usage (D times more, where D = number of GPUs)
• Communication overhead for synchronizing full particle lists
• Computational redundancy (all GPUs compute all particle movements, but only apply forces for local particles)

Would this approach be acceptable for enabling multi-GPU particle support in FluidX3D? Are there any concerns with this strategy or suggestions for improvement?

[ProjectPhysX]

Hi @jasonxauat,

that approach sounds very reasonable. The memory overhead isn't even that bad - it could happen that all particles move from one domain into another, and then no memory reallocation is required at all, which is a big plus. Each domain therefore must have enough free VRAM to hold all particles, so can/should allocate the entire particle buffer in the first place.

I'll give the implementation a shot, have another week of vacation in 2 weeks.

Thanks a lot for sharing your ideas!

Kind regards,
Moritz

[jasonxauat]

Thank you for your approval. With the help of qwen3-coder, I have tried to modify the code in recent days, which can be basically implemented. However, due to the existence of host and device domain array data reading and writing, the efficiency is still not very high, even with the offset read-and-write function(enqueue_read_from_device & enqueue_write_to_device).
The difficulty here is that at least one array needs to be synchronized, whose value is used to mark the updated particles.

Is there any way to achieve single particle fast reading and writing? eg. only exchange data on GPUs? Could you see my code？

src.zip

20250805_bugfixed.zip

[pk-guest]

Hi Jason,

did you implemented immersed boundary for moving/rotation parts too? Is it better compare to revoxeling, or just a philosophy different?

regards
Peter

[jasonxauat]

Hi Jason,

did you implemented immersed boundary for moving/rotation parts too? Is it better compare to revoxeling, or just a philosophy different?

regards Peter

Hi Peter,

I have tried implementing the immersed boundary method by treating particles as nodes. Since the particle functionality is inherently based on the immersed boundary approach, it involves calculating particle forces and updating their positions. When using the immersed boundary for walls, it's necessary to appropriately modify the kernel to control the particle positions. So far, I've only experimented with stationary nodes, but moving or rotating nodes should also be feasible with additional control mechanisms.

There have been numerous studies comparing IB with revoxeling (bounce-back), such as C. Shu's or my previous paper. Simply, IB is more suitable for handling surfaces and moving boundaries, but the accuracy requires additional correction. Here, fluidx3d, the classical explicit diffused IB method is used, and additional processing is required to meet the condition of no wall slip, eg.

Wu, J. and C. Shu, Implicit velocity correction-based immersed boundary-lattice Boltzmann method and its applications. Journal of Computational Physics, 2009. 228(6): p. 1963-1979.
Kang, S.K. and Y.A. Hassan, A comparative study of direct-forcing immersed boundary-lattice Boltzmann methods for stationary complex boundaries. International Journal for Numerical Methods in Fluids, 2011. 66(9): p. 1132-1158.

Best regards,
Jason

[pk-guest]

Hello Jason,,

thx for your feedback.
I tought you implemented the IBM for geometry motion. For particle is also interesting special if you want to model complex shapes. However DEM kernel is at the moment missing. But coupling with external DEM tool could be a good feature.

Peter

[jasonxauat]

Hi Peter,

I completely agree with the DEM coupling idea. Since the current setup lacks a built-in DEM kernel, linking with an external DEM tool makes perfect sense. However, it's crucial to find a suitable DEM tool based on OpenCL; otherwise, FluidX3D would lose its advantage of high computational efficiency.

Best regards,
Jason

[pk-guest]

To couple it with external DEM tool is the easiest part. The tricky part is the solver from CFD and maybe the communication with huge number of particles, but until 10M should be fine.
I dont know any good open source DEM tool with high performance GPU solver at the moment. Only commertial tools have usable GPU solvers. Most of them are CUDA based, but the communication is running via CPU which slow down a little bit. So its possible to couple two tools with maybe some disadvatage compare to one tool with internal DEM. But I dont know if somebody started to implement a DEM kernel for fluidX3D

[jasonxauat]

Hi Peter,

High-performance open-source GPU DEM tools are rare too—most are commercial, CUDA-based, and (as you said) have CPU communication lags. So finding OpenCL-based open-source DEM code is key to keeping FluidX3D’s efficiency; if not, we’ll need a custom DEM kernel.
Thanks for the insights—they help refine our direction.

Best regards,
Jason

[jasonxauat]

To couple it with external DEM tool is the easiest part. The tricky part is the solver from CFD and maybe the communication with huge number of particles, but until 10M should be fine. I dont know any good open source DEM tool with high performance GPU solver at the moment. Only commertial tools have usable GPU solvers. Most of them are CUDA based, but the communication is running via CPU which slow down a little bit. So its possible to couple two tools with maybe some disadvatage compare to one tool with internal DEM. But I dont know if somebody started to implement a DEM kernel for fluidX3D

Hi Peter, @pk-guest

I've also been looking into open-source DEM tools that could potentially be coupled with FluidX3D. The main method is to use DEM software to calculate the motion of particles. Fluidx3D can obtain the hydraulic resistance experienced by particles, solve for particle collisions and motion in DEM, obtain new coordinates, and return them to Fluidx3D.
Do you have any more efficient suggestions?

Here are some options:

• LIGGGHTS
• YADE
• ...

If you have any other questions or need further information, please let me know.

Best regards,
Jason

[pk-guest]

Hi Jason, I am not up to date in opensource DEM softwares, but most of them has no native GPU solver yet, or they have a lot of other limitation. Liggghts has no GPU solver yet. But I have to check the current situation to make a correct view. Only one/two commertial DEM tools are good. EDEM and rocky. To make a coupling to EDEM is not so difficult, because there is a coupling interface, but than you have a hybrid approach, a mix of open source and commertial code. But advantage of full GPU solvers. How much particle do you want to handle with the DEM solver? Coupling will communicate via CPU which will slow a little bit the solvers. Peter jasonxauat ***@***.***> schrieb am Di., 16. Sept. 2025, 11:21:

…

*jasonxauat* left a comment (ProjectPhysX/FluidX3D#306) <#306 (comment)> To couple it with external DEM tool is the easiest part. The tricky part is the solver from CFD and maybe the communication with huge number of particles, but until 10M should be fine. I dont know any good open source DEM tool with high performance GPU solver at the moment. Only commertial tools have usable GPU solvers. Most of them are CUDA based, but the communication is running via CPU which slow down a little bit. So its possible to couple two tools with maybe some disadvatage compare to one tool with internal DEM. But I dont know if somebody started to implement a DEM kernel for fluidX3D Hi Peter, @pk-guest <https://github.com/pk-guest> I've also been looking into open-source DEM tools that could potentially be coupled with FluidX3D. The main method is to use DEM software to calculate the motion of particles. Fluidx3D can obtain the hydraulic resistance experienced by particles, solve for particle collisions and motion in DEM, obtain new coordinates, and return them to Fluidx3D. Do you have any more efficient suggestions? Here are some options: - LIGGGHTS - YADE - ... If you have any other questions or need further information, please let me know. Best regards, Jason — Reply to this email directly, view it on GitHub <#306 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BJ2YNNAHIXBQ5YWPNSNEF3D3S7JCJAVCNFSM6AAAAACCSTMOUCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTEOJWHE3TSNBUGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[jasonxauat]

Hi Peter,
It may be necessary to simulate less than 10 million particles for CFD coupling. It is best to compile and run it together with fluidx3d to reduce communication costs. I have gained some understanding of common DEM open source software, such as YADE and LIGGGHTS. It seems to have been run directly with commands, but I haven't found a suitable way to couple the code yet.

Best regards,
Jason

[pk-guest]

Hi, I never used open source, but liggghts should have the capability for coupling. I can check it, give me a little bit time. 10M particle is ok, still not too much. Do you want simulate high particle density, so dense phase simulation, where the particles have also influence on air? Or just low particle density where only the air has influence on particles? Regards Peter jasonxauat ***@***.***> schrieb am Mi., 17. Sept. 2025, 09:11:

…

*jasonxauat* left a comment (ProjectPhysX/FluidX3D#306) <#306 (comment)> Hi Peter, It may be necessary to simulate less than 10 million particles for CFD coupling. It is best to compile and run it together with fluidx3d to reduce communication costs. I have gained some understanding of common DEM open source software, such as YADE and LIGGGHTS. It seems to have been run directly with commands, but I haven't found a suitable way to couple the code yet. Best regards, Jason — Reply to this email directly, view it on GitHub <#306 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BJ2YNND5EEUEBKSJB6XC6D33TECRJAVCNFSM6AAAAACCSTMOUCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTGMBRGYYTGMJVGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>