Emitters and particle injectors need to move particles

[jjaraalm]

In some use cases such as open boundaries, emitters and particle injectors are used to inject a net current in addition to particle flux. To do this properly, injected particles must be moved off the cell face and accumulated. Without accumulation, the local current is inconsistent with curl(B) and leads to very incorrect results. Injected particles may also leave the local domain, and this should be allowed as well. Moving particles from emitters and injectors is difficult right now because of the way device/host copies are staged.

Device-side emission

To support device-side emission, transfer from sp->k_pc_d to sp->k_pc_h

vpic-kokkos/src/species_advance/standard/advance_p.cc

Lines 336 to 358 in fba1997

	KOKKOS_TIC();
	// I need to know the number of movers that got populated so I can call the
	// compress. Let's copy it back
	Kokkos::deep_copy(sp->k_nm_h, sp->k_nm_d);
	// TODO: which way round should this copy be?
	// int nm = sp->k_nm_h(0);
	// printf("nm = %d \n", nm);
	// Copy particle mirror movers back so we have their data safe. Ready for
	// boundary_p_kokkos
	auto pc_d_subview = Kokkos::subview(sp->k_pc_d, std::make_pair(0, sp->k_nm_h(0)), Kokkos::ALL);
	auto pci_d_subview = Kokkos::subview(sp->k_pc_i_d, std::make_pair(0, sp->k_nm_h(0)));
	auto pc_h_subview = Kokkos::subview(sp->k_pc_h, std::make_pair(0, sp->k_nm_h(0)), Kokkos::ALL);
	auto pci_h_subview = Kokkos::subview(sp->k_pc_i_h, std::make_pair(0, sp->k_nm_h(0)));
	Kokkos::deep_copy(pc_h_subview, pc_d_subview);
	Kokkos::deep_copy(pci_h_subview, pci_d_subview);
	// Kokkos::deep_copy(sp->k_pc_h, sp->k_pc_d);
	// Kokkos::deep_copy(sp->k_pc_i_h, sp->k_pc_i_d);
	KOKKOS_TOC( PARTICLE_DATA_MOVEMENT, 1);

must occur after apply_emitter_list() and user_particle_injection(). Also, since emission/injection typically require random numbers, #58 would be helpful.

Host-side emission

Support for host-side emission is far more complicated. At a minimum any support requires

1. Instead of copying the whole particle array, logic should be added around apply_emitter_list() and user_particle_injection() to monitor sp->np and sp->nm for changes. New particles and movers would be copied to the device. Emission may generate movers, and these need to be on the device for the compressor to work correctly.
2. Copy interpolator memory to host before user_particle_injection() if kokkos_particle_injection == false. Emitters should be responsible for doing this themselves in order to support both device and host-side emitters, but this requires Cleanup copy to/from device/host methods #50 since emitters do not have access to the copy methods currently.
3. Host-side accumulators must be cleared before apply_emitter_list() and user_particle_injection() and not after

   vpic-kokkos/src/vpic/advance.cc

   Lines 91 to 123 in fba1997

   	if( emitter_list )
   	{
   	TIC apply_emitter_list( emitter_list ); TOC( emission_model, 1 );
   	}
   	if((particle_injection_interval>0) && ((step() % particle_injection_interval)==0)) {
   	if(!kokkos_particle_injection) {
   	KOKKOS_TIC();
   	KOKKOS_COPY_PARTICLE_MEM_TO_HOST(species_list);
   	KOKKOS_TOC(PARTICLE_DATA_MOVEMENT, 1);
   	}
   	TIC user_particle_injection(); TOC( user_particle_injection, 1 );
   	if(!kokkos_particle_injection) {
   	KOKKOS_TIC();
   	KOKKOS_COPY_PARTICLE_MEM_TO_DEVICE(species_list);
   	KOKKOS_TOC(PARTICLE_DATA_MOVEMENT, 1);
   	}
   	}
   	bool accumulate_in_place = false; // This has to be outside the scoped timing block
   	KOKKOS_TIC(); // Time this data movement
   	// This could technically be done once per simulation, not every timestep
   	if (accumulator_array->k_a_h.data() == accumulator_array->k_a_d.data() )
   	{
   	accumulate_in_place = true;
   	}
   	else {
   	// Zero out the host accumulator
   	Kokkos::deep_copy(accumulator_array->k_a_h, 0.0f);
   	}
   	KOKKOS_TOC( ACCUMULATOR_DATA_MOVEMENT, 1);

4. One of the following:
   • Allow accumulation into accumulator_array->k_a_h from move_p() (the host-side version).
   • clear_accumulator_array() must be called every step in the advance loop (next to the deep_copy above) and combine_accumulators() must first combine a reduced accumulator_array->a with accumulator_array->k_a_h before copying to device.

I think this is everything required for host-side emission to work, but there might be things I missed.