cutoff-solver: rocketrig crashes when calling particle_comm->migrate(_comm, particle_array) in Migrator.hpp when run on Hopper with CUDA

[JStewart28]

Declaration:
auto particle_comm = Cabana::Grid::createGlobalParticleComm<memory_space>(*_spm.localGrid());

migrate is a built-in Cabana function.

[JStewart28]

Stack trace:

• It happens during the creation of the Cabana::Distributor<memory_space> object in Cabana_Grid_GlobalParticleComm.hpp (Line 287)
• In line 140 of Cabana::Distributer.hpp: this->createExportSteering( neighbor_ids, element_export_ranks );
• In line 946 of Cabana_CommunicationPlan.hpp: createSteering( true, neighbor_ids, element_export_ranks, element_export_ranks );
• The error occurs in the Kokkos parallel for loop starting on line 1019 of Cabana_Communication.hpp:
  Kokkos::parallel_for( "Cabana::createSteering", Kokkos::RangePolicy<ExecutionSpace>( 0, _num_export_element ), KOKKOS_LAMBDA( const int i ) { if ( element_export_ranks( i ) >= 0 ) steer_vec( rank_offsets( element_export_ranks( i ) ) + neighbor_ids( i ) ) = ( use_iota ) ? i : element_export_ids( i ); } );

[patrickb314]

Let's put this on our list to track down.

[JStewart28]

Currently seg faulting at the initialization of solver with a cuda backend, for both the condo and cup-ecs nodes