[Issue]: Do concurrent minimum reductions fails with GPU offload

[Lauritskyst]

Problem Description

When offloading a do concurrent to the gpu with a minimum reduce, the initial value of the reduction variable is kept after the do concurrent. This is not the case when I don't offload.

The code (see below) was compiled with (using a dowload of the newest amdfortran)
./rocm-afar-10004-drop-22.3.0/bin/amdflang -fopenmp --offload-arch=gfx942 -fdo-concurrent-to-openmp=device do_con.f90 -o do_con

And with (using the rocm installation already on the AMD GPU cloud from digitalocean)
/opt/rocm/bin/amdflang -fopenmp --offload-arch=gfx942 -
fdo-concurrent-to-openmp=device do_con.f90 -o do_con

I get

min_dt after reduction = 100.
Expected = 25.0

But should get (as I do when I don't offload):

min_dt after reduction = 25.
Expected = 25.0

Operating System

Ubuntu 24.04.4 LTS

CPU

INTEL XEON PLATINIUM 8568Y+

GPU

8568Y+

ROCm Version

rocm-afar-10004-drop-22.3.0

ROCm Component

flang, aomp

Steps to Reproduce

program test_reduce
   !
   implicit none
   !
   integer, parameter :: n = 10
   real*4             :: min_dt
   real*4             :: arr(n)
   integer            :: i
   !
   ! Initialize min_dt and transfer to GPU
   !
   min_dt = 100.0
   !
   ! Set up array with values both larger and smaller than min_dt
   !
   arr = (/ 200.0, 150.0, 80.0, 50.0, 300.0, 25.0, 175.0, 60.0, 400.0, 90.0 /)
   !
   ! Expected result: min_dt = 25.0 (the smallest value in arr)
   !
   !$omp target enter data map(to: min_dt, arr)
   !
   !$omp target update to(min_dt)
   !
   do concurrent (i = 1:n) reduce(min:min_dt)
      min_dt = min(min_dt, arr(i))
   enddo
   !
   !$omp target update from(min_dt)
   !
   !$omp target exit data map(delete: min_dt, arr)
   !
   print *, 'min_dt after reduction = ', min_dt
   print *, 'Expected               =  25.0'
   !
end program test_reduce

(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support

ROCk module version 6.16.13 is loaded

HSA System Attributes

Runtime Version: 1.18
Runtime Ext Version: 1.15
System Timestamp Freq.: 1000.000000MHz
Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
Machine Model: LARGE
System Endianness: LITTLE
Mwaitx: DISABLED
XNACK enabled: NO
DMAbuf Support: YES
VMM Support: YES

==========
HSA Agents

---

Agent 1

---

Name: INTEL(R) XEON(R) PLATINUM 8568Y+
Uuid: CPU-XX
Marketing Name: INTEL(R) XEON(R) PLATINUM 8568Y+
Vendor Name: CPU
Feature: None specified
Profile: FULL_PROFILE
Float Round Mode: NEAR
Max Queue Number: 0(0x0)
Queue Min Size: 0(0x0)
Queue Max Size: 0(0x0)
Queue Type: MULTI
Node: 0
Device Type: CPU
Cache Info:
L1: 32768(0x8000) KB
Chip ID: 0(0x0)
ASIC Revision: 0(0x0)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 0
BDFID: 0
Internal Node ID: 0
Compute Unit: 20
SIMDs per CU: 0
Shader Engines: 0
Shader Arrs. per Eng.: 0
WatchPts on Addr. Ranges:1
Memory Properties:
Features: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: FINE GRAINED
Size: 247409300(0xebf2a94) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 2
Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED
Size: 247409300(0xebf2a94) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 3
Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED
Size: 247409300(0xebf2a94) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 4
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 247409300(0xebf2a94) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
ISA Info:

---

Agent 2

---

Name: gfx942
Uuid: GPU-3a4eb06774917936
Marketing Name: AMD Instinct MI300X VF
Vendor Name: AMD
Feature: KERNEL_DISPATCH
Profile: BASE_PROFILE
Float Round Mode: NEAR
Max Queue Number: 128(0x80)
Queue Min Size: 64(0x40)
Queue Max Size: 131072(0x20000)
Queue Type: MULTI
Node: 1
Device Type: GPU
Cache Info:
L1: 32(0x20) KB
L2: 4096(0x1000) KB
L3: 262144(0x40000) KB
Chip ID: 29877(0x74b5)
ASIC Revision: 1(0x1)
Cacheline Size: 128(0x80)
Max Clock Freq. (MHz): 2100
BDFID: 33536
Internal Node ID: 1
Compute Unit: 304
SIMDs per CU: 4
Shader Engines: 32
Shader Arrs. per Eng.: 1
WatchPts on Addr. Ranges:4
Coherent Host Access: FALSE
Memory Properties:
Features: KERNEL_DISPATCH
Fast F16 Operation: TRUE
Wavefront Size: 64(0x40)
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension:
x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Max Waves Per CU: 32(0x20)
Max Work-item Per CU: 2048(0x800)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension:
x 2147483647(0x7fffffff)
y 65535(0xffff)
z 65535(0xffff)
Max fbarriers/Workgrp: 32
Packet Processor uCode:: 189
SDMA engine uCode:: 24
IOMMU Support:: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 200998912(0xbfb0000) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:2048KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 2
Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED
Size: 200998912(0xbfb0000) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:2048KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 3
Segment: GLOBAL; FLAGS: FINE GRAINED
Size: 200998912(0xbfb0000) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:2048KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 4
Segment: GROUP
Size: 64(0x40) KB
Allocatable: FALSE
Alloc Granule: 0KB
Alloc Recommended Granule:0KB
Alloc Alignment: 0KB
Accessible by all: FALSE
ISA Info:
ISA 1
Name: amdgcn-amd-amdhsa--gfx942:sramecc+:xnack-
Machine Models: HSA_MACHINE_MODEL_LARGE
Profiles: HSA_PROFILE_BASE
Default Rounding Mode: NEAR
Default Rounding Mode: NEAR
Fast f16: TRUE
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension:
x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension:
x 2147483647(0x7fffffff)
y 65535(0xffff)
z 65535(0xffff)
FBarrier Max Size: 32
ISA 2
Name: amdgcn-amd-amdhsa--gfx9-4-generic:sramecc+:xnack-
Machine Models: HSA_MACHINE_MODEL_LARGE
Profiles: HSA_PROFILE_BASE
Default Rounding Mode: NEAR
Default Rounding Mode: NEAR
Fast f16: TRUE
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension:
x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension:
x 2147483647(0x7fffffff)
y 65535(0xffff)
z 65535(0xffff)
FBarrier Max Size: 32
*** Done ***

Additional Information

No response

[zichguan-amd]

Hi @Lauritskyst, thanks for the report. I can repro the issue with all available flang tarballs from https://repo.radeon.com/rocm/misc/flang/.

Digging into the mlir after omp-do-concurrent-conversion, it shows that the omp.target block maps min_dt with capture(ByCopy):

%33 = omp.map.info var_ptr(%13#1 : !fir.ref<f32>, f32) map_clauses(implicit) capture(ByCopy) -> !fir.ref<f32> {name = "_QFEmin_dt"}

while target enter data and target update use capture(ByRef)

%19 = omp.map.info var_ptr(%13#1 : !fir.ref<f32>, f32) map_clauses(to) capture(ByRef) -> !fir.ref<f32> {name = "min_dt"}
%23 = omp.map.info var_ptr(%13#1 : !fir.ref<f32>, f32) map_clauses(to) capture(ByRef) -> !fir.ref<f32> {name = "min_dt"}
%35 = omp.map.info var_ptr(%13#1 : !fir.ref<f32>, f32) map_clauses(from) capture(ByRef) -> !fir.ref<f32> {name = "min_dt"}
%36 = omp.map.info var_ptr(%13#1 : !fir.ref<f32>, f32) map_clauses(delete, storage) capture(ByRef) -> !fir.ref<f32> {name = "min_dt"}

The reduction kernel is effectively updating a different device mapping than the one target update from(min_dt) reads from.

Do note that do concurrent mapping to OpenMP is still experimental, you can explicitly use a $omp target teams distribute parallel do block to get the right result.

I'll discuss with flang team to address the issue.

[Lauritskyst]

Thanks @zichguan-amd for the update!
In our code, we use quite a bit of do conccurent and this was the only problem we experienced.

[ergawy]

Thanks @Lauritskyst for reporting and thanks for @zichguan-amd for looking into this. I opened llvm#189378 to address the issue.