CM1 r21.1 GPU/OpenACC support on local NVIDIA workstation

[tpeterson]

Hi all,

I’m trying to understand the current state of GPU support in CM1.

I’m running CM1 on a desktop tower with 16 CPU cores and an NVIDIA GeForce RTX 3060 Ti. My normal target is cm1r21.1.

From what I can tell:

- The public 21.1 tag is cm1r21.1.
- The public gpu-opt branch appears to be cm1r20.2.
- The r21.1/current Makefile has NVHPC / USE_OPENACC options.
- The gpu-opt branch appears to have more substantial OpenACC/GPU work, but it is older and not directly compatible with my r21.1 namelist without changes.

What I’m trying to figure out:

1. Is there a current r21.x GPU/OpenACC branch that I should be using?
2. Is the gpu-opt branch still the recommended GPU path, even though it appears to be r20.2-era?
3. If I compile r21.1 with NVHPC and USE_OPENACC=true, should I expect meaningful GPU offload, or is that not really supported for r21.1?
4. Are there specific CM1 test cases that are known to work well on GPU?
5. What is the best way to verify real GPU use: -Minfo=accel, NV_ACC_TIME=1, nvidia-smi, Nsight, or something else?

For now I can run r21.1 on CPU/OpenMP, but I’d like to understand whether using the 3060 Ti is realistic or whether I should treat GPU support as experimental.

Thanks,
Tim

 

[ahijevyc]

Good questions, Tim.
I am curious about these myself.
You are right about the GPU development being based on an older branch. The current branch's Makefile mentions OPENACC and GPU_TYPE, but I don't think their use would result in any offloading onto the GPU. The current FORTRAN source doesn't have any acc directives, as far as I can tell.
I will do some more digging see if there are any specific test cases that are known to work well on GPU, or if there are updated time estimates for getting the gpu-opt branch merged into main.
Unfortunately, The GPU support in the older branch is still experimental, and I don't know much about -Minfo=accel, NV_ACC_TIME=1, or Nsight. I do use nvidia-smi all the time for unrelated machine learning GPU work to check VRAM usage, however.