Monitor performance of target GPU simulations

Running for Sprint meeting

Failed in 1h 46m
buildkite-agent pipeline upload .buildkite/gpu_...
init :GPU:
moist Held-Suarez - 4 gpusmkdir -p gpu_hs_rhoe_equil_55km_nz63_0M_4process && srun --cpu-bind=threads --cpus-per-task=4 nsys profile --delay 100 --trace=nvtx,mpi,cuda,osrt --output=gpu_hs_rhoe_equil_55km_nz63_0M_4process/output_active/report-%q{PMI_RANK} julia --threads=3 --color=yes --project=.buildkite examples/hybrid/driver.jl --config_file config/gpu_configs/gpu_hs_rhoe_equil_0M.yml --job_id gpu_hs_rhoe_equil_55km_nz63_0M_4process
Waited 51s
·
Ran in 8m 4s
moist Held-Suarez - 4 gpusmkdir -p gpu_hs_rhoe_equil_55km_nz63_0M_4process && srun --cpu-bind=threads --cpus-per-task=4 nsys profile --delay 100 --trace=nvtx,mpi,cuda,osrt --output=gpu_hs_rhoe_equil_55km_nz63_0M_4process/output_active/report-%q{PMI_RANK} julia --threads=3 --color=yes --project=.buildkite examples/hybrid/driver.jl --config_file config/gpu_configs/gpu_hs_rhoe_equil_0M.yml --job_id gpu_hs_rhoe_equil_55km_nz63_0M_4process
Waited 47m 29s
·
Ran in 8m 52s
moist Held-Suarez - 4 gpusmkdir -p gpu_hs_rhoe_equil_55km_nz63_0M_4process && srun --cpu-bind=threads --cpus-per-task=4 nsys profile --delay 100 --trace=nvtx,mpi,cuda,osrt --output=gpu_hs_rhoe_equil_55km_nz63_0M_4process/output_active/report-%q{PMI_RANK} julia --threads=3 --color=yes --project=.buildkite examples/hybrid/driver.jl --config_file config/gpu_configs/gpu_hs_rhoe_equil_0M.yml --job_id gpu_hs_rhoe_equil_55km_nz63_0M_4process
Waited 22s
·
Ran in 7m 26s
moist Held-Suarez - 4 gpusmkdir -p gpu_hs_rhoe_equil_55km_nz63_0M_4process && srun --cpu-bind=threads --cpus-per-task=4 nsys profile --delay 100 --trace=nvtx,mpi,cuda,osrt --output=gpu_hs_rhoe_equil_55km_nz63_0M_4process/output_active/report-%q{PMI_RANK} julia --threads=3 --color=yes --project=.buildkite examples/hybrid/driver.jl --config_file config/gpu_configs/gpu_hs_rhoe_equil_0M.yml --job_id gpu_hs_rhoe_equil_55km_nz63_0M_4process
Waited 22s
·
Ran in 7m 49s
moist Held-Suarez - 4 gpusmkdir -p gpu_hs_rhoe_equil_55km_nz63_0M_4process && srun --cpu-bind=threads --cpus-per-task=4 nsys profile --delay 100 --trace=nvtx,mpi,cuda,osrt --output=gpu_hs_rhoe_equil_55km_nz63_0M_4process/output_active/report-%q{PMI_RANK} julia --threads=3 --color=yes --project=.buildkite examples/hybrid/driver.jl --config_file config/gpu_configs/gpu_hs_rhoe_equil_0M.yml --job_id gpu_hs_rhoe_equil_55km_nz63_0M_4process
Waited 42s
·
Ran in 7m 17s
moist Held-Suarez - 4 gpusmkdir -p gpu_hs_rhoe_equil_55km_nz63_0M_4process && srun --cpu-bind=threads --cpus-per-task=4 nsys profile --delay 100 --trace=nvtx,mpi,cuda,osrt --output=gpu_hs_rhoe_equil_55km_nz63_0M_4process/output_active/report-%q{PMI_RANK} julia --threads=3 --color=yes --project=.buildkite examples/hybrid/driver.jl --config_file config/gpu_configs/gpu_hs_rhoe_equil_0M.yml --job_id gpu_hs_rhoe_equil_55km_nz63_0M_4process
Waited 53s
·
Ran in 7m 20s
dry baroclinic wave - 4 gpusmkdir -p target_gpu_implicit_baroclinic_wave_4process && srun --cpu-bind=threads --cpus-per-task=4 nsys profile --delay 100 --trace=osrt,nvtx,cuda,mpi,ucx --output=target_gpu_implicit_baroclinic_wave_4process/output_active/report-%q{PMI_RANK} julia --threads=3 --color=yes --project=.buildkite examples/hybrid/driver.jl --config_file config/gpu_configs/target_gpu_implicit_baroclinic_wave.yml --job_id target_gpu_implicit_baroclinic_wave_4process
Waited 8m 59s
·
Ran in 6m 58s
gpu_aquaplanet_diagedmf - 1 GPUmkdir -p gpu_aquaplanet_diagedmf && nsys profile --delay 200 --trace=nvtx,mpi,cuda,osrt --output=gpu_aquaplanet_diagedmf/output_active/report julia --threads=3 --color=yes --project=.buildkite examples/hybrid/driver.jl --config_file config/model_configs/aquaplanet_diagedmf.yml --job_id gpu_aquaplanet_diagedmf
Waited 16m 21s
·
Ran in 9m 43s
Total Job Run Time: 4h 18m