ClimaAtmos - Target GPU simulations

Public

Monitor performance of target GPU simulations

Compiling update for Sprint meeting

#392
/
main/76a35c69
Passed in 1h 7m
buildkite-agent pipeline upload .buildkite/gpu_...
init :GPU:
Sriharsha Kandala
Sriharsha Kandala
Created
Triggered from Web
buildkite-agent pipeline upload .buildkite/gpu_...buildkite-agent pipeline upload .buildkite/gpu_pipeline/pipeline.yml
Waited 1m 4s
·
Ran in 21s
Agent
init :GPU:echo "--- Instantiate examples" && julia --project=examples -e 'using Pkg; Pkg.instantiate(;verbose=true)' && julia --project=examples -e 'using Pkg; Pkg.precompile()' && julia --project=examples -e 'using CUDA; CUDA.precompile_runtime()' && julia --project=examples -e 'using Pkg; Pkg.status()' && echo "--- Instantiate perf" && julia --project=perf -e 'using Pkg; Pkg.instantiate(;verbose=true)' && julia --project=perf -e 'using Pkg; Pkg.precompile()' && julia --project=perf -e 'using CUDA; CUDA.precompile_runtime()' && julia --project=perf -e 'using Pkg; Pkg.status()' && echo "--- Download artifacts" && julia --project=examples artifacts/download_artifacts.jl
Waited 33s
·
Ran in 8m 43s
Agent
dry baroclinic wavemkdir -p target_gpu_implicit_baroclinic_wave && nsys profile --delay 100 --trace=nvtx,mpi,cuda,osrt --output=target_gpu_implicit_baroclinic_wave/output_active/report julia --threads=3 --color=yes --project=examples examples/hybrid/driver.jl --config_file config/gpu_configs/target_gpu_implicit_baroclinic_wave.yml --job_id target_gpu_implicit_baroclinic_wave
Waited 29s
·
Ran in 7m 46s
Agent
moist Held-Suarezmkdir -p gpu_hs_rhoe_equil_55km_nz63_0M && nsys profile --delay 100 --trace=nvtx,mpi,cuda,osrt --output=gpu_hs_rhoe_equil_55km_nz63_0M/output_active/report julia --threads=3 --color=yes --project=examples examples/hybrid/driver.jl --config_file config/gpu_configs/gpu_hs_rhoe_equil_0M.yml --job_id gpu_hs_rhoe_equil_55km_nz63_0M
Waited 22s
·
Ran in 9m 5s
Agent
moist Held-Suarez - 4 gpusmkdir -p gpu_hs_rhoe_equil_55km_nz63_0M_4process && srun --cpu-bind=threads --cpus-per-task=4 nsys profile --delay 100 --trace=nvtx,mpi,cuda,osrt --output=gpu_hs_rhoe_equil_55km_nz63_0M_4process/output_active/report-%q{PMI_RANK} julia --threads=3 --color=yes --project=examples examples/hybrid/driver.jl --config_file config/gpu_configs/gpu_hs_rhoe_equil_0M.yml --job_id gpu_hs_rhoe_equil_55km_nz63_0M_4process
Waited 24s
·
Ran in 8m 52s
Agent
dry baroclinic wave - 4 gpusmkdir -p target_gpu_implicit_baroclinic_wave_4process && srun --cpu-bind=threads --cpus-per-task=4 nsys profile --delay 100 --trace=osrt,nvtx,cuda,mpi,ucx --output=target_gpu_implicit_baroclinic_wave_4process/output_active/report-%q{PMI_RANK} julia --threads=3 --color=yes --project=examples examples/hybrid/driver.jl --config_file config/gpu_configs/target_gpu_implicit_baroclinic_wave.yml --job_id target_gpu_implicit_baroclinic_wave_4process
Waited 9m 18s
·
Ran in 5m 29s
Agent
gpu_aquaplanet_dyamond with diagnostics - strong scaling - 1 GPUmkdir -p gpu_aquaplanet_dyamond_diag_1process && srun --cpu-bind=threads --cpus-per-task=4 nsys profile --delay 100 --trace=nvtx,mpi,cuda,osrt --output=gpu_aquaplanet_dyamond_diag_1process/output_active/report julia --threads=3 --color=yes --project=examples examples/hybrid/driver.jl --config_file config/gpu_configs/gpu_aquaplanet_dyamond_diag_1process.yml --job_id gpu_aquaplanet_dyamond_diag_1process
Waited 9m 21s
·
Ran in 16m 9s
Agent
gpu_aquaplanet_dyamond - strong scaling - 1 GPUmkdir -p gpu_aquaplanet_dyamond_ss_1process && srun --cpu-bind=threads --cpus-per-task=4 nsys profile --delay 100 --trace=nvtx,mpi,cuda,osrt --output=gpu_aquaplanet_dyamond_ss_1process/output_active/report julia --threads=3 --color=yes --project=examples examples/hybrid/driver.jl --config_file config/gpu_configs/gpu_aquaplanet_dyamond_ss.yml --job_id gpu_aquaplanet_dyamond_ss_1process
Waited 9m 21s
·
Ran in 15m 11s
Agent
gpu_aquaplanet_dyamond - strong scaling - 2 GPUsmkdir -p gpu_aquaplanet_dyamond_ss_2process && srun --cpu-bind=threads --cpus-per-task=4 julia --threads=3 --color=yes --project=examples examples/hybrid/driver.jl --config_file config/gpu_configs/gpu_aquaplanet_dyamond_ss.yml --job_id gpu_aquaplanet_dyamond_ss_2process
Waited 14m 49s
·
Ran in 9m 57s
Agent
gpu_aquaplanet_dyamond - strong scaling - 4 GPUsmkdir -p gpu_aquaplanet_dyamond_ss_4process && srun --cpu-bind=threads --cpus-per-task=4 julia --threads=3 --color=yes --project=examples examples/hybrid/driver.jl --config_file config/gpu_configs/gpu_aquaplanet_dyamond_ss.yml --job_id gpu_aquaplanet_dyamond_ss_4process
Waited 25m 32s
·
Ran in 8m 51s
Agent
gpu_aquaplanet_dyamond - strong scaling plotsmkdir -p gpu_aquaplanet_dyamond_ss && julia --color=yes --project=examples post_processing/plot_gpu_strong_scaling.jl gpu_aquaplanet_dyamond_ss
Waited 2m 9s
·
Ran in 15s
Agent
gpu_aquaplanet_dyamond - weak scaling - 1 GPUmkdir -p gpu_aquaplanet_dyamond_ws_1process && srun --cpu-bind=threads --cpus-per-task=4 julia --threads=3 --color=yes --project=examples examples/hybrid/driver.jl --config_file config/gpu_configs/gpu_aquaplanet_dyamond_ws_1process.yml --job_id gpu_aquaplanet_dyamond_ws_1process
Waited 27m 8s
·
Ran in 12m 17s
Agent
gpu_aquaplanet_dyamond - weak scaling - 2 GPUsmkdir -p gpu_aquaplanet_dyamond_ws_2process && srun --cpu-bind=threads --cpus-per-task=4 julia --threads=3 --color=yes --project=examples examples/hybrid/driver.jl --config_file config/gpu_configs/gpu_aquaplanet_dyamond_ws_2process.yml --job_id gpu_aquaplanet_dyamond_ws_2process
Waited 34m 25s
·
Ran in 12m 7s
Agent
gpu_aquaplanet_dyamond - weak scaling - 4 GPUsmkdir -p gpu_aquaplanet_dyamond_ws_4process && srun --cpu-bind=threads --cpus-per-task=4 julia --threads=3 --color=yes --project=examples examples/hybrid/driver.jl --config_file config/gpu_configs/gpu_aquaplanet_dyamond_ws_4process.yml --job_id gpu_aquaplanet_dyamond_ws_4process
Waited 36m 28s
·
Ran in 12m 31s
Agent
gpu_aquaplanet_dyamond - weak scaling plotsmkdir -p gpu_aquaplanet_dyamond_ws && julia --color=yes --project=examples post_processing/plot_gpu_weak_scaling.jl gpu_aquaplanet_dyamond_ws
Waited 14s
·
Ran in 16s
Agent
gpu_aquaplanet_diagedmf - 1 GPUmkdir -p gpu_aquaplanet_diagedmf && nsys profile --delay 200 --trace=nvtx,mpi,cuda,osrt --output=gpu_aquaplanet_diagedmf/output_active/report julia --threads=3 --color=yes --project=examples examples/hybrid/driver.jl --config_file config/model_configs/aquaplanet_diagedmf.yml --job_id gpu_aquaplanet_diagedmf
Waited 15m 3s
·
Ran in 12m 4s
Agent
gpu_aquaplanet_diagedmf_benchmarkjulia --color=yes --project=perf perf/benchmark.jl --config_file config/model_configs/aquaplanet_diagedmf.yml --job_id gpu_aquaplanet_diagedmf_benchmark
Waited 15m 2s
·
Ran in 15m 4s
Agent
gpu_aquaplanet_progedmf - 1 GPUmkdir -p gpu_aquaplanet_progedmf && nsys profile --delay 100 --trace=nvtx,mpi,cuda,osrt --output=gpu_aquaplanet_progedmf/output_active/report julia --threads=3 --color=yes --project=examples examples/hybrid/driver.jl --config_file config/model_configs/aquaplanet_progedmf.yml --job_id gpu_aquaplanet_progedmf
Waited 15m 2s
·
Ran in 21m 24s
Agent
gpu_aquaplanet_progedmf_benchmarkjulia --color=yes --project=perf perf/benchmark.jl --config_file config/model_configs/aquaplanet_progedmf.yml --job_id gpu_aquaplanet_progedmf_benchmark
Waited 39m 27s
·
Ran in 18m 0s
Agent
Total Job Run Time: 3h 14m