MPI.jl

Public

https://github.com/JuliaParallel/MPI.jl

set environment variables before dlopen (#847)

#1471

master/170fefa

Passed in 1h 35m

Pipeline upload

Valentin Churavy

Created Tue 16th Jul 2024 at 4:44 PM

Triggered from Webhook

Agent for job :pipeline::arrow_up: Upload Pipeline: .buildkite/pull_request_pipeline.yml (0195fcba-558f-4f1b-92ac-bedda6107d14): bk-agent-prod-k8s-1743701837734287925 (k8s) - [metrics] [events]

Agent for job :passport_control: License check (0195fcba-b483-42da-a2a5-5d25e045f616): bk-agent-prod-k8s-1743701855288969277 (k8s) - [metrics] [events]

Agent for job :rspec: Ruby unit tests (0195fcba-b484-497f-bfcf-92dee56a4838): bk-agent-prod-k8s-1743701855284006547 (k8s) - [metrics] [events]

Agent for job :lab_coat: Integration Tests / part 1-of-3 (0195fcba-b486-4b74-a9ca-0a4746618396): bk-agent-prod-k8s-1743701855391579205 (k8s) - [metrics] [events]

Agent for job :lab_coat: IT Persistent Queues / part 2-of-3 (0195fcba-b489-42f8-909c-d48ed999588e): bk-agent-prod-k8s-1743701855392414604 (k8s) - [metrics] [events]

Agent for job :lab_coat: IT Persistent Queues / part 1-of-3 (0195fcba-b488-471c-924b-5d06a44ad29f): bk-agent-prod-k8s-1743701855390595585 (k8s) - [metrics] [events]

Agent for job :java: Java unit tests (0195fcba-b484-48ad-89cc-bc8532598e15): bk-agent-prod-k8s-1743701855294244527 (k8s) - [metrics] [events]

Agent for job :lab_coat: Integration Tests / part 2-of-3 (0195fcba-b487-40bc-9ab9-c375b12475ca): bk-agent-prod-k8s-1743701855385696772 (k8s) - [metrics] [events]

Agent for job :lab_coat: x-pack integration (0195fcba-b48b-44a5-aad6-0caf4dfa8099): bk-agent-prod-k8s-1743701855394264792 (k8s) - [metrics] [events]

Agent for job :lab_coat: IT Persistent Queues / part 3-of-3 (0195fcba-b48a-419d-b3ee-15941661d462): bk-agent-prod-k8s-1743701855393586009 (k8s) - [metrics] [events]

Agent for job :lab_coat: x-pack unit tests (0195fcba-b48a-4a16-b551-1ec67c3268a9): bk-agent-prod-k8s-1743701855485873559 (k8s) - [metrics] [events]

Agent for job :lab_coat: Integration Tests / part 3-of-3 (0195fcba-b487-438b-89ef-00d70827046d): bk-agent-prod-k8s-1743701855682667532 (k8s) - [metrics] [events]

Agent for job :sonarqube: Continuous Code Inspection (0195fcba-b486-4094-a8a8-2a95f12413eb): bk-agent-prod-k8s-1743702643303617055 (k8s) - [metrics] [events]

Agent for job 🏁 Annotate JUnit results (0195fcba-b48c-414c-b0b0-cdab38bdd8d4): bk-agent-prod-gcp-1743703019182771625 (gcp) - [metrics] [events]

Pipeline uploadbuildkite-agent pipeline upload

Ran in 3s

Build OpenMPIecho "--- Install packages" && apt-get install --yes --no-install-recommends curl ccache && export PATH="/usr/lib/ccache/:$PATH" && echo "--- Build UCX" && curl -L https://github.com/openucx/ucx/releases/download/v${UCX_VER}/ucx-${UCX_VER}.tar.gz --output ucx.tar.gz && tar -zxf ucx.tar.gz && pushd ucx-* && ./configure --with-cuda=/usr/local/cuda --enable-mt --prefix=$(realpath ../mpi-prefix) && make -j && make install && popd && echo "--- Build OpenMPI" && curl -L https://download.open-mpi.org/release/open-mpi/v${OPENMPI_VER}/openmpi-${OPENMPI_VER_FULL}.tar.gz --output openmpi.tar.gz && tar -zxf openmpi.tar.gz && pushd openmpi-${OPENMPI_VER_FULL} && ./configure --with-ucx=$(realpath ../mpi-prefix) --with-cuda=/usr/local/cuda --prefix=$(realpath ../mpi-prefix) && make -j && make install && popd && echo "--- Package prefix" && tar -zcf mpi-prefix.tar.gz mpi-prefix/ && echo "--- ccache stats" && ccache -s

Ran in 5m 50s

Matrix

Tests -- Julia 1.6echo "--- Configure MPI" && buildkite-agent artifact download --step "cuda-build-openmpi" mpi-prefix.tar.gz . && mkdir -p ${JULIA_MPI_PATH} && tar -zxf mpi-prefix.tar.gz --strip-components 1 -C ${JULIA_MPI_PATH} && export PATH=${JULIA_MPI_PATH}/bin:${PATH} && export LD_LIBRARY_PATH=${JULIA_MPI_PATH}/lib:${LD_LIBRARY_PATH} && echo "--- Setup Julia packages" && julia --color=yes --project=. -e ' && import Pkg && Pkg.develop(; path = joinpath(pwd(), "lib", "MPIPreferences")) && ' && julia --color=yes --project=test -e ' && using Pkg && Pkg.develop(path="lib/MPIPreferences") && using MPIPreferences && MPIPreferences.use_system_binary(export_prefs=true) && rm("test/Manifest.toml") && ' && echo "+++ Run tests" && julia --color=yes --project=. -e ' && import Pkg && Pkg.test("MPI"; test_args=["--backend=CUDA"]) && '

Ran in 16m 52s

Matrix

Tests -- Julia 1.7echo "--- Configure MPI" && buildkite-agent artifact download --step "cuda-build-openmpi" mpi-prefix.tar.gz . && mkdir -p ${JULIA_MPI_PATH} && tar -zxf mpi-prefix.tar.gz --strip-components 1 -C ${JULIA_MPI_PATH} && export PATH=${JULIA_MPI_PATH}/bin:${PATH} && export LD_LIBRARY_PATH=${JULIA_MPI_PATH}/lib:${LD_LIBRARY_PATH} && echo "--- Setup Julia packages" && julia --color=yes --project=. -e ' && import Pkg && Pkg.develop(; path = joinpath(pwd(), "lib", "MPIPreferences")) && ' && julia --color=yes --project=test -e ' && using Pkg && Pkg.develop(path="lib/MPIPreferences") && using MPIPreferences && MPIPreferences.use_system_binary(export_prefs=true) && rm("test/Manifest.toml") && ' && echo "+++ Run tests" && julia --color=yes --project=. -e ' && import Pkg && Pkg.test("MPI"; test_args=["--backend=CUDA"]) && '

Ran in 17m 19s

Matrix

Tests -- Julia 1.8echo "--- Configure MPI" && buildkite-agent artifact download --step "cuda-build-openmpi" mpi-prefix.tar.gz . && mkdir -p ${JULIA_MPI_PATH} && tar -zxf mpi-prefix.tar.gz --strip-components 1 -C ${JULIA_MPI_PATH} && export PATH=${JULIA_MPI_PATH}/bin:${PATH} && export LD_LIBRARY_PATH=${JULIA_MPI_PATH}/lib:${LD_LIBRARY_PATH} && echo "--- Setup Julia packages" && julia --color=yes --project=. -e ' && import Pkg && Pkg.develop(; path = joinpath(pwd(), "lib", "MPIPreferences")) && ' && julia --color=yes --project=test -e ' && using Pkg && Pkg.develop(path="lib/MPIPreferences") && using MPIPreferences && MPIPreferences.use_system_binary(export_prefs=true) && rm("test/Manifest.toml") && ' && echo "+++ Run tests" && julia --color=yes --project=. -e ' && import Pkg && Pkg.test("MPI"; test_args=["--backend=CUDA"]) && '

Ran in 22m 23s

Matrix

Tests -- Julia 1.9echo "--- Configure MPI" && buildkite-agent artifact download --step "cuda-build-openmpi" mpi-prefix.tar.gz . && mkdir -p ${JULIA_MPI_PATH} && tar -zxf mpi-prefix.tar.gz --strip-components 1 -C ${JULIA_MPI_PATH} && export PATH=${JULIA_MPI_PATH}/bin:${PATH} && export LD_LIBRARY_PATH=${JULIA_MPI_PATH}/lib:${LD_LIBRARY_PATH} && echo "--- Setup Julia packages" && julia --color=yes --project=. -e ' && import Pkg && Pkg.develop(; path = joinpath(pwd(), "lib", "MPIPreferences")) && ' && julia --color=yes --project=test -e ' && using Pkg && Pkg.develop(path="lib/MPIPreferences") && using MPIPreferences && MPIPreferences.use_system_binary(export_prefs=true) && rm("test/Manifest.toml") && ' && echo "+++ Run tests" && julia --color=yes --project=. -e ' && import Pkg && Pkg.test("MPI"; test_args=["--backend=CUDA"]) && '

Ran in 18m 37s

Matrix

Tests -- Julia 1.10echo "--- Configure MPI" && buildkite-agent artifact download --step "cuda-build-openmpi" mpi-prefix.tar.gz . && mkdir -p ${JULIA_MPI_PATH} && tar -zxf mpi-prefix.tar.gz --strip-components 1 -C ${JULIA_MPI_PATH} && export PATH=${JULIA_MPI_PATH}/bin:${PATH} && export LD_LIBRARY_PATH=${JULIA_MPI_PATH}/lib:${LD_LIBRARY_PATH} && echo "--- Setup Julia packages" && julia --color=yes --project=. -e ' && import Pkg && Pkg.develop(; path = joinpath(pwd(), "lib", "MPIPreferences")) && ' && julia --color=yes --project=test -e ' && using Pkg && Pkg.develop(path="lib/MPIPreferences") && using MPIPreferences && MPIPreferences.use_system_binary(export_prefs=true) && rm("test/Manifest.toml") && ' && echo "+++ Run tests" && julia --color=yes --project=. -e ' && import Pkg && Pkg.test("MPI"; test_args=["--backend=CUDA"]) && '

Ran in 13m 49s

Build OpenMPIecho "--- Install packages" && apt-get install --yes --no-install-recommends curl ccache && export PATH="/usr/lib/ccache/:$PATH" && echo "--- Build UCX" && curl -L https://github.com/openucx/ucx/releases/download/v${UCX_VER}/ucx-${UCX_VER}.tar.gz --output ucx.tar.gz && tar -zxf ucx.tar.gz && pushd ucx-* && ./configure --with-rocm --enable-mt --prefix=$(realpath ../mpi-prefix) && make -j && make install && popd && echo "--- Build OpenMPI" && curl -L https://download.open-mpi.org/release/open-mpi/v${OPENMPI_VER}/openmpi-${OPENMPI_VER_FULL}.tar.gz --output openmpi.tar.gz && tar -zxf openmpi.tar.gz && pushd openmpi-${OPENMPI_VER_FULL} && ./configure --with-ucx=$(realpath ../mpi-prefix) --with-rocm --prefix=$(realpath ../mpi-prefix) && make -j && make install && popd && echo "--- Package prefix" && tar -zcf mpi-prefix.tar.gz mpi-prefix/ && echo "--- ccache stats" && ccache -s

Ran in 2m 54s

Matrix

Tests -- Julia 1.10echo "--- Configure MPI" && buildkite-agent artifact download --step "rocm-build-openmpi" mpi-prefix.tar.gz . && mkdir -p ${JULIA_MPI_PATH} && tar -zxf mpi-prefix.tar.gz --strip-components 1 -C ${JULIA_MPI_PATH} && export PATH=${JULIA_MPI_PATH}/bin:${PATH} && export LD_LIBRARY_PATH=${JULIA_MPI_PATH}/lib:${LD_LIBRARY_PATH} && echo "--- Setup Julia packages" && julia --color=yes --project=. -e ' && import Pkg && Pkg.develop(; path = joinpath(pwd(), "lib", "MPIPreferences")) && ' && julia --color=yes --project=test -e ' && using Pkg && Pkg.develop(path="lib/MPIPreferences") && using MPIPreferences && MPIPreferences.use_system_binary(export_prefs=true) && rm("test/Manifest.toml") && ' && echo "+++ Run tests" && export JULIA_MPI_TEST_EXCLUDE="test_allreduce.jl,test_reduce.jl,test_scan.jl" && julia --color=yes --project=. -e ' && import Pkg && Pkg.test("MPI"; test_args=["--backend=AMDGPU"]) && '

Ran in 6m 6s

Total Job Run Time: 1h 43m