Skip to content

Commit 410a280

Browse files
odowodow
authored
Add a blog post for testing cross-compiled JLL binaries (#171)
1 parent ae9e95d commit 410a280

File tree

1 file changed

+235
-0
lines changed

1 file changed

+235
-0
lines changed

‎_posts/2025-05-14-jll.md

Lines changed: 235 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,235 @@
1+
---
2+
layout: post
3+
title: "Building and testing JLLs in GitHub actions"
4+
date: 2025-05-13
5+
categories: [tutorials]
6+
author: "Oscar Dowson (@odow)"
7+
---
8+
9+
_The purpose of this blog post is to document a workflow that I have found
10+
useful during the development and maintenance of solver wrappers. By the time
11+
you read this, it might be out of date, but I hope it's a helpful prod in the
12+
right direction._
13+
14+
## Background
15+
16+
Many JuMP solvers are implemented in compiled languages such as C or C++. We
17+
build and distribute open-source solvers via Julia's excellent
18+
[Yggdrasil build infrastructure](https://github.com/JuliaPackaging/Yggdrasil).
19+
20+
Yggdrasil is similar to other build systems such as homebrew. Each project that
21+
we want to build gets a separate directory. For example, the conic solver
22+
[embotech/ecos](https://github.com/embotech/ecos) has a directory [E/ECOS](https://github.com/JuliaPackaging/Yggdrasil/blob/master/E/ECOS).
23+
Inside that directory is a [build_tarballs.jl](https://github.com/JuliaPackaging/Yggdrasil/blob/master/E/ECOS/build_tarballs.jl)
24+
file, which is a Julia script that describes where to download the source code,
25+
how to compile it, what platforms it should be compiled to, and so on.
26+
27+
The output of running the script is a Julia package, for example, [ECOS_jll.jl](https://github.com/JuliaBinaryWrappers/ECOS_jll.jl),
28+
and a [set of binary artifacts](https://github.com/JuliaBinaryWrappers/ECOS_jll.jl/releases/tag/ECOS-v200.0.800%2B0)
29+
with the compiled binaries for each platform. "JLL" packages are regular Julia
30+
packages. The `_jll` suffix is naming convention to signify that they have been
31+
automatically built to distribute compiled binaries.
32+
33+
When you make a pull request to add or edit a `build_tarballs.jl` file on
34+
Yggdrasil, the CI machines start a Linux machine, run your script, and build all
35+
variations required. When the PR is merged, these get uploaded to the associated
36+
`_jll.jl` package and a new release is tagged.
37+
38+
An important point that will soon be relevant is that Yggdrasil cross-compiles
39+
each project from a Linux build to the target platform.
40+
41+
ECOS is a pretty simple project to compile, but we still cross-compile 16
42+
different versions for a variety of different platforms. For example, there are
43+
Windows builds for 32- and 64-bit systems, Mac builds for Intel and ARM
44+
processors, and Linux builds for x64_64, i686, aarch64, armv6l, and armv7l
45+
processors, as well builds that depend on whether the user is running
46+
[glibc](https://en.wikipedia.org/wiki/Glibc) or [musl](https://en.wikipedia.org/wiki/Musl).
47+
A more compilicated project like [Ipopt_jll](https://github.com/JuliaBinaryWrappers/Ipopt_jll.jl)
48+
has 85 different versions, with additional builds that depend on the C++ and
49+
Fortran versions used by the linked code.
50+
51+
Once things are built and running, this system works pretty smoothly. But there
52+
are two major issues.
53+
54+
First, upstream solvers may change their compilation workflow, or accidentally
55+
break compilation on some platforms because they do not test on the full suite
56+
of platforms that Yggdrasil supports. This means that small maintenance jobs
57+
like "hey, upstream has released a new version, let's rebuild the JLL" can turn
58+
into long back-and-forth debugging as we attempt to compile, find a bug, develop
59+
a patch, and then either carry it in the JLL build, or upstream it and wait for
60+
a new release.
61+
62+
One solution is for the upstream project to add a GitHub action workflow that
63+
uses the Yggdrasil build script in their CI. This should give upstream advance
64+
notice of compilation failures. We have already added such workflows to
65+
[ERGO-Code/HiGHS](https://github.com/ERGO-Code/HiGHS/blob/latest/.github/workflows/julia-tests-ubuntu.yml)
66+
and [cvanaret/Uno](https://github.com/cvanaret/Uno/blob/main/.github/workflows/julia-tests-ubuntu.yml).
67+
As a side benefit, if they build a Linux binary, they can then install related
68+
Julia packages and run their tests to ensure that no breaking changes have been
69+
introduced. For example, the HiGHS script installs and tests [HiGHS.jl](https://github.com/jump-dev/HiGHS.jl),
70+
which indirectly provides a few thousand tests of the HiGHS C API.
71+
72+
The second major problem is that cross-compiling from Linux to the target
73+
platforms means that we cannot test the binaries that are built during the
74+
Yggdrasil run. We can ensure only that compilation succeeds, and thus any
75+
runtime bugs cannot be checked or tested against. As one example, [SCIP.jl](https://github.com/scipopt/SCIP.jl)
76+
was unusable on Windows because of a runtime segfault. The debugging step for
77+
this build issue was painful, because it required someone to start a Linux
78+
machine, cross-compile to Windows using Yggdrasil, copy the binaries to a
79+
Windows machine, run, debug, make changes to the Linux build, and repeat. I
80+
developed the following script for use in SCIP.jl to help debug the problem.
81+
82+
At a high level, it starts a Linux machine, uses the Yggdrasil build script to
83+
compile a binary, saves that to GITHUB_OUTPUT, starts a new machine (in this
84+
case, Windows), downloads the compiled binary, and then patches the relevant
85+
JLL package using Julia's [Override.toml mechanism](https://jump.dev/JuMP.jl/stable/developers/custom_solver_binaries/).
86+
87+
## A workflow to test the cross-compiled binaries
88+
89+
To make things a bit simpler, I've changed it to use the relevant data for ECOS
90+
instead of SCIP.
91+
92+
### Layout
93+
94+
This workflow assumes that you have a Julia package with the layout:
95+
96+
```
97+
ECOS/
98+
.github/
99+
julia/
100+
build_tarballs.jl
101+
workflows/
102+
cross-compile-and-test.yml
103+
src/
104+
ECOS.jl
105+
test/
106+
runtests.jl
107+
Project.toml
108+
```
109+
110+
### Julia script
111+
112+
Here is the contents for `.github/julia/build_tarballs.jl`:
113+
114+
```julia
115+
using BinaryBuilder
116+
117+
name = "ECOS"
118+
version = v"2.0.8"
119+
120+
# Collection of sources required to build ECOSBuilder
121+
sources = [
122+
GitSource("https://github.com/embotech/ecos.git", "3b98fe0376ceeeb8310a06694b0a84ac59920f3f")
123+
]
124+
125+
# Bash recipe for building across all platforms
126+
script = raw"""
127+
cd $WORKSPACE/srcdir/ecos*
128+
make shared
129+
mkdir -p ${libdir}
130+
cp libecos.${dlext} ${libdir}
131+
cp -r include ${prefix}
132+
"""
133+
134+
# These are the platforms we will build for by default, unless further
135+
# platforms are passed in on the command line
136+
platforms = supported_platforms()
137+
138+
# The products that we will ensure are always built
139+
products = [
140+
LibraryProduct("libecos", :libecos)
141+
]
142+
143+
# Dependencies that must be installed before this package can be built
144+
dependencies = Dependency[]
145+
146+
build_tarballs(
147+
ARGS,
148+
name,
149+
version,
150+
sources,
151+
script,
152+
platforms,
153+
products,
154+
dependencies,
155+
julia_compat = "1.6",
156+
)
157+
```
158+
159+
### YAML
160+
161+
Here is the contents for `.github/workflows/cross-compile-and-test.yml`:
162+
163+
````yaml
164+
name: Build on Linux, Run on Windows
165+
on:
166+
push:
167+
# The branch might be named something else, like `main` or `latest`
168+
branches: [master]
169+
pull_request:
170+
types: [opened, synchronize, reopened]
171+
# needed to allow julia-actions/cache to delete old caches that it has created
172+
permissions:
173+
actions: write
174+
contents: read
175+
jobs:
176+
# The purpose of this job is to install Julia, run binary builder, and store
177+
# the solver artifact for the next job.
178+
build-linux:
179+
runs-on: ubuntu-latest
180+
steps:
181+
- uses: actions/checkout@v4
182+
# Install Julia 1.7 for BinaryBuilder. Note that this is an old version of
183+
# Julia, but it is required for compatibility with BinaryBuilder.
184+
- uses: julia-actions/setup-julia@v2
185+
with:
186+
version: "1.7"
187+
arch: x64
188+
- uses: julia-actions/cache@v2
189+
- run: |
190+
# Replace as needed
191+
PACKAGE=ECOS_jll
192+
PLATFORM=x86_64-w64-mingw32-cxx11
193+
julia --color=yes -e 'using Pkg; Pkg.add("BinaryBuilder")'
194+
julia --color=yes .github/julia/build_tarballs.jl ${PLATFORM} --verbose --deploy=local
195+
file=/home/runner/.julia/dev/${PACKAGE}/Artifacts.toml
196+
sha1=$(grep '^git-tree-sha1' "$file" | cut -d '"' -f2)
197+
echo "ARTIFACT_SHA=${sha1}" >> $GITHUB_ENV
198+
- uses: actions/upload-artifact@v4
199+
with:
200+
name: artifacts
201+
path: '/home/runner/.julia/artifacts/${ env.ARTIFACT_SHA }'
202+
# The purpose of this job is to install Julia, download the artifact from
203+
# build-linux, and use it to run the solver's tests.
204+
run-windows:
205+
# You could replace this `runs-on` with `macOS-latest` if desired.
206+
runs-on: windows-latest
207+
# Declare that we need `build-linux` to finish first.
208+
needs: build-linux
209+
steps:
210+
- uses: actions/checkout@v4
211+
- uses: julia-actions/setup-julia@v2
212+
with:
213+
version: "1"
214+
arch: x64
215+
- uses: julia-actions/cache@v2
216+
- uses: julia-actions/julia-buildpkg@v1
217+
# Download the artifact from the `build-linux` job, and store it in
218+
# ${{ GITHUB_WORKSPACE }}/override
219+
- uses: actions/download-artifact@v4
220+
with:
221+
name: artifacts
222+
path: override
223+
# Replace ECOS_jll with the desired JLL
224+
- shell: julia --color=yes --project=. {0}
225+
run: |
226+
import ECOS_jll
227+
artifact_dir = ECOS_jll.artifact_dir
228+
sha = last(splitpath(artifact_dir))
229+
dir = escape_string(joinpath(ENV["GITHUB_WORKSPACE"], "override"))
230+
content = "$sha = \"$(dir)\"\n"
231+
write(replace(artifact_dir, sha => "Overrides.toml"), content)
232+
# This assumes that CI is being run from the root of a Julia package. If
233+
# not, you could change this as desired.
234+
- uses: julia-actions/julia-runtest@v1
235+
````

0 commit comments

Comments
 (0)