Is it possible/practical to run cibuildwheel locally?

[ndw]

I have cibuildwheel running successfully in GitHub actions for Linux and Windows Intel wheels and Mac universal wheels. That's all working just fine.

I'd like to be able to make Linux ARM wheels (to, for example, run in containers on Apple Silicon Macs). There's a fair bit of complexity in the GitHub action because we don't actually have the sources for building the libraries that are being linked in the repository. The action grabs them from another server and builds on top of them. That works, but looking at the instructions for doing QEMU emulation to get the ARM wheels doesn't fill me with confidence that I'll be able to navigate that much more complexity.

Instructions, pointers, etc. for building wheels in, for example a local docker container, would be much appreciated.

[Czaki]

Building Linux wheels is safe. You will end with few additional docker images on machine

[henryiii]

You don’t need a vm either. As long as you tell it --platform linux you can (and should) run it natively. You might need to pass the archs flag too, will have to check and see if we get the default right on AS.

[joerick]

You might be overthinking this...! Don't worry about VMs etc on Mac, Docker Desktop for Mac handles that for you. As long as you have Docker for Mac running, you should be able to run cibuildwheel --platform linux from a Mac terminal and it will start building inside a docker container. It will even try to emulate other archs if you want, but i've had mixed success with that on mac.

[ndw]

Ah. I may have failed to make clear that part of what needs to be in the wheel is a binary shared library. So I have to build that on a Linux platform of some sort.

[henryiii]

It does run in a Linux docker container - it always does this for Linux. --platform linux will always start up a Linux Docker image. On macOS, Docker sets up a light weight VM when you run a Linux Docker container.

If you need to add something to the container, you have to either build it in CIBW_BEFORE_ALL/tool.cibuildwheel.before-all or add it to the folder that gets mounted.

[ndw]

Right. I got it to work. Thank you for all your help!

[tabbyrobin]

Not sure if I should start a new conversation for this? I have a related question: Is running cibuildwheel locally a goal/feature of the project? Will it be supported indefinitely, with (reasonably) full feature parity to cloud runs?

I see that it is mentioned in the docs, but there it is described as a convenient thing to do before doing the real (in-cloud) thing.

The context for my question is for enabling to verify reproducible builds. For this, the exact same logic needs to run in-cloud and locally.

[Czaki]

It depends what you mean locally. On your machine. I expect yes. With side effect on Windows and macOS of installing python from sources selected by project. That may not be suitable for all personal machines used for development purposes.

[henryiii]

An example is if you pre-install a python.org Python on your machine, cibuildwheel will use that instead of uninstalling yours and installing its own. It might be a different version or be native/universal when we would have selected the other one. That's exactly the same on CI, actually, but it's more likely to happen locally. And it's useful in CI in some cases, like when you need the one we don't use.

But yes, mostly the same, minus interactions with installed software on non-Linux systems.

[tabbyrobin]

Ok, great! Thanks