Static Linking doesn't work yet

[helje5]

All the required static libs should be available in the SDK / X toolchain, but I still got linking error. Need to find the right combination of flags.

[pokryfka]

It's on my todo list as well.

@fabianfett post compilation flags he used: swift-server/swift-aws-lambda-runtime#17 (comment)

[pokryfka]

I can confirm that adding

-latomic
-lDispatchStubs

fixed static compilation in my setup on swift:5.2-amazonlinux2 docker image with glibc-static.

/usr/lib/swift_static/linux/static-executable-args.lnk

-static
-lswiftCore
-lswiftImageInspectionShared
-Xlinker
--defsym=__import_pthread_self=pthread_self
-Xlinker
--defsym=__import_pthread_once=pthread_once
-Xlinker
--defsym=__import_pthread_key_create=pthread_key_create
-lpthread
-licui18nswift
-licuucswift
-licudataswift
-ldl
-lstdc++
-lm
-lDispatchStubs
-latomic

build with:

swift build --product HelloWorldAPIPerf -c release -Xswiftc -static-executable

[helje5]

Didn't know about that static-executable-args.lnk file. Might be enough to just put that into the xtoolchain?

[helje5]

Looks like -static makes the linker not pick up .so links to dynamic libraries. But some (like libuuid) only exist as dynamic libraries. Or in other words, even when linking statically, this still needs to build a dynamically linked binary.

[pokryfka]

I did a quick test last week, it worked, it did noticeably improve cold start, QED.

Revisited it today with partial success.

Looks like -static makes the linker not pick up .so links to dynamic libraries. But some (like libuuid) only exist as dynamic libraries. Or in other words, even when linking statically, this still needs to build a dynamically linked binary.

static-stdlib settings, as defined in /usr/lib/swift_static/linux/static-stdlib-args.lnk link statically some libraries and dynamically others.

Examples:

all static

# swift build --product HelloWorldAPIPerf -c release -Xswiftc -static-executable -Xswiftc -lDispatchStubs
  
# ldd .build/release/HelloWorldAPIPerf
  	not a dynamic executable

static + dyn

# swift build --product HelloWorldAPIPerf -c release -Xswiftc -static-stdlib -Xswiftc -lDispatchStubs

# ldd .build/release/HelloWorldAPIPerf
	linux-vdso.so.1 (0x00007ffe86d83000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fcf0011f000)
	libutil.so.1 => /lib64/libutil.so.1 (0x00007fcefff1c000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007fceffd18000)
	libm.so.6 => /lib64/libm.so.6 (0x00007fceff9d8000)
	libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fceff656000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fceff440000)
	libc.so.6 => /lib64/libc.so.6 (0x00007fceff095000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fcf02750000)

static + dyn including libuuid

# swift build --product HelloWorldAPIPerf -c release -Xswiftc -static-stdlib -Xswiftc -lDispatchStubs -Xlinker -export-dynamic -Xlinker -luuid

# ldd .build/release/HelloWorldAPIPerf
	linux-vdso.so.1 (0x00007ffdb1faa000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f1d13fad000)
	libutil.so.1 => /lib64/libutil.so.1 (0x00007f1d13daa000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f1d13ba6000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f1d13866000)
	libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f1d134e4000)
	libuuid.so.1 => /lib64/libuuid.so.1 (0x00007f1d132df000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f1d130c9000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f1d12d1e000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f1d165de000)

all dyn

# swift build --product HelloWorldAPIPerf -c release

# bash-4.2# ldd .build/release/HelloWorldAPIPerf
	linux-vdso.so.1 (0x00007ffd59791000)
	libswiftCore.so => /usr/lib/swift/linux/libswiftCore.so (0x00007fe697755000)
	libswiftGlibc.so => /usr/lib/swift/linux/libswiftGlibc.so (0x00007fe6981ca000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fe697537000)
	libutil.so.1 => /lib64/libutil.so.1 (0x00007fe697334000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007fe697130000)
	libm.so.6 => /lib64/libm.so.6 (0x00007fe696df0000)
	libswiftDispatch.so => /usr/lib/swift/linux/libswiftDispatch.so (0x00007fe698190000)
	libdispatch.so => /usr/lib/swift/linux/libdispatch.so (0x00007fe696b90000)
	libBlocksRuntime.so => /usr/lib/swift/linux/libBlocksRuntime.so (0x00007fe69698d000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fe696777000)
	libc.so.6 => /lib64/libc.so.6 (0x00007fe6963cc000)
	libicui18nswift.so.65 => /usr/lib/swift/linux/libicui18nswift.so.65 (0x00007fe695ea4000)
	libicuucswift.so.65 => /usr/lib/swift/linux/libicuucswift.so.65 (0x00007fe695a9d000)
	libicudataswift.so.65 => /usr/lib/swift/linux/libicudataswift.so.65 (0x00007fe693dee000)
	libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fe693a6c000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fe697fbd000)
	librt.so.1 => /lib64/librt.so.1 (0x00007fe693864000)

---

Now I guess I should have mentioned that and it does make big difference, HelloWorldAPIPerf does not use Foundation.

So while the approach above should be useful, I have not yet done it with Foundation.

[helje5]

Yes, I was specifically trying with Foundation (with static linking there should be very little reason to avoid it in the first place). This introduces the problematic libuuid dependency.

I should eventually disable SIP and check what the linker is actually doing. Not sure when I'll find the time.

[helje5]

Hm, why is --export-dynamic required, that feels wrong 🤔

[helje5]

FWIW, AFAIK Swift 5.3 brought static linking fixes, which might solve the whole Radar. Need to figure out the required flags.

https://forums.swift.org/t/static-linking-on-linux-in-swift-5-3-1/41989

-static-stdlib: only statically link Swift stdlib, Foundation, Dispatch, but not ICU
-static-executable: self contained executable

So we probably need the first when building for a regular Ubuntu and the latter for Amazon Linux 🤔

[helje5]

swift lambda already specifies both, -Xswiftc -static-stdlib -Xswiftc -static-executable, not sure which one wins :-) But a build doesn't go through for other reasons:

<module-includes>:5:10: note: in file included from <module-includes>:5:
#include "private_includes/utmp.h"
         ^
/usr/local/lib/swift/dst/x86_64-unknown-linux/swift-5.3-amazonlinux2.xtoolchain/x86_64-amazonlinux2.sdk/usr/lib/swift/linux/x86_64/private_includes/utmp.h:1:10: note: in file included from /usr/local/lib/swift/dst/x86_64-unknown-linux/swift-5.3-amazonlinux2.xtoolchain/x86_64-amazonlinux2.sdk/usr/lib/swift/linux/x86_64/private_includes/utmp.h:1:
#include <utmp.h>
         ^

Which we have in

x86_64-amazonlinux2.sdk/usr/include/utmp.h
x86_64-amazonlinux2.sdk/usr/include/bits/utmp.h
x86_64-amazonlinux2.sdk/usr/lib/swift/linux/x86_64/private_includes/utmp.h
swift.xctoolchain/usr/lib/swift/linux/x86_64/private_includes/utmp.h

Note sure why this works for dynamic, but not for static. The include is from the glibc.modulemap which is present in both, SDK and toolchain.

[helje5]

utmp.h doesn't actually seem to be missing, it is stddef.h.

It gets a little further when specifying -Xswiftc -I -Xswiftc /usr/local/lib/swift/dst/x86_64-unknown-linux/swift-5.3-amazonlinux2.xtoolchain/x86_64-amazonlinux2.sdk/usr/include/linux, but ends up in redefinition errors.