Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LDC: Don't keep around temporary object files for static libs #2690

Merged
merged 2 commits into from
Jun 13, 2024

Conversation

kinke
Copy link
Contributor

@kinke kinke commented Aug 24, 2023

This was a workaround for older LDC versions. Since LDC v1.25, -cleanup-obj implies that the temporary object files are placed into a unique temp directory (per compiler invocation), so that object file collisions across parallel dub invocations cannot happen anymore.

@github-actions
Copy link

github-actions bot commented Aug 24, 2023

✅ PR OK, no changes in deprecations or warnings

Total deprecations: 0

Total warnings: 0

Build statistics:

 statistics (-before, +after)
 executable size=5259464 bin/dub
 rough build time=62s
Full build output
DUB version 1.37.0, built on May 11 2024
LDC - the LLVM D compiler (1.38.0):
  based on DMD v2.108.1 and LLVM 18.1.5
  built with LDC - the LLVM D compiler (1.38.0)
  Default target: x86_64-unknown-linux-gnu
  Host CPU: znver3
  http://dlang.org - http://wiki.dlang.org/LDC


  Registered Targets:
    aarch64     - AArch64 (little endian)
    aarch64_32  - AArch64 (little endian ILP32)
    aarch64_be  - AArch64 (big endian)
    amdgcn      - AMD GCN GPUs
    arm         - ARM
    arm64       - ARM64 (little endian)
    arm64_32    - ARM64 (little endian ILP32)
    armeb       - ARM (big endian)
    avr         - Atmel AVR Microcontroller
    bpf         - BPF (host endian)
    bpfeb       - BPF (big endian)
    bpfel       - BPF (little endian)
    hexagon     - Hexagon
    lanai       - Lanai
    loongarch32 - 32-bit LoongArch
    loongarch64 - 64-bit LoongArch
    mips        - MIPS (32-bit big endian)
    mips64      - MIPS (64-bit big endian)
    mips64el    - MIPS (64-bit little endian)
    mipsel      - MIPS (32-bit little endian)
    msp430      - MSP430 [experimental]
    nvptx       - NVIDIA PTX 32-bit
    nvptx64     - NVIDIA PTX 64-bit
    ppc32       - PowerPC 32
    ppc32le     - PowerPC 32 LE
    ppc64       - PowerPC 64
    ppc64le     - PowerPC 64 LE
    r600        - AMD GPUs HD2XXX-HD6XXX
    riscv32     - 32-bit RISC-V
    riscv64     - 64-bit RISC-V
    sparc       - Sparc
    sparcel     - Sparc LE
    sparcv9     - Sparc V9
    spirv       - SPIR-V Logical
    spirv32     - SPIR-V 32-bit
    spirv64     - SPIR-V 64-bit
    systemz     - SystemZ
    thumb       - Thumb
    thumbeb     - Thumb (big endian)
    ve          - VE
    wasm32      - WebAssembly 32-bit
    wasm64      - WebAssembly 64-bit
    x86         - 32-bit X86: Pentium-Pro and above
    x86-64      - 64-bit X86: EM64T and AMD64
    xcore       - XCore
   Upgrading project in /home/runner/work/dub/dub/
    Starting Performing "release" build using /opt/hostedtoolcache/dc/ldc2-1.38.0/x64/ldc2-1.38.0-linux-x86_64/bin/ldc2 for x86_64.
    Building dub 1.38.0+commit.49.g88b64c5e: building configuration [application]
     Linking dub
STAT:statistics (-before, +after)
STAT:executable size=5259464 bin/dub
STAT:rough build time=62s

@kinke kinke force-pushed the ldc_temp_objects branch from 7eae6f1 to 050ca7f Compare August 24, 2023 11:28
settings.addDFlags("-lib");
// -oq: name object files uniquely (so the files don't collide)
// -cleanup-obj: remove object files after archiving to static lib, and put them in a unique temp directory
settings.addDFlags("-lib", "-oq", "-cleanup-obj");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are the new flags not required for the dynamicLibrary case below?

Copy link
Contributor Author

@kinke kinke Aug 24, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope; the object files for executables and shared libs are persisted/kept around, by dub and the compilers alike. Static libs with DMD are a special case, as it writes the static lib directly itself, without creating separate object files and then archiving them, like LDC and GDC. ldmd2 -lib also injects these 2 extra flags for the ldc2 cmdline to mimic the DMD behavior.


exit 0
numObjectFiles=$(find "$DUB_CACHE_PATH" -type f -iname '*.o*' | wc -l)
[ "$numObjectFiles" -eq 0 ] || die $LINENO "Found left-over object files in $DUB_CACHE_PATH"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this change seems to revert what has previously been published in the changelogs: kinke@176ea9c

Enables co-existence and parallel compilation of the same project with different settings (e.g. cross compilation) by moving .dub/obj to $DUB_TARGET_PATH/obj.

Although that changelog entry seems faulty right now, since it didn't actually write to the target path (e.g. "targetPath": "bin") but to the build cache path.

So I think this change is fine.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With LDC v1.25+, this PR solves the concurrency problem more robustly - each compiler invocation uses a unique temp dir for its object files, no need to specify unique-ish paths explicitly.

Copy link
Member

@WebFreak001 WebFreak001 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since the -cleanup-obj flag was introduced in 2021 and we don't really have any tool version / stability policy, can you restrict the changes into if (platform.frontendVersion >= 2095) to be on the safe side?

@kinke
Copy link
Contributor Author

kinke commented Aug 24, 2023

The flag was introduced in LDC v1.1 in 2016: ldc-developers/ldc@141310c

It's just that since v1.25 it additionally implies a concurrent-safe unique temp dir instead of the cwd (if you don't specify a -od objects dir explicitly).

@kassane kassane mentioned this pull request Jan 11, 2024
13 tasks
@kinke kinke force-pushed the ldc_temp_objects branch 2 times, most recently from 23f95fa to 0fb13ce Compare June 12, 2024 13:47
This was a workaround for older LDC versions. Since LDC v1.25,
`-cleanup-obj` implies that the temporary object files are placed
into a unique temp directory (per compiler invocation), so that
object file collisions across parallel dub invocations cannot happen
anymore.
@kinke kinke force-pushed the ldc_temp_objects branch from 0fb13ce to 5304cbe Compare June 12, 2024 13:48
@kinke
Copy link
Contributor Author

kinke commented Jun 12, 2024

@WebFreak001: Okay, I've re-added support for all LDC versions.

Co-authored-by: Mathias LANG <geod24@gmail.com>
@Geod24 Geod24 enabled auto-merge (squash) June 13, 2024 19:08
@Geod24 Geod24 merged commit 6610ddf into dlang:master Jun 13, 2024
30 of 31 checks passed
@kinke kinke deleted the ldc_temp_objects branch June 13, 2024 19:32
@Geod24
Copy link
Member

Geod24 commented Jan 24, 2025

@kinke : This broke dub build -b ddox for one of my project:

Error: Output file '__dummy_docs/package.html' for module `barcode.qr` collides with previous module `barcode`. See the -oq option

While building a project that depends on https://github.com/deviator/barcode (although I think that will trigger for any dependency, this is just the first one).
Working:

ldc2 -o- -w -lowmem -Xf=docs.json -Dd=__dummy_docs --oq -od=submodules/barcode/obj -d-version=_GLIBCXX_USE_CXX98_ABI -d-version=Have_barcode -d-debug=AgoraDebugAPIs -Isubmodules/barcode/source/ submodules/barcode/source/barcode/code128.d submodules/barcode/source/barcode/code39.d submodules/barcode/source/barcode/ean13.d submodules/barcode/source/barcode/itf.d submodules/barcode/source/barcode/package.d submodules/barcode/source/barcode/qr/ecl.d submodules/barcode/source/barcode/qr/package.d submodules/barcode/source/barcode/qr/qrcode.d submodules/barcode/source/barcode/qr/qrsegment.d submodules/barcode/source/barcode/qr/util.d submodules/barcode/source/barcode/qrwrap.d submodules/barcode/source/barcode/svgdraw.d submodules/barcode/source/barcode/types.d submodules/barcode/source/barcode/util.d -preview=in -revert=dtorfields -preview=in -revert=dtorfields -vcolumns

Broken:

ldc2 -o- -w -lowmem -Xf=docs.json -Dd=__dummy_docs -d-version=_GLIBCXX_USE_CXX98_ABI -d-version=Have_barcode -d-debug=AgoraDebugAPIs -Isubmodules/barcode/source/ submodules/barcode/source/barcode/code128.d submodules/barcode/source/barcode/code39.d submodules/barcode/source/barcode/ean13.d submodules/barcode/source/barcode/itf.d submodules/barcode/source/barcode/package.d submodules/barcode/source/barcode/qr/ecl.d submodules/barcode/source/barcode/qr/package.d submodules/barcode/source/barcode/qr/qrcode.d submodules/barcode/source/barcode/qr/qrsegment.d submodules/barcode/source/barcode/qr/util.d submodules/barcode/source/barcode/qrwrap.d submodules/barcode/source/barcode/svgdraw.d submodules/barcode/source/barcode/types.d submodules/barcode/source/barcode/util.d -preview=in -revert=dtorfields -preview=in -revert=dtorfields -vcolumns

@kinke
Copy link
Contributor Author

kinke commented Jan 24, 2025

I guess it needs -oq (what about DMD, which AFAIK doesn't have that switch? maybe just no colliding-output-files check, so a silent overwrite?). The command doesn't seem to go through setTarget(), where the -oq would be applied (for static libs).

@kinke
Copy link
Contributor Author

kinke commented Jan 24, 2025

Oh, looks like a potential misconfig for BuildOption._ddox - DMD and GDC use a single file (via -Df / -fdoc-file), while LDC gets a dir (-Dd).

@kinke
Copy link
Contributor Author

kinke commented Jan 24, 2025

=> #2994

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants