Switch to the new C++ toolchains API

[AustinSchuh]

The new BUILD file based C++ toolchains promise to be much easier to maintain.

https://github.com/bazelbuild/rules_cc/tree/main/examples/rule_based_toolchain

What do you all think about switching over to using those? I'm happy to help.

The example makes it look like we can reuse pieces from rules_cc.

cc_toolchain(                                                                                                                           
    name = "host_clang",                                                                                                           
    args = select({                                                                                                                              
        "@platforms//os:linux": [                                                                                                                            
            "//toolchain/args:linux_sysroot",                                                                             
        ],                                                                                                                                         
        "//conditions:default": [],    
    }) + [                                                                                                                                                       
        "//toolchain/args:no_canonical_prefixes",                                                                                                   
        "//toolchain/args:warnings",                                                                                  
    ],                                                                    
    enabled_features = ["@rules_cc//cc/toolchains/args:experimental_replace_legacy_action_config_features"],                                                                                                  
    known_features = ["@rules_cc//cc/toolchains/args:experimental_replace_legacy_action_config_features"],    
    tool_map = "//toolchain/tools:all_tools",                                                                       
)                                                                                                                                                            
                                                                                                                 
toolchain(                                                                                                                         
    name = "host_cc_toolchain",                                                                                                   
    toolchain = ":host_clang",                                                                                                                                    
    toolchain_type = "@bazel_tools//tools/cpp:toolchain_type",                                                                                
)  

[keith]

note that this repo mostly just doesn't use any of the old stuff for this either since it just shells out to the default toolchain setup code.

toolchains_llvm/toolchain/cc_toolchain_config.bzl

Line 368 in bb3f7d0

unix_cc_toolchain_config(

[AustinSchuh]

Yup. I'm trying to upstream a bunch of fixes we've built up over the years that let us cross compile against sysroots. I'm quickly finding all the places where new flags need to be injected in very specific spots through unix_cc_toolchain_config which don't exist.

[dmivankov]

I think this will enable use of rule-based sysroots. Currently toolchains_llvm seems to only set correct paths for http_archive or for in-source filegroup, but not for genrule outputs or other ways to produce sysroots. While new API in rules_cc looks more promising handling generated sysroots (Note: there's still a bit of hacking needed to have a rule that produces multiple files, just producing sysroot archive is much more simpler but unusable as sysroot directly)

[AustinSchuh]

Keith and I (more Keith than me) sent some changes upstream to rules_cc to get things closer to working there, but most of those changes haven't landed yet. I think it is clear from playing with the new API that it isn't fully debugged and tested.

[keith]

FWIW I rolled out a toolchain internally using the new stuff with a custom sysroot and vendored clang and it's working well. But it does still require some patches on rules_cc

[AustinSchuh]

@keith, are the patches needed just the ones you've sent out for review on rules_cc? Or are there more?

[keith]

All submitted, no private ones. But I do have a significant amount of internal setup that isn't clear how it should fit upstream yet. For example sanitizer flags etc.

[dmivankov]

Would you be able to share current patch to toolchains_llvm that sets up new API (to give it a try ahead of rules_cc accepting the changes)?

[keith]

I'm not using toolchains_llvm at all. since I don't need the conditional setup of versions of llvm or the other things it does when im vendoring exactly what i need.

[dmivankov]

That works too, I'm just not quite sure how much setup is needed to get to a "simple hermetic toolchain" setup using new APIs, bare-bones pointing tools to compiler binaries seems to miss many default or required flags and also differs noticeably from toolchains_llvm (it's nice that aquery can be used to discover differences but a proven starting point would be nice too 🙂 )

[AustinSchuh]

If you want a proven starting point, https://github.com/RealtimeRoboticsGroup/aos/tree/main/third_party/bazel-toolchain is mine. It supports x86 debian, and a NVIDIA Orin yocto. It is based off an old version of toolchains_llvm. I'm slowly trying to get the patches upstreamed to stop maintaining a fork, but keep running into gaps in what is there in rules_cc.

[keith]

there is also https://github.com/bazelbuild/rules_cc/blob/main/examples/rule_based_toolchain/toolchains/ but it's pretty bare bones

[keith]

there's also https://github.com/google/pigweed/tree/main/pw_toolchain/cc but it's a bit interesting since much of it is targeting embedded things

[thesayyn]

This repo was mostly written following the https://bazel.build/tutorials/ccp-toolchain-config. New rule based toolchains seem much cleaner. I'd be also happy to give it a try.

[thesayyn]

I think this will enable use of rule-based sysroots

I experimented with rule based toolchains a bit, unfortunately it does not help much performance since sysroot is expected to be extract during loading phase, which is not so different than traditional cc toolchains. I filed bazelbuild/bazel-skylib#566 to see what can be done there.