[native_toolchain_c] Shared library build doesn't include libc++_shared.so despite requiring it (Android)

[zeyus]

When targeting android with CBuilder.library if the sources use the stdlib then officially the android documentation state that you are required to manually bundle the libc++_shared.so but this isn't bundled, and as far as I can tell there's no easy way to do this.

The same build works for all other targets I've tested so far (Windows, iOS, OSX).

the full implementation I've done is available here: https://github.com/zeyus/liblsl.dart/blob/5335a97670a28a1b2464210f7c25a3878afb8ecd/packages/liblsl/hook/build.dart

I've come up with a workaround, but it feels like a bit of a perversion of the "public" API.

import 'package:logging/logging.dart';
import 'package:native_assets_cli/code_assets.dart';
import 'package:native_toolchain_c/native_toolchain_c.dart';
import 'package:native_toolchain_c/src/cbuilder/run_cbuilder.dart';
import 'package:native_toolchain_c/src/native_toolchain/android_ndk.dart';

void main(List<String> args) async {
  await build(args, (input, output) async {
    final packageName = input.packageName;
    final OS targetOs = input.config.code.targetOS;
    final Architecture targetArchitecture =
        input.config.code.targetArchitecture;

    // ... other build stuff, 
    final builder = CBuilder.library(
      // ...
    );
    await builder.run(
      input: input,
      output: output,
      logger:
          Logger('')
            ..level = Level.ALL
            ..onRecord.listen((record) => print(record.message)),
    );
    // no error handling here yet, but if the build works, we should be able to assume
    // that the android NDK / tools exist?
    if (targetOs == OS.android) {
          // add libc++_shared.so from the NDK
          final aclang = await androidNdkClang.defaultResolver!.resolve(
            logger: Logger(''),
          );
          for (final tool in aclang) {
            if (tool.tool.name == 'Clang') {
              final sysroot = tool.uri.resolve('../sysroot/').toString();
              final androidArch =
                  RunCBuilder.androidNdkClangTargetFlags[targetArchitecture];
              final libPath = '$sysroot/usr/lib/$androidArch/libc++_shared.so';
              output.assets.code.add(
                CodeAsset(
                  package: packageName,
                  name: 'libc++_shared.so',
                  file: Uri.parse(libPath),
                  linkMode: DynamicLoadingBundled(),
                  os: targetOs,
                  architecture: targetArchitecture,
                ),
              );
              break;
            }
          }
        }

What might be nice is exposing a target-specific stdlib path, or otherwise perhaps just a flag to include the libc++_shared.so asset.

Of course, maybe I missed something obvious where adding some compiler flag or define would bundle the .so, but I tried a few things and it didn't work.

[blaugold]

A solution also needs to take into account that multiple packages might need libc++_shared.so to be bundled, but they can't all individually add a native asset with the same name. Or maybe the validation step could allow multiple packages adding the same native asset if it is identical.

[dcharkes]

If we wanted to dedupe multiple libc++_shared.so inside a single hook, that would work, but that wouldn't work if we have two packages who want to share such dynamic library.

A way to make this work, with multiple packages collaborating, would be to define a package:android_libcpp_shared with its own hook/build.dart. And then you'd depend on this package if you know you need libc++_shared.so for your Android build. The downside is that this requires collaboration between packages. The upside this that it's unambigious which version of libc++_shared.so is used.

The alternative, deduplicating duplicately defined assets seems like a bad design choice.

However, this kind of means that we'll start needing Dart wrapper packages for all kinds of shared libraries that people might be needing. Maybe that's the right thing to do, it ensures a single-version policy. But it also feels somewhat painful to have to collaborate across possibly unrelated packages.

I'm open to alternative suggestions.

Some related issues:

• Helpers for discovering dynamic dependencies of dylibs #191
• (I can't find the relevant GitHub issue, I thought we have one.) When we do tree-shaking, if all dylibs that require libc++_shared.so are completely removed, we should also not bundle libc++_shared.so. So we'll need to report in every CodeAsset what other dylibs it wants to open at runtime.

[zeyus]

@dcharkes I guess conceptually, it could go in a few directions:

• it's all handled in a way that's completely transparent to whoever is making a build hook, using some tool or parsing to see if additional libraries are required. this seems...nightmarish to maintain, but I suppose these kind of things are the norm for Dart :)
• slightly abstract the process -> make a public api / param to add shared libs (maybe by target OS), or essentially wrap what I have above so you could just call something like output.assets.bin.add('libwhatever')
• require manually adding the dependency / copying the lib, which might require a discrete implementation for each os/tool/arch
• externalize shared libs (but as you point out, that requires collab + potentially having flavors for each os/arch as well)

I definitely see your point about duplicate assets, but I don't think I understand enough about the destination structure, because if I get the point, then that means if someone uses two different dart packages that e.g. provide different layers of database abstraction, but both decided to implement their own native libsqlite.so, and both of these packages used the exact same source code version to compile the dynamic lib (forgetting about additional libc++_shared.so etc) then the final apk / app would end up with two identical copies of the libsqlite.so but in different directories?

That's probably an extreme edge case, but the implications extend to shared libraries and any native compiled code, and I'm not sure that having packages for shared libs would avoid the issue.

Alternatively (this may be a stupid suggestion) would it be possible to do some kind of final build step, where all assets from the root package and dependencies are collated and where there's name match (e.g. libc++_shared.so) if the hashes are identical, then it could be consolidated? (this would assume a shared path for all libs/bins, but it could also apply to images or other resources if multiple packages included some icon library)

I haven't even started with tree-shaking yet, but I will have to get around to it once I've built up a stable API.

Edit:
Just wanted to add, I think it's safe to assume if someone is making a native build hook, they would have to have some understanding of the underlying source code of the libraries they require, but in my case, the gap in my knowledge for this process came from the varying requirements per target platform, and of course doing the perhaps questionable thing of bypassing all the existing CMake scripts, because for liblsl, the iOS and Android versions are all over the place and it's not a matter of cloning one repo, but 4 or 5 to make it work...so I made it harder for myself in hopes of saving future work...which seems to have panned out?

[dcharkes]

but both decided to implement their own native libsqlite.so, and both of these packages used the exact same source code version to compile the dynamic lib (forgetting about additional libc++_shared.so etc) then the final apk / app would end up with two identical copies of the libsqlite.so but in different directories?

They cannot be in separate directories. Moving dynamic libraries to be in a different location requires rewriting dynamic libraries that open them to change the opening path. This is supported on MacOS/iOS but not on the other OSes. This means all dynamic libraries are in one directory and one namespace. This is typically how C compiler work.

Hence, therefore everyone would need to agree on what libsqlite.so to use. We need a package:native_sqlite_asset (or some other name), that basically only provides the dylib. And then other packages would depend on that asset. Because sqlite never breaks it API, that package simply needs to be kept up to date with the latests version of sqlite. This is very similar to how sqlite would be updated in apt-get, brew etc, and how other packages in such package manager don't bundle sqlite themselves but depend on sqlite. The package:native_sqlite_asset should probably use as it's semantic version the version of the bundled sqlite.

The only way to have multiple of the same dynamic library is if you modify the compilation of sqlite to output a dylib with a different name libmy_sqlite.so or libsqlite.so.3.49.1, and use that specific name load sqlite in your own native code. (I believe this is how native package managers deal with multiple versions.) We will not automatically do this and put things in different directories. The person writing the hook will need to do this. (But again, if at all possible, we should aim for sharing assets in a shared dependency.)

Alternatively (this may be a stupid suggestion) would it be possible to do some kind of final build step, where all assets from the root package and dependencies are collated and where there's name match (e.g. libc++_shared.so) if the hashes are identical, then it could be consolidated? (this would assume a shared path for all libs/bins, but it could also apply to images or other resources if multiple packages included some icon library)

I thought about this option as well, but I'm thinking that this is a bad design, because that's not how other package managers work. It would complicate the logic for the SDKs as well.

Unduplicating code between two packages is done by moving it to a shared dependency. Unduplicating assets between two packages is done by moving the assets to a shared dependency.

I haven't even started with tree-shaking yet, but I will have to get around to it once I've built up a stable API.

For tree-shaking you need to pass --enable-experiment=record-use in Dart. In Flutter it's not available yet.

[zeyus]

The package:native_sqlite_asset should probably use as it's semantic version the version of the bundled sqlite.

That's a good point, and has implications for the lib I'm making.

We will not automatically do this and put things in different directories. The person writing the hook will need to do this. (But again, if at all possible, we should aim for sharing assets in a shared dependency.)

I agree, that would add unnecessary complexity, and clearly it would be better to have one source for a particular native library anyway.

Unduplicating assets between two packages is done by moving the assets to a shared dependency.

Yeah, this is probably the way to go. I think it would be nice to have some kind of (soft)enforced standard. For this design, what could help a lot is:

• dart pub search - so you could search for libsqlite; and/or
• dart pub provides - or something similar (along the lines of dpkg -S or yum provides that is specific to native assets to see which packages, if any, give you a libsqlite[.so], or libsqlite-3

For tree-shaking you need to pass --enable-experiment=record-use in Dart. In Flutter it's not available yet.

Great, I'll definitely be testing this out, I don't plan to require flutter for the native asset anyway, apart from the separate test package which will use integration tests to make sure it's working on different platforms.

As an aside, (maybe this is common/known knowledge, I haven't searched the issues), getting this working in Android mean I could immediately also deploy to a Quest 2, and it worked without issues which is awesome for research and also game contexts.

For the rest of this discussion, I think I will bow out because it seems like a major design decision which I don't have particularly strong opinions on, and probably should be decided by others who know more about the roadmap and internals.