pipewire full features

[knyipab]

Intro

We know that pipewire is the future of Linux audio. Pipewire could serve pulseaudio, jack, alsa (and probably gstreamer) clients.

flowchart LR
    A1[pulseaudio clients] <-->|OK| B(pipewire)
    A2[jack clients] <-->|OK?| B(pipewire)
    A3[alsa clients] <-->|OK| B(pipewire)
    A4[gstreamer clients] <-->|possible?| B(pipewire)
    B <--> C(module-oboe-sink/source.cpp)
    C <--> D(Oboe)
    D <--> E(AAudio/SLES)
linkStyle 3 stroke:red,color:red;

Loading

This PR is currently more an initial attempt and also serves as a tracker. Cuz this initial attempt already consumed me a lot of time, I want to gauge whether devs are interested in this PR.

A screenshot of qpwgraph from my phone:

What we have now in this PR

• functioning oboe-sink and oboe-source (FYR oboe is Google's suggested replacement and wrapper of AAudio and SLES additionally with some workarounds)
• pulseaudio clients can connect
• jack clients can connect
• pipewire graph software qpwgraph that can play with software audio I/O

Test script

• Server: run three server command combo pipewire, wireplumber and pipewire-pulse.
• pulseaudio clients (vlc as an example): PULSE_SERVER=tcp:127.0.0.1:4713 vlc
PULSE_SERVER=... can be dropped after ln -sfn $PREFIX/tmp/pulse ~/.config/pulse/[device-id]-runtime
• jack clients (I use a jack-enabled version of mpv in this PR as an example): pw-jack mpv --ao=jack test.wav and then use qpwgraph to feed that into oboe sink
• alsa clients: not working currently, aplay test.wav and mpv --ao=alsa test.wav prompts errors

TODO (tracking the staging dev/pipewire-group branch)

Some of these TODOs I might have indeed raised in https://gitlab.freedesktop.org/pipewire/pipewire/-/issues/4032.

• fine-tine buffer and timeout for oboe sink and source to optimize latency, glitches and xruns (callback approach might not be very feasible unless using a custom buffer, as pipewire dev confirms no pa_asyncmsgq_send equivalence).
• let pulseaudio clients connect without PULSE_SERVER=. Soft linking $PREFIX/tmp/pulse to ~/.config/pulse/<device-id>-runtime, reimplementing of this legacy fallback in pa_get_runtime_dir (pulseaudio/src/pulsecore/core-util.c) because that part is not implemented by pipewire's module-protocol-native.c
• jack client connect to oboe sink by default (without using qpwgraph)
• accept alsa clients (some alsa device errors now) and see if -Dalsa=enabled is really needed
• stop oboe-source when it is not used
• new packages of graph software:
  • pwvucontrol (await for wireplumber-5.0 support: Support wireplumber 0.5  saivert/pwvucontrol#10)
  • helvum (await for next release new package: helvum termux-user-repository/tur#1087)
  • qpwgraph (await for QSharedMemory fix)
• gstreamer support
• a proper pthread_cancel patch to pipewire-jack.c. Currently jack client does not get killed properly when mpv is still connected and pipewire exits
• replicate Archlinux's pipewire-jack subpackage (so it don't need to run command with pw-jack), seems requiring jack2 or at least its /include files (see Archlinux build script)
• replicate Archlinux's pipewire-alsa subpackage
• register all audio devices in pipewire (help: requires calling Java AudioManager.getDevices())
• remove module-aaudio-sink and module-aaudio-source as they are more for testing
• rebuild and test X11 packages compatibility with pipewire (jack / alsa) sink and source.
  • ardour
  • audacity
  • shotcut
  • chromium
  • firefox
  • mpv-x

Off topic thought

Off topic but it would really be great to have a metapackage like termux-desktop to fire up the best practice software at once: Termux:X11, a DE, pipewire (this is where include the command combo pipewire, wireplumber and pipewire-pulse), CUPS, system-wide dbus, IME, and even mesa (and perhaps a functioning libusb in the future). The script should also clean everything up after user logs out or intercepts in the terminal with Ctrl+C. It could really save everyone's time for reinventing the wheel (for example, a usual Linux user may take quite some time to figure that three command combos to fire up a working pipwire, because pipewire is usually what Linux distros have done for us). It alsp helps to align a baseline settings for community's discussion and promote to new Termux users who may just want a full-fledged working Linux desktop.

[tomty89]

While it's unlikely I would devote myself to writing any code for this, I have some ideas for you that may or may not help.

I don't really know pipewire much, but it seems to me that a more "proper" way to implement this would be writing an "SPA plugin" (for oobe), but maybe writing "modules" is fine as well (maybe they don't really differ and it just depends on the complexity of your "implementation"; but I feel like you may want to figure it out first)

callback approach might not be very feasible unless using a custom buffer, as pipewire dev confirms no pa_asyncmsgq_send equivalence

Does pipewire by any chance provide pa_asyncmsgq_send equivalence?

No. I think you could implement some kind of buffering in the pipewire callback, then consume that buffer in the aaudio callback.

I think you might have asked the wrong question, and/or they didn't get what you really want to know.

pa_asyncmsgq_send is just a way to tell "the pulse side" of the sink to "do something", such as to copy a chunk of audio data to an available buffer offered from "the Android side" (through the means of calling the callback).

What you need isn't a similar "message queue", but really just any way that allows you to pass a buffer pointer and the number of frames the Android side wants to the pipewire side, and ask pipewire to do the copy. (sles is a bit different, due to the design of the API; it expects you to pass a buffer pointer to it and tell it how much data it holds; note that it's still "pulling" though; just that it doesn't create the buffer for you or tell you how much it wants; I think oboe offers an interface that is more similar to aaudio)

(The process_render function is really what you want to look at in the pulse aaudio and sles sinks.)

The sles and aaudio sink modules in pulseaudio work a lot like the jack sink module (from upstream). Therefore, I think at least potentially, you may want to implement the oboe support in a way like the jack tunnel module / client (what allows you to connect pipewire to a "real" jack server) / SPA plugin in pipewire.

Either way, the point is, I think you'll still want to implement it in the "pull" manner anyway (as in, let the Android side "request" audio data from pipewire), instead of making pipewire "push" audio data to it. (AFAIK, you will eventually find that it's not possible to make it work well for different devices or audio device type with the latter approach. I could be wrong though.)

It does seem that "architecturally" it would be significantly more complicated in pipewire (due to the design of it?) than in pulse. Either way, you may want to look at these code lines / sections first to get some idea regarding the "flow":

https://gitlab.freedesktop.org/pipewire/pipewire/-/blob/1.1.83/spa/plugins/jack/plugin.c?ref_type=tags#L24
https://gitlab.freedesktop.org/pipewire/pipewire/-/blob/1.1.83/spa/plugins/jack/jack-device.c?ref_type=tags#L111

As you can see, the JACK client in pipewire also register a callback to the JACK server:
https://gitlab.freedesktop.org/pipewire/pipewire/-/blob/1.1.83/spa/plugins/jack/jack-client.c?ref_type=tags#L76

https://gitlab.freedesktop.org/pipewire/pipewire/-/blob/1.1.83/spa/plugins/jack/jack-client.c?ref_type=tags#L9
https://gitlab.freedesktop.org/pipewire/pipewire/-/blob/1.1.83/spa/plugins/jack/jack-sink.c?ref_type=tags#L372
https://gitlab.freedesktop.org/pipewire/pipewire/-/blob/1.1.83/spa/plugins/jack/jack-sink.c?ref_type=tags#L367

I believe the last linked line would result in this method to be called:
https://gitlab.freedesktop.org/pipewire/pipewire/-/blob/1.1.83/spa/plugins/jack/jack-sink.c?ref_type=tags#L789

And finally, what the callback registered to Android side really need to trigger:
https://gitlab.freedesktop.org/pipewire/pipewire/-/blob/1.1.83/spa/plugins/jack/jack-sink.c?ref_type=tags#L767

---

It might be worth highlighting these lines to show that it is obviously possible to pass stuff from the callback to the actual function that calls the data copying function:

https://gitlab.freedesktop.org/pipewire/pipewire/-/blob/1.1.83/spa/plugins/jack/jack-client.c?ref_type=tags#L19
https://gitlab.freedesktop.org/pipewire/pipewire/-/blob/1.1.83/spa/plugins/jack/jack-sink.c?ref_type=tags#L751

---

One thing I do want to note is that, AFAICT, pa_asyncmsgq_send seems to be a "synchronous send", i.e., it returns after the message sent is processed, while in the pipewire jack client code, it seems that the copying of the audio data is triggered in an asynchronous manner, which might not be acceptable to the Android side (i.e., the callback should not return before the buffer is filled). I don't know if by design it is possible implement something synchronous in pipewire, so maybe in this sense, there is indeed "no pa_asyncmsgq_send equivalent in pipewire".

[tomty89]

Actually, maybe what you need to do is just eliminate (some of) the "client" bits (since in the Android case we are not exposed a single server that handles both audio input and audio output), but keep the "node" bits (instead of implementing the sink and source as "streams"), then open and start the aaudio / oboe stream at the very end of impl_init of the node (sink / source), which might actually allow you to just make the callback a wrapper of spa_node_process, which seems to be the synchronous way to trigger impl_node_process, which is in turn where the data copying should be triggered. (You'll still need to keep the bits that allow you to have some "node-specific data" where you can store the buffer pointer and the number of frames requested in each callback call for the necessary usage in impl_node_process.)

[twaik]

Probably some typo here. module-aaudio-sink.c and module-aaudio.sink.cpp?

[knyipab]

Yes, I was aware of the typo and it was fixed it in the below dev branch. This branch is functional (uses oboe, accepts alsa/jack/pulseaudio clients and works with qpwgraph) and is much better than the existing pipewire package sitting in termux-packages.
https://github.com/termux/termux-packages/compare/master...knyipab:termux-packages:dev/pipewire-group?expand=1

But as indicated in the TODO list, it is far from being perfect. Most importantly, I do not have time to complete this work (likely will never have time to work on it). If you prefer, I can tidy it up a little bit for merging and at least some progress will be made. But you know, that is probably not a good consideration for maintainer.

[TomJo2000]

This is a huge amount of work already.
I think it'd be great if we can get that reorganized and merged, even if it's not the full solution.

Thank you a lot for putting in the time and effort.

[cbix]

As an active user of PW on desktop I'm quite happy about this work so far and would gladly help it move forward :)
Some initial feedback without having tested anything:

jack client connect to oboe sink by default (without using qpwgraph)

IMO not necessary, in the pro-audio/jack world it's not expected that apps connect automatically by default, since often you'd manually want to wire them in a modular way (like audio plugins) or they internally allow the user to select an output (e.g. ardour).

register all audio devices in pipewire (help: requires calling Java AudioManager.getDevices())

Sounds reasonable, but is it actually possible to play back on multiple output devices simultaneously on Android?

Further thoughts:

1. IIRC Oboe allows setting a devices "performance" to low latency, maybe it could be possible to make use of this when setting a device's profile to Pro Audio.
2. MIDI support: likely requires Android MIDI API access from the Termux app or creating a companion app for MIDI support. Official example apps are here and here (UMP/MIDI 2.0). Worth noting MIDI 2.0/UMP is only supported since Android 13 and ALSA so far, but neither in PW nor JACK, so starting with (backwards compatible) legacy MIDI seems reasonable. Anyway I think it makes sense to not include that here for now :)
3. Camera support for e.g. pw-v4l2, might also work by creating a companion app?

Anyway thanks for all your work, I'll look into compiling and testing this branch soon!