Skip to content

Audio device controls for mobile native audio device selection #3270

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 47 commits into from
May 22, 2025
Merged
Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
5632810
Add custom audio renderer to only render joined participants & add io…
toger5 May 14, 2025
6b8c620
Add tests
toger5 May 14, 2025
18a59dd
use optional audio context and effect to initiate it + review
toger5 May 14, 2025
53adfa4
WIP
robintown Apr 29, 2025
f9b04ae
temp
toger5 May 14, 2025
f69c753
add earpice mode
toger5 May 14, 2025
6b39d0a
turn on url flag by default
toger5 May 14, 2025
86beaeb
apply mute from mobile controls
toger5 May 14, 2025
c8091ac
Quickfix for testing
toger5 May 15, 2025
7fa534d
refactor
toger5 May 15, 2025
610e792
rename setOutputDevices-> setAvailableOutputDevices
toger5 May 15, 2025
abd66f5
fix mute all audio via controls
toger5 May 15, 2025
2012b09
review cleanup
toger5 May 16, 2025
7227c7b
Merge branch 'livekit' into robin/audio-output-controls
toger5 May 16, 2025
7a4c189
test for mute all audio
toger5 May 16, 2025
acaf69c
add change audio button with callback on ios
toger5 May 16, 2025
abf683f
Hide the input list on both, android+ios.
toger5 May 16, 2025
35963bb
Add flags to optimize EC device handling
toger5 May 16, 2025
7f4b0a3
`isBluetooth` -> `isExternalHeadset`
toger5 May 16, 2025
01a2cd1
bump livekit client (the current version has an issue on safari)
toger5 May 16, 2025
c22e0cb
better logging
toger5 May 19, 2025
2946b30
fix no audio thinko.
toger5 May 19, 2025
956b7fc
actually test the impl
toger5 May 19, 2025
5d6ec19
Allow some controls to be set before the call view is loaded.
toger5 May 19, 2025
6d0697c
inform ios about earpice mode
toger5 May 19, 2025
d94feaa
smaller diff
toger5 May 19, 2025
e8c6d79
logger instead of native window picker
toger5 May 19, 2025
d7e0abc
remove the whole button on click logic
toger5 May 19, 2025
ed234a1
change label on button
toger5 May 19, 2025
7fd7dc3
use normal button
toger5 May 19, 2025
aa00a95
ITS A FORM NOOOOO
toger5 May 19, 2025
fb95ba2
make the button prettty again
toger5 May 19, 2025
c11a37c
back to non translated label
toger5 May 19, 2025
0412629
fix start with for output devices.
toger5 May 20, 2025
1cf11b9
Back to translated button
toger5 May 20, 2025
a1759a4
rename everything to `controlledMediaDevices` to make it consistent w…
toger5 May 20, 2025
435a7d0
earpice -> earpiece
toger5 May 20, 2025
a056a28
review
toger5 May 21, 2025
ab9dfc7
Comment to explain the usage/impact of: `controlledMediaDevices`
toger5 May 22, 2025
323e088
use js-sdk branch as in livekit branch
toger5 May 22, 2025
4eb8674
Merge branch 'livekit' into robin/audio-output-controls
toger5 May 22, 2025
9f84a5c
Deprecate old naming and introduce new words
toger5 May 22, 2025
fa0b521
Update docs/controls.md
toger5 May 22, 2025
f0403c8
also add non deprecated audio url parameter
toger5 May 22, 2025
269565d
Update docs/url-params.md
toger5 May 22, 2025
045d861
Merge branch 'livekit' into robin/audio-output-controls
robintown May 22, 2025
78c59bb
Fix formatting
robintown May 22, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion docs/controls.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,18 @@
# Global JS controls

A few aspects of Element Call's interface can be controlled through a global API on the `window`:
A few aspects of Element Call's interface can be controlled through a global API on the `window`.

## Picture-in-picture

- `controls.canEnterPip(): boolean` Determines whether it's possible to enter picture-in-picture mode.
- `controls.enablePip(): void` Puts the call interface into picture-in-picture mode. Throws if not in a call.
- `controls.disablePip(): void` Takes the call interface out of picture-in-picture mode, restoring it to its natural display mode. Throws if not in a call.

## Audio output devices

These functions must be used in conjunction with the `controlledOutput` URL parameter in order to have any effect.

- `controls.setOutputDevices(devices: { id: string, name: string, forEarpiece?: boolean }[]): void` Sets the list of available audio outputs. `forEarpiece` is used on ios only.
It flags the device that should be used if the user selects earpice mode. This should be the main (stereo loudspeaker) of the device.
- `controls.onOutputDeviceSelect: ((id: string) => void) | undefined` Callback called whenever the user or application selects a new audio output.
- `controls.setOutputEnabled(enabled: boolean)` Enables/disables all audio output from the application. This can be useful for temporarily pausing audio while the controlling application is switching output devices. Output is enabled by default.
1 change: 1 addition & 0 deletions docs/url-params.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ These parameters are relevant to both [widget](./embedded-standalone.md) and [st
| `lang` | [BCP 47](https://www.rfc-editor.org/info/bcp47) code | No | No | The language the app should use. |
| `password` | | No | No | E2EE password when using a shared secret. (For individual sender keys in embedded mode this is not required.) |
| `perParticipantE2EE` | `true` or `false` | No, defaults to `false` | No, defaults to `false` | Enables per participant encryption with Keys exchanged over encrypted matrix room messages. |
| `controlledOutput` | `true` or `false` | No, defaults to `false` | No, defaults to `false` | Whether the [global JS controls for audio output devices](./controls.md#audio-devices) should be enabled, allowing the list of output devices to be controlled by the app hosting Element Call. |
| `roomId` | [Matrix Room ID](https://spec.matrix.org/v1.12/appendices/#room-ids) | Yes | No | Anything about what room we're pointed to should be from useRoomIdentifier which parses the path and resolves alias with respect to the default server name, however roomId is an exception as we need the room ID in embedded widget mode, and not the room alias (or even the via params because we are not trying to join it). This is also not validated, where it is in `useRoomIdentifier()`. |
| `showControls` | `true` or `false` | No, defaults to `true` | No, defaults to `true` | Displays controls like mute, screen-share, invite, and hangup buttons during a call. |
| `skipLobby` (deprecated: use `intent` instead) | `true` or `false` | No. If `intent` is explicitly `start_call` then defaults to `true`. Otherwise defaults to `false` | No, defaults to `false` | Skips the lobby to join a call directly, can be combined with preload in widget. When `true` the audio and video inputs will be muted by default. (This means there currently is no way to start without muted video if one wants to skip the lobby. Also not in widget mode.) |
Expand Down
2 changes: 2 additions & 0 deletions locales/en/app.json
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@
"video": "Video"
},
"developer_mode": {
"always_show_iphone_earpiece": "Show iPhone earpiece option on all platforms",
"crypto_version": "Crypto version: {{version}}",
"debug_tile_layout_label": "Debug tile layout",
"device_id": "Device ID: {{id}}",
Expand Down Expand Up @@ -174,6 +175,7 @@
"camera_numbered": "Camera {{n}}",
"default": "Default",
"default_named": "Default <2>({{name}})</2>",
"earpiece": "Earpiece",
"microphone": "Microphone",
"microphone_numbered": "Microphone {{n}}",
"speaker": "Speaker",
Expand Down
2 changes: 1 addition & 1 deletion playwright/access.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ test("Sign up a new account, then login, then logout", async ({ browser }) => {

// logout
await returningUserPage.getByTestId("usermenu_open").click();
await returningUserPage.locator('[data-test-id="usermenu_logout"]').click();
await returningUserPage.locator('[data-testid="usermenu_logout"]').click();

await expect(
returningUserPage.getByRole("link", { name: "Log In" }),
Expand Down
50 changes: 29 additions & 21 deletions src/App.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,14 @@ import { ClientProvider } from "./ClientContext";
import { ErrorPage, LoadingPage } from "./FullScreenView";
import { DisconnectedBanner } from "./DisconnectedBanner";
import { Initializer } from "./initializer";
import { MediaDevicesProvider } from "./livekit/MediaDevicesContext";
import {
ControlledOutputMediaDevicesProvider,
MediaDevicesProvider,
} from "./livekit/MediaDevicesContext";
import { widget } from "./widget";
import { useTheme } from "./useTheme";
import { ProcessorProvider } from "./livekit/TrackProcessorContext";
import { useUrlParams } from "./UrlParams";

const SentryRoute = Sentry.withSentryReactRouterV7Routing(Route);

Expand Down Expand Up @@ -51,6 +55,7 @@ const ThemeProvider: FC<SimpleProviderProps> = ({ children }) => {
};

export const App: FC = () => {
const { controlledOutput } = useUrlParams();
const [loaded, setLoaded] = useState(false);
useEffect(() => {
Initializer.init()
Expand All @@ -62,6 +67,20 @@ export const App: FC = () => {
.catch(logger.error);
});

const inner = (
<Sentry.ErrorBoundary
fallback={(error) => <ErrorPage error={error} widget={widget} />}
>
<DisconnectedBanner />
<Routes>
<SentryRoute path="/" element={<HomePage />} />
<SentryRoute path="/login" element={<LoginPage />} />
<SentryRoute path="/register" element={<RegisterPage />} />
<SentryRoute path="*" element={<RoomPage />} />
</Routes>
</Sentry.ErrorBoundary>
);

return (
// eslint-disable-next-line @typescript-eslint/ban-ts-comment
// @ts-ignore
Expand All @@ -72,26 +91,15 @@ export const App: FC = () => {
{loaded ? (
<Suspense fallback={null}>
<ClientProvider>
<MediaDevicesProvider>
<ProcessorProvider>
<Sentry.ErrorBoundary
fallback={(error) => (
<ErrorPage error={error} widget={widget} />
)}
>
<DisconnectedBanner />
<Routes>
<SentryRoute path="/" element={<HomePage />} />
<SentryRoute path="/login" element={<LoginPage />} />
<SentryRoute
path="/register"
element={<RegisterPage />}
/>
<SentryRoute path="*" element={<RoomPage />} />
</Routes>
</Sentry.ErrorBoundary>
</ProcessorProvider>
</MediaDevicesProvider>
<ProcessorProvider>
{controlledOutput ? (
<ControlledOutputMediaDevicesProvider>
{inner}
</ControlledOutputMediaDevicesProvider>
) : (
<MediaDevicesProvider>{inner}</MediaDevicesProvider>
)}
</ProcessorProvider>
</ClientProvider>
</Suspense>
) : (
Expand Down
13 changes: 9 additions & 4 deletions src/UrlParams.ts
Original file line number Diff line number Diff line change
Expand Up @@ -124,9 +124,15 @@ export interface UrlParams {
*/
password: string | null;
/**
* Whether we the app should use per participant keys for E2EE.
* Whether the app should use per participant keys for E2EE.
*/
perParticipantE2EE: boolean;
/**
* Whether the global JS controls for audio output devices should be enabled,
* allowing the list of output devices to be controlled by the app hosting
* Element Call.
*/
controlledOutput: boolean;
/**
* Setting this flag skips the lobby and brings you in the call directly.
* In the widget this can be combined with preload to pass the device settings
Expand Down Expand Up @@ -156,19 +162,16 @@ export interface UrlParams {
* creating a spa link.
*/
homeserver: string | null;

/**
* The user's intent with respect to the call.
* e.g. if they clicked a Start Call button, this would be `start_call`.
* If it was a Join Call button, it would be `join_existing`.
*/
intent: string | null;

/**
* The rageshake submit URL. This is only used in the embedded package of Element Call.
*/
rageshakeSubmitUrl: string | null;

/**
* The Sentry DSN. This is only used in the embedded package of Element Call.
*/
Expand Down Expand Up @@ -281,6 +284,8 @@ export const getUrlParams = (
fontScale: Number.isNaN(fontScale) ? null : fontScale,
allowIceFallback: parser.getFlagParam("allowIceFallback"),
perParticipantE2EE: parser.getFlagParam("perParticipantE2EE"),
// TODO this should not default to true!
controlledOutput: parser.getFlagParam("controlledMediaDevices", true),
skipLobby: parser.getFlagParam(
"skipLobby",
isWidget && intent === UserIntent.StartNewCall,
Expand Down
2 changes: 1 addition & 1 deletion src/UserMenu.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,7 @@ export const UserMenu: FC<Props> = ({
key={key}
Icon={Icon}
label={label}
data-test-id={dataTestid}
data-testid={dataTestid}
onSelect={() => onAction(key)}
/>
))}
Expand Down
27 changes: 24 additions & 3 deletions src/controls.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,23 @@ Please see LICENSE in the repository root for full details.
import { Subject } from "rxjs";

export interface Controls {
canEnterPip: () => boolean;
enablePip: () => void;
disablePip: () => void;
canEnterPip(): boolean;
enablePip(): void;
disablePip(): void;
setOutputDevices(devices: OutputDevice[]): void;
onOutputDeviceSelect?: (id: string) => void;
setOutputEnabled(enabled: boolean): void;
}

export interface OutputDevice {
id: string;
name: string;
forEarpiece?: boolean;
}

export const setPipEnabled$ = new Subject<boolean>();
export const setOutputDevices$ = new Subject<OutputDevice[]>();
export const setOutputEnabled$ = new Subject<boolean>();

window.controls = {
canEnterPip(): boolean {
Expand All @@ -27,4 +38,14 @@ window.controls = {
if (!setPipEnabled$.observed) throw new Error("No call is running");
setPipEnabled$.next(false);
},
setOutputDevices(devices: OutputDevice[]): void {
if (!setOutputDevices$.observed)
throw new Error("Output controls are disabled");
setOutputDevices$.next(devices);
},
setOutputEnabled(enabled: boolean): void {
if (!setOutputEnabled$.observed)
throw new Error("Output controls are disabled");
setOutputEnabled$.next(enabled);
},
};
104 changes: 104 additions & 0 deletions src/livekit/MatrixAudioRenderer.test.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
/*
Copyright 2023, 2024 New Vector Ltd.

SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/

import { afterEach, beforeEach, expect, it, vi } from "vitest";
import { render } from "@testing-library/react";
import { type CallMembership } from "matrix-js-sdk/lib/matrixrtc";
import {
getTrackReferenceId,
type TrackReference,
} from "@livekit/components-core";
import { type RemoteAudioTrack } from "livekit-client";
import { type ReactNode } from "react";
import { useTracks } from "@livekit/components-react";

import { testAudioContext } from "../useAudioContext.test";
import * as MediaDevicesContext from "./MediaDevicesContext";
import { MatrixAudioRenderer } from "./MatrixAudioRenderer";
import { mockTrack } from "../utils/test";

export const TestAudioContextConstructor = vi.fn(() => testAudioContext);

beforeEach(() => {
vi.stubGlobal("AudioContext", TestAudioContextConstructor);
});

afterEach(() => {
vi.unstubAllGlobals();
vi.clearAllMocks();
});

vi.mock("@livekit/components-react", async (importOriginal) => {
return {
...(await importOriginal()),
AudioTrack: (props: { trackRef: TrackReference }): ReactNode => {
return (
<audio data-testid={"audio"}>
{getTrackReferenceId(props.trackRef)}
</audio>
);
},
useTracks: vi.fn(),
};
});

const tracks = [mockTrack("test:123")];
vi.mocked(useTracks).mockReturnValue(tracks);

it("should render for member", () => {
const { container, queryAllByTestId } = render(
<MatrixAudioRenderer
members={[{ sender: "test", deviceId: "123" }] as CallMembership[]}
/>,
);
expect(container).toBeTruthy();
expect(queryAllByTestId("audio")).toHaveLength(1);
});
it("should not render without member", () => {
const { container, queryAllByTestId } = render(
<MatrixAudioRenderer
members={[{ sender: "othermember", deviceId: "123" }] as CallMembership[]}
/>,
);
expect(container).toBeTruthy();
expect(queryAllByTestId("audio")).toHaveLength(0);
});

it("should not setup audioContext gain and pan if there is no need to.", () => {
render(
<MatrixAudioRenderer
members={[{ sender: "test", deviceId: "123" }] as CallMembership[]}
/>,
);
const audioTrack = tracks[0].publication.track! as RemoteAudioTrack;

expect(audioTrack.setAudioContext).toHaveBeenCalledTimes(1);
expect(audioTrack.setAudioContext).toHaveBeenCalledWith(undefined);
expect(audioTrack.setWebAudioPlugins).toHaveBeenCalledTimes(1);
expect(audioTrack.setWebAudioPlugins).toHaveBeenCalledWith([]);

expect(testAudioContext.gain.gain.value).toEqual(1);
expect(testAudioContext.pan.pan.value).toEqual(0);
});
it("should setup audioContext gain and pan", () => {
vi.spyOn(MediaDevicesContext, "useEarpieceAudioConfig").mockReturnValue({
pan: 1,
volume: 0.1,
});
render(
<MatrixAudioRenderer
members={[{ sender: "test", deviceId: "123" }] as CallMembership[]}
/>,
);

const audioTrack = tracks[0].publication.track! as RemoteAudioTrack;
expect(audioTrack.setAudioContext).toHaveBeenCalled();
expect(audioTrack.setWebAudioPlugins).toHaveBeenCalled();

expect(testAudioContext.gain.gain.value).toEqual(0.1);
expect(testAudioContext.pan.pan.value).toEqual(1);
});
Loading
Loading