Skip to content

Audio device controls for mobile native audio device selection #3288

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 22, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 15 additions & 1 deletion docs/controls.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,21 @@
# Global JS controls

A few aspects of Element Call's interface can be controlled through a global API on the `window`:
A few aspects of Element Call's interface can be controlled through a global API on the `window`.

## Picture-in-picture

- `controls.canEnterPip(): boolean` Determines whether it's possible to enter picture-in-picture mode.
- `controls.enablePip(): void` Puts the call interface into picture-in-picture mode. Throws if not in a call.
- `controls.disablePip(): void` Takes the call interface out of picture-in-picture mode, restoring it to its natural display mode. Throws if not in a call.

## Audio devices

On mobile platforms (iOS, Android), web views do not reliably support selecting audio output devices such as the main speaker, earpiece, or headset. To address this limitation, the following functions allow the hosting application (e.g., Element Web, Element X) to manage audio devices via exposed JavaScript interfaces. These functions must be enabled using the URL parameter `controlledAudioDevices` to take effect.

- `controls.setAvailableAudioDevices(devices: { id: string, name: string, forEarpiece?: boolean, isEarpiece?: boolean isSpeaker?: boolean, isExternalHeadset?, boolean; }[]): void` Sets the list of available audio outputs. `forEarpiece` is used on iOS only.
It flags the device that should be used if the user selects earpiece mode. This should be the main stereo loudspeaker of the device.
- `controls.onAudioDeviceSelect: ((id: string) => void) | undefined` Callback called whenever the user or application selects a new audio output.
- `controls.setAudioDevice(id: string): void` Sets the selected audio device in Element Call's menu. This should be used if the OS decides to automatically switch to Bluetooth, for example.
- `controls.setAudioEnabled(enabled: boolean)` Enables/disables all audio output from the application. Output is enabled by default.
- `showNativeAudioDevicePicker: (() => void) | undefined`. Callback called whenever the user presses the output button in the settings menu.
This button is only shown on iOS. (`userAgent.includes("iPhone")`)
1 change: 1 addition & 0 deletions docs/url-params.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ These parameters are relevant to both [widget](./embedded-standalone.md) and [st
| `lang` | [BCP 47](https://www.rfc-editor.org/info/bcp47) code | No | No | The language the app should use. |
| `password` | | No | No | E2EE password when using a shared secret. (For individual sender keys in embedded mode this is not required.) |
| `perParticipantE2EE` | `true` or `false` | No, defaults to `false` | No, defaults to `false` | Enables per participant encryption with Keys exchanged over encrypted matrix room messages. |
| `controlledAudioDevices` | `true` or `false` | No, defaults to `false` | No, defaults to `false` | Whether the [global JS controls for audio devices](./controls.md#audio-devices) should be enabled, allowing the list of audio devices to be controlled by the app hosting Element Call. |
| `roomId` | [Matrix Room ID](https://spec.matrix.org/v1.12/appendices/#room-ids) | Yes | No | Anything about what room we're pointed to should be from useRoomIdentifier which parses the path and resolves alias with respect to the default server name, however roomId is an exception as we need the room ID in embedded widget mode, and not the room alias (or even the via params because we are not trying to join it). This is also not validated, where it is in `useRoomIdentifier()`. |
| `showControls` | `true` or `false` | No, defaults to `true` | No, defaults to `true` | Displays controls like mute, screen-share, invite, and hangup buttons during a call. |
| `skipLobby` (deprecated: use `intent` instead) | `true` or `false` | No. If `intent` is explicitly `start_call` then defaults to `true`. Otherwise defaults to `false` | No, defaults to `false` | Skips the lobby to join a call directly, can be combined with preload in widget. When `true` the audio and video inputs will be muted by default. (This means there currently is no way to start without muted video if one wants to skip the lobby. Also not in widget mode.) |
Expand Down
1 change: 1 addition & 0 deletions locales/en/app.json
Original file line number Diff line number Diff line change
Expand Up @@ -173,6 +173,7 @@
"devices": {
"camera": "Camera",
"camera_numbered": "Camera {{n}}",
"change_device_button": "Change audio device",
"default": "Default",
"default_named": "Default <2>({{name}})</2>",
"earpiece": "Earpiece",
Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@
"i18next-parser": "^9.1.0",
"jsdom": "^26.0.0",
"knip": "^5.27.2",
"livekit-client": "^2.11.3",
"livekit-client": "^2.13.0",
"lodash-es": "^4.17.21",
"loglevel": "^1.9.1",
"matrix-js-sdk": "github:matrix-org/matrix-js-sdk#head=develop",
Expand Down
14 changes: 13 additions & 1 deletion src/UrlParams.ts
Original file line number Diff line number Diff line change
Expand Up @@ -124,9 +124,15 @@ export interface UrlParams {
*/
password: string | null;
/**
* Whether we the app should use per participant keys for E2EE.
* Whether the app should use per participant keys for E2EE.
*/
perParticipantE2EE: boolean;
/**
* Whether the global JS controls for audio output devices should be enabled,
* allowing the list of output devices to be controlled by the app hosting
* Element Call.
*/
controlledAudioDevices: boolean;
/**
* Setting this flag skips the lobby and brings you in the call directly.
* In the widget this can be combined with preload to pass the device settings
Expand Down Expand Up @@ -173,6 +179,7 @@ export interface UrlParams {
* The Sentry DSN. This is only used in the embedded package of Element Call.
*/
sentryDsn: string | null;

/**
* The Sentry environment. This is only used in the embedded package of Element Call.
*/
Expand Down Expand Up @@ -281,6 +288,11 @@ export const getUrlParams = (
fontScale: Number.isNaN(fontScale) ? null : fontScale,
allowIceFallback: parser.getFlagParam("allowIceFallback"),
perParticipantE2EE: parser.getFlagParam("perParticipantE2EE"),
controlledAudioDevices: parser.getFlagParam(
"controlledAudioDevices",
// the deprecated property name
parser.getFlagParam("controlledMediaDevices"),
),
skipLobby: parser.getFlagParam(
"skipLobby",
isWidget && intent === UserIntent.StartNewCall,
Expand Down
72 changes: 68 additions & 4 deletions src/controls.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,55 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/

import { Subject } from "rxjs";
import { BehaviorSubject, Subject } from "rxjs";

export interface Controls {
canEnterPip: () => boolean;
enablePip: () => void;
disablePip: () => void;
canEnterPip(): boolean;
enablePip(): void;
disablePip(): void;
/** @deprecated use setAvailableAudioDevices instead*/
setAvailableOutputDevices(devices: OutputDevice[]): void;
setAvailableAudioDevices(devices: OutputDevice[]): void;
/** @deprecated use setAudioDevice instead*/
setOutputDevice(id: string): void;
setAudioDevice(id: string): void;
/** @deprecated use onAudioDeviceSelect instead*/
onOutputDeviceSelect?: (id: string) => void;
onAudioDeviceSelect?: (id: string) => void;
/** @deprecated use setAudioEnabled instead*/
setOutputEnabled(enabled: boolean): void;
setAudioEnabled(enabled: boolean): void;
/** @deprecated use showNativeAudioDevicePicker instead*/
showNativeOutputDevicePicker?: () => void;
showNativeAudioDevicePicker?: () => void;
}

export interface OutputDevice {
id: string;
name: string;
forEarpiece?: boolean;
isEarpiece?: boolean;
isSpeaker?: boolean;
isExternalHeadset?: boolean;
}

/**
* If pipMode is enabled, EC will render a adapted call view layout.
*/
export const setPipEnabled$ = new Subject<boolean>();
// BehaviorSubject since the client might set this before we have subscribed (GroupCallView still in "loading" state)
// We want the devices that have been set during loading to be available immediately once loaded.
export const availableOutputDevices$ = new BehaviorSubject<OutputDevice[]>([]);
// BehaviorSubject since the client might set this before we have subscribed (GroupCallView still in "loading" state)
// We want the device that has been set during loading to be available immediately once loaded.
export const outputDevice$ = new BehaviorSubject<string | undefined>(undefined);
/**
* This allows the os to mute the call if the user
* presses the volume down button when it is at the minimum volume.
*
* This should also be used to display a darkened overlay screen letting the user know that audio is muted.
*/
export const setAudioEnabled$ = new Subject<boolean>();

window.controls = {
canEnterPip(): boolean {
Expand All @@ -27,4 +67,28 @@ window.controls = {
if (!setPipEnabled$.observed) throw new Error("No call is running");
setPipEnabled$.next(false);
},
setAvailableAudioDevices(devices: OutputDevice[]): void {
availableOutputDevices$.next(devices);
},
setAudioDevice(id: string): void {
outputDevice$.next(id);
},
setAudioEnabled(enabled: boolean): void {
if (!setAudioEnabled$.observed)
throw new Error(
"Output controls are disabled. No setAudioEnabled$ observer",
);
setAudioEnabled$.next(enabled);
},

// wrappers for the deprecated controls fields
setOutputEnabled(enabled: boolean): void {
this.setAudioEnabled(enabled);
},
setAvailableOutputDevices(devices: OutputDevice[]): void {
this.setAvailableAudioDevices(devices);
},
setOutputDevice(id: string): void {
this.setAudioDevice(id);
},
};
Loading