Skip to content

Avatar support + remove krisp #14

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 13 commits into from
Apr 23, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file modified .github/assets/screenshot.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@

# Swift Voice Assistant

This is a starter template for [LiveKit Agents](https://docs.livekit.io/agents/overview/) that provides a simple voice interface using the [LiveKit Swift SDK](https://github.com/livekit/client-sdk-swift).
This is a starter template for [LiveKit Agents](https://docs.livekit.io/agents/overview/) that provides a simple voice interface using the [LiveKit Swift SDK](https://github.com/livekit/client-sdk-swift). It supports [voice](https://docs.livekit.io/agents/start/voice-ai), [transcriptions](https://docs.livekit.io/agents/build/text/), and [virtual avatars](https://docs.livekit.io/agents/integrations/avatar/).

This template is comaptible with iOS, iPadOS, macOS, and visionOS and is free for you to use or modify as you see fit.
This template is compatible with iOS, iPadOS, macOS, and visionOS and is free for you to use or modify as you see fit.

<img src="./.github/assets/screenshot.png" alt="Voice Assistant Screenshot" height="500">

Expand All @@ -22,7 +22,7 @@ lk app create --template voice-assistant-swift --sandbox <token_server_sandbox_i

Built and run the app from Xcode by opening `VoiceAssistant.xcodeproj`. You may need to adjust your app signing settings to run the app on your device.

You'll also need an agent to speak with. Try our sample voice assistant agent for [Python](https://github.com/livekit-examples/voice-pipeline-agent-python), [Node.js](https://github.com/livekit-examples/voice-pipeline-agent-node), or [create your own from scratch](https://docs.livekit.io/agents/quickstart/).
You'll also need an agent to speak with. Try our [voice AI quickstart](https://docs.livekit.io/agents/start/voice-ai) for the easiest way to get started.

> [!NOTE]
> To setup without the LiveKit CLI, clone the repository and then either create a `VoiceAssistant/.env.xcconfig` with a `LIVEKIT_SANDBOX_ID` (if using a [Sandbox Token Server](https://cloud.livekit.io/projects/p_/sandbox/templates/token-server)), or open `TokenService.swift` and add your [manually generated](#token-generation) URL and token.
Expand All @@ -33,10 +33,10 @@ In a production environment, you will be responsible for developing a solution t

## Chat transcription

The app uses LiveKit [text stream](https://docs.livekit.io/agents/v1/build/text/) functionality to deliver transcription events from the agent. It requires some client-side processing to aggregate the partial results into messages. `TranscriptionStreamReceiver` is responsible for this aggregation. It buffers stream chunks and publishes complete messages when the transcription is finished. Messages have unique IDs and timestamps to help with ordering and display in the UI.
The app supports agent [transcriptions](https://docs.livekit.io/agents/build/text/). It requires some client-side processing to aggregate the partial results into messages. `TranscriptionStreamReceiver` is responsible for this aggregation. It buffers stream chunks and publishes complete messages when the transcription is finished. Messages have unique IDs and timestamps to help with ordering and display in the UI.

> [!NOTE]
> Text streams are fully supported in LiveKit Agents v1, for v0.x, you'll need to use legacy [transcription events](https://docs.livekit.io/agents/v1/build/text/#transcription-events) as shown in `TranscriptionDelegateReceiver.swift`.
> Text streams are fully supported in LiveKit Agents v1, for v0.x, you'll need to use legacy [transcription events](https://docs.livekit.io/agents/build/text/#transcription-events) as shown in `TranscriptionDelegateReceiver.swift`.

## Contributing

Expand Down
27 changes: 4 additions & 23 deletions VoiceAssistant.xcodeproj/project.pbxproj
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,8 @@
/* Begin PBXBuildFile section */
ACBC177A2D8C34B5007EE71F /* AsyncAlgorithms in Frameworks */ = {isa = PBXBuildFile; productRef = ACBC17792D8C34B5007EE71F /* AsyncAlgorithms */; };
ACFBA1DB2D8D5CBE0021202B /* Collections in Frameworks */ = {isa = PBXBuildFile; productRef = ACFBA1DA2D8D5CBE0021202B /* Collections */; };
B577A98A2D16796000B59A0B /* LiveKitKrispNoiseFilter in Frameworks */ = {isa = PBXBuildFile; platformFilters = (ios, macos, ); productRef = B577A9892D16796000B59A0B /* LiveKitKrispNoiseFilter */; };
B5E1B90F2D14E9EC00A38CB6 /* LiveKitComponents in Frameworks */ = {isa = PBXBuildFile; productRef = B5E1B90E2D14E9EC00A38CB6 /* LiveKitComponents */; };
B5E1B9122D14E9F500A38CB6 /* LiveKit in Frameworks */ = {isa = PBXBuildFile; productRef = B5E1B9112D14E9F500A38CB6 /* LiveKit */; };
B5E46D202D496CD200ACC2E6 /* LiveKitKrispNoiseFilter in Frameworks */ = {isa = PBXBuildFile; platformFilters = (ios, macos, ); productRef = B5E46D1F2D496CD200ACC2E6 /* LiveKitKrispNoiseFilter */; };
/* End PBXBuildFile section */

/* Begin PBXFileReference section */
Expand Down Expand Up @@ -49,8 +47,6 @@
buildActionMask = 2147483647;
files = (
ACFBA1DB2D8D5CBE0021202B /* Collections in Frameworks */,
B5E46D202D496CD200ACC2E6 /* LiveKitKrispNoiseFilter in Frameworks */,
B577A98A2D16796000B59A0B /* LiveKitKrispNoiseFilter in Frameworks */,
B5E1B90F2D14E9EC00A38CB6 /* LiveKitComponents in Frameworks */,
B5E1B9122D14E9F500A38CB6 /* LiveKit in Frameworks */,
ACBC177A2D8C34B5007EE71F /* AsyncAlgorithms in Frameworks */,
Expand Down Expand Up @@ -99,7 +95,6 @@
B5E1B90E2D14E9EC00A38CB6 /* LiveKitComponents */,
B5E1B9112D14E9F500A38CB6 /* LiveKit */,
B577A9892D16796000B59A0B /* LiveKitKrispNoiseFilter */,
B5E46D1F2D496CD200ACC2E6 /* LiveKitKrispNoiseFilter */,
ACBC17792D8C34B5007EE71F /* AsyncAlgorithms */,
ACFBA1DA2D8D5CBE0021202B /* Collections */,
);
Expand Down Expand Up @@ -134,7 +129,6 @@
packageReferences = (
B5E1B90D2D14E9EC00A38CB6 /* XCRemoteSwiftPackageReference "components-swift" */,
B5E1B9102D14E9F500A38CB6 /* XCRemoteSwiftPackageReference "client-sdk-swift" */,
B5E46D1E2D496CD200ACC2E6 /* XCRemoteSwiftPackageReference "swift-krisp-noise-filter" */,
ACBC17782D8C34B5007EE71F /* XCRemoteSwiftPackageReference "swift-async-algorithms" */,
ACFBA1D92D8D5CBE0021202B /* XCRemoteSwiftPackageReference "swift-collections" */,
);
Expand Down Expand Up @@ -312,8 +306,8 @@
"INFOPLIST_KEY_UILaunchScreen_Generation[sdk=iphonesimulator*]" = YES;
"INFOPLIST_KEY_UIStatusBarStyle[sdk=iphoneos*]" = UIStatusBarStyleDefault;
"INFOPLIST_KEY_UIStatusBarStyle[sdk=iphonesimulator*]" = UIStatusBarStyleDefault;
INFOPLIST_KEY_UISupportedInterfaceOrientations_iPad = "UIInterfaceOrientationPortrait UIInterfaceOrientationPortraitUpsideDown UIInterfaceOrientationLandscapeLeft UIInterfaceOrientationLandscapeRight";
INFOPLIST_KEY_UISupportedInterfaceOrientations_iPhone = "UIInterfaceOrientationPortrait UIInterfaceOrientationLandscapeLeft UIInterfaceOrientationLandscapeRight";
INFOPLIST_KEY_UISupportedInterfaceOrientations = UIInterfaceOrientationPortrait;
INFOPLIST_KEY_UISupportedInterfaceOrientations_iPad = "UIInterfaceOrientationLandscapeLeft UIInterfaceOrientationLandscapeRight UIInterfaceOrientationPortrait UIInterfaceOrientationPortraitUpsideDown";
IPHONEOS_DEPLOYMENT_TARGET = 18.2;
LD_RUNPATH_SEARCH_PATHS = "@executable_path/Frameworks";
"LD_RUNPATH_SEARCH_PATHS[sdk=macosx*]" = "@executable_path/../Frameworks";
Expand Down Expand Up @@ -355,8 +349,8 @@
"INFOPLIST_KEY_UILaunchScreen_Generation[sdk=iphonesimulator*]" = YES;
"INFOPLIST_KEY_UIStatusBarStyle[sdk=iphoneos*]" = UIStatusBarStyleDefault;
"INFOPLIST_KEY_UIStatusBarStyle[sdk=iphonesimulator*]" = UIStatusBarStyleDefault;
INFOPLIST_KEY_UISupportedInterfaceOrientations_iPad = "UIInterfaceOrientationPortrait UIInterfaceOrientationPortraitUpsideDown UIInterfaceOrientationLandscapeLeft UIInterfaceOrientationLandscapeRight";
INFOPLIST_KEY_UISupportedInterfaceOrientations_iPhone = "UIInterfaceOrientationPortrait UIInterfaceOrientationLandscapeLeft UIInterfaceOrientationLandscapeRight";
INFOPLIST_KEY_UISupportedInterfaceOrientations = UIInterfaceOrientationPortrait;
INFOPLIST_KEY_UISupportedInterfaceOrientations_iPad = "UIInterfaceOrientationLandscapeLeft UIInterfaceOrientationLandscapeRight UIInterfaceOrientationPortrait UIInterfaceOrientationPortraitUpsideDown";
IPHONEOS_DEPLOYMENT_TARGET = 18.2;
LD_RUNPATH_SEARCH_PATHS = "@executable_path/Frameworks";
"LD_RUNPATH_SEARCH_PATHS[sdk=macosx*]" = "@executable_path/../Frameworks";
Expand Down Expand Up @@ -427,14 +421,6 @@
minimumVersion = 2.3.1;
};
};
B5E46D1E2D496CD200ACC2E6 /* XCRemoteSwiftPackageReference "swift-krisp-noise-filter" */ = {
isa = XCRemoteSwiftPackageReference;
repositoryURL = "https://github.com/livekit/swift-krisp-noise-filter";
requirement = {
kind = upToNextMajorVersion;
minimumVersion = 0.0.7;
};
};
/* End XCRemoteSwiftPackageReference section */

/* Begin XCSwiftPackageProductDependency section */
Expand Down Expand Up @@ -462,11 +448,6 @@
package = B5E1B9102D14E9F500A38CB6 /* XCRemoteSwiftPackageReference "client-sdk-swift" */;
productName = LiveKit;
};
B5E46D1F2D496CD200ACC2E6 /* LiveKitKrispNoiseFilter */ = {
isa = XCSwiftPackageProductDependency;
package = B5E46D1E2D496CD200ACC2E6 /* XCRemoteSwiftPackageReference "swift-krisp-noise-filter" */;
productName = LiveKitKrispNoiseFilter;
};
/* End XCSwiftPackageProductDependency section */
};
rootObject = B5B5E3AA2D124AE00099C9BE /* Project object */;
Expand Down
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
{
"originHash" : "427c44f75b04feb1c4e9a9f495419b817da0c8b3930e9f2d6d5e7539c8d3ca40",
"originHash" : "b296e14c458c92b5325292dd05b50453c8b214c0b7935936aa5b8cf3967fbb60",
"pins" : [
{
"identity" : "client-sdk-swift",
"kind" : "remoteSourceControl",
"location" : "https://github.com/livekit/client-sdk-swift",
"state" : {
"revision" : "4fba03e8934ec538e226c362275dd78692991424",
"version" : "2.3.1"
"revision" : "4f9db591da71bfd7c556589e43963e1d777cf337",
"version" : "2.5.0"
}
},
{
Expand Down Expand Up @@ -37,15 +37,6 @@
"version" : "1.1.4"
}
},
{
"identity" : "swift-krisp-noise-filter",
"kind" : "remoteSourceControl",
"location" : "https://github.com/livekit/swift-krisp-noise-filter",
"state" : {
"revision" : "c12a41f2afc95ee67994faf2ea7c87cde6d65cc6",
"version" : "0.0.8"
}
},
{
"identity" : "swift-log",
"kind" : "remoteSourceControl",
Expand All @@ -69,8 +60,8 @@
"kind" : "remoteSourceControl",
"location" : "https://github.com/livekit/webrtc-xcframework.git",
"state" : {
"revision" : "d1a5661a412668829a22e44f875eeef109526c65",
"version" : "125.6422.24"
"revision" : "70bbd9b51f2c46333a91469c8a8be8303b0797b7",
"version" : "125.6422.28"
}
}
],
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
<?xml version="1.0" encoding="UTF-8"?>
<Scheme
LastUpgradeVersion = "1620"
version = "1.7">
<BuildAction
parallelizeBuildables = "YES"
buildImplicitDependencies = "YES"
buildArchitectures = "Automatic">
<BuildActionEntries>
<BuildActionEntry
buildForTesting = "YES"
buildForRunning = "YES"
buildForProfiling = "YES"
buildForArchiving = "YES"
buildForAnalyzing = "YES">
<BuildableReference
BuildableIdentifier = "primary"
BlueprintIdentifier = "B5B5E3B12D124AE00099C9BE"
BuildableName = "VoiceAssistant.app"
BlueprintName = "VoiceAssistant"
ReferencedContainer = "container:VoiceAssistant.xcodeproj">
</BuildableReference>
</BuildActionEntry>
</BuildActionEntries>
</BuildAction>
<TestAction
buildConfiguration = "Debug"
selectedDebuggerIdentifier = "Xcode.DebuggerFoundation.Debugger.LLDB"
selectedLauncherIdentifier = "Xcode.DebuggerFoundation.Launcher.LLDB"
shouldUseLaunchSchemeArgsEnv = "YES"
shouldAutocreateTestPlan = "YES">
</TestAction>
<LaunchAction
buildConfiguration = "Debug"
selectedDebuggerIdentifier = "Xcode.DebuggerFoundation.Debugger.LLDB"
selectedLauncherIdentifier = "Xcode.DebuggerFoundation.Launcher.LLDB"
launchStyle = "0"
useCustomWorkingDirectory = "NO"
ignoresPersistentStateOnLaunch = "NO"
debugDocumentVersioning = "YES"
debugServiceExtension = "internal"
allowLocationSimulation = "YES">
<BuildableProductRunnable
runnableDebuggingMode = "0">
<BuildableReference
BuildableIdentifier = "primary"
BlueprintIdentifier = "B5B5E3B12D124AE00099C9BE"
BuildableName = "VoiceAssistant.app"
BlueprintName = "VoiceAssistant"
ReferencedContainer = "container:VoiceAssistant.xcodeproj">
</BuildableReference>
</BuildableProductRunnable>
</LaunchAction>
<ProfileAction
buildConfiguration = "Release"
shouldUseLaunchSchemeArgsEnv = "YES"
savedToolIdentifier = ""
useCustomWorkingDirectory = "NO"
debugDocumentVersioning = "YES">
<BuildableProductRunnable
runnableDebuggingMode = "0">
<BuildableReference
BuildableIdentifier = "primary"
BlueprintIdentifier = "B5B5E3B12D124AE00099C9BE"
BuildableName = "VoiceAssistant.app"
BlueprintName = "VoiceAssistant"
ReferencedContainer = "container:VoiceAssistant.xcodeproj">
</BuildableReference>
</BuildableProductRunnable>
</ProfileAction>
<AnalyzeAction
buildConfiguration = "Debug">
</AnalyzeAction>
<ArchiveAction
buildConfiguration = "Release"
revealArchiveInOrganizer = "YES">
</ArchiveAction>
</Scheme>
32 changes: 32 additions & 0 deletions VoiceAssistant/Agent/AgentView.swift
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
import LiveKit
import LiveKitComponents
import SwiftUI

struct AgentView: View {
@EnvironmentObject private var room: Room
@EnvironmentObject private var participant: Participant

private var worker: RemoteParticipant? {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we move it to Room extension for a clearer picture?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it would need to move to a new Participant extension if anything, but our focus should be on just solving this in the SDK automatically.

room.remoteParticipants.values.first { $0.kind == .agent && $0.attributes["lk.publish_on_behalf"] == participant.identity?.stringValue }
}

private var cameraTrack: VideoTrack? {
return participant.firstCameraVideoTrack ?? worker?.firstCameraVideoTrack
}

private var micTrack: AudioTrack? {
return participant.firstAudioTrack ?? worker?.firstAudioTrack
}

var body: some View {
ZStack {
if let cameraTrack, !cameraTrack.isMuted {
SwiftUIVideoView(cameraTrack)
.clipShape(RoundedRectangle(cornerRadius: 8))
} else {
AgentBarAudioVisualizer(audioTrack: micTrack, agentState: participant.agentState, barColor: .primary, barCount: 5)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see some opportunity to merge it with ParticipantView with very similar logic.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah participantview needs to be updated to handle agent state. ideally once we make the necessary framework updates this whole sample app becomes quite simple

}
}
.id("\(participant.identity?.stringValue ?? "none")-\(cameraTrack?.sid?.stringValue ?? "none")-\(micTrack?.sid?.stringValue ?? "none")")
}
}
7 changes: 7 additions & 0 deletions VoiceAssistant/Chat/View/ChatView.swift
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,13 @@ struct ChatView: View {
}
.listStyle(.plain)
.scrollIndicators(.hidden)
.mask(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 nice

LinearGradient(
gradient: Gradient(colors: [.clear, .black, .black]),
startPoint: .top,
endPoint: .init(x: 0.5, y: 0.2)
)
)
}
.animation(.default, value: viewModel.messages)
.alert("Error while connecting to Chat", isPresented: .constant(viewModel.error != nil)) {
Expand Down
17 changes: 0 additions & 17 deletions VoiceAssistant/Connect/StatusView.swift

This file was deleted.

Loading