Skip to content

Commit 32ee03b

Browse files
bcherrypblazej
bcherry
and
pblazej
authored
Avatar support + remove krisp (#14)
* Minimal avatar view * Disable portrait * Format * fixes * fix * wip * fixes * adjust layout * More fixes * readme * screen * add scheme --------- Co-authored-by: Błażej Pankowski <86720177+pblazej@users.noreply.github.com>
1 parent 4d03762 commit 32ee03b

File tree

10 files changed

+174
-95
lines changed

10 files changed

+174
-95
lines changed

‎.github/assets/screenshot.png

214 KB
Loading

‎README.md

+5-5
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,9 @@
22

33
# Swift Voice Assistant
44

5-
This is a starter template for [LiveKit Agents](https://docs.livekit.io/agents/overview/) that provides a simple voice interface using the [LiveKit Swift SDK](https://github.com/livekit/client-sdk-swift).
5+
This is a starter template for [LiveKit Agents](https://docs.livekit.io/agents/overview/) that provides a simple voice interface using the [LiveKit Swift SDK](https://github.com/livekit/client-sdk-swift). It supports [voice](https://docs.livekit.io/agents/start/voice-ai), [transcriptions](https://docs.livekit.io/agents/build/text/), and [virtual avatars](https://docs.livekit.io/agents/integrations/avatar/).
66

7-
This template is comaptible with iOS, iPadOS, macOS, and visionOS and is free for you to use or modify as you see fit.
7+
This template is compatible with iOS, iPadOS, macOS, and visionOS and is free for you to use or modify as you see fit.
88

99
<img src="./.github/assets/screenshot.png" alt="Voice Assistant Screenshot" height="500">
1010

@@ -22,7 +22,7 @@ lk app create --template voice-assistant-swift --sandbox <token_server_sandbox_i
2222

2323
Built and run the app from Xcode by opening `VoiceAssistant.xcodeproj`. You may need to adjust your app signing settings to run the app on your device.
2424

25-
You'll also need an agent to speak with. Try our sample voice assistant agent for [Python](https://github.com/livekit-examples/voice-pipeline-agent-python), [Node.js](https://github.com/livekit-examples/voice-pipeline-agent-node), or [create your own from scratch](https://docs.livekit.io/agents/quickstart/).
25+
You'll also need an agent to speak with. Try our [voice AI quickstart](https://docs.livekit.io/agents/start/voice-ai) for the easiest way to get started.
2626

2727
> [!NOTE]
2828
> To setup without the LiveKit CLI, clone the repository and then either create a `VoiceAssistant/.env.xcconfig` with a `LIVEKIT_SANDBOX_ID` (if using a [Sandbox Token Server](https://cloud.livekit.io/projects/p_/sandbox/templates/token-server)), or open `TokenService.swift` and add your [manually generated](#token-generation) URL and token.
@@ -33,10 +33,10 @@ In a production environment, you will be responsible for developing a solution t
3333

3434
## Chat transcription
3535

36-
The app uses LiveKit [text stream](https://docs.livekit.io/agents/v1/build/text/) functionality to deliver transcription events from the agent. It requires some client-side processing to aggregate the partial results into messages. `TranscriptionStreamReceiver` is responsible for this aggregation. It buffers stream chunks and publishes complete messages when the transcription is finished. Messages have unique IDs and timestamps to help with ordering and display in the UI.
36+
The app supports agent [transcriptions](https://docs.livekit.io/agents/build/text/). It requires some client-side processing to aggregate the partial results into messages. `TranscriptionStreamReceiver` is responsible for this aggregation. It buffers stream chunks and publishes complete messages when the transcription is finished. Messages have unique IDs and timestamps to help with ordering and display in the UI.
3737

3838
> [!NOTE]
39-
> Text streams are fully supported in LiveKit Agents v1, for v0.x, you'll need to use legacy [transcription events](https://docs.livekit.io/agents/v1/build/text/#transcription-events) as shown in `TranscriptionDelegateReceiver.swift`.
39+
> Text streams are fully supported in LiveKit Agents v1, for v0.x, you'll need to use legacy [transcription events](https://docs.livekit.io/agents/build/text/#transcription-events) as shown in `TranscriptionDelegateReceiver.swift`.
4040
4141
## Contributing
4242

‎VoiceAssistant.xcodeproj/project.pbxproj

+4-23
Original file line numberDiff line numberDiff line change
@@ -9,10 +9,8 @@
99
/* Begin PBXBuildFile section */
1010
ACBC177A2D8C34B5007EE71F /* AsyncAlgorithms in Frameworks */ = {isa = PBXBuildFile; productRef = ACBC17792D8C34B5007EE71F /* AsyncAlgorithms */; };
1111
ACFBA1DB2D8D5CBE0021202B /* Collections in Frameworks */ = {isa = PBXBuildFile; productRef = ACFBA1DA2D8D5CBE0021202B /* Collections */; };
12-
B577A98A2D16796000B59A0B /* LiveKitKrispNoiseFilter in Frameworks */ = {isa = PBXBuildFile; platformFilters = (ios, macos, ); productRef = B577A9892D16796000B59A0B /* LiveKitKrispNoiseFilter */; };
1312
B5E1B90F2D14E9EC00A38CB6 /* LiveKitComponents in Frameworks */ = {isa = PBXBuildFile; productRef = B5E1B90E2D14E9EC00A38CB6 /* LiveKitComponents */; };
1413
B5E1B9122D14E9F500A38CB6 /* LiveKit in Frameworks */ = {isa = PBXBuildFile; productRef = B5E1B9112D14E9F500A38CB6 /* LiveKit */; };
15-
B5E46D202D496CD200ACC2E6 /* LiveKitKrispNoiseFilter in Frameworks */ = {isa = PBXBuildFile; platformFilters = (ios, macos, ); productRef = B5E46D1F2D496CD200ACC2E6 /* LiveKitKrispNoiseFilter */; };
1614
/* End PBXBuildFile section */
1715

1816
/* Begin PBXFileReference section */
@@ -49,8 +47,6 @@
4947
buildActionMask = 2147483647;
5048
files = (
5149
ACFBA1DB2D8D5CBE0021202B /* Collections in Frameworks */,
52-
B5E46D202D496CD200ACC2E6 /* LiveKitKrispNoiseFilter in Frameworks */,
53-
B577A98A2D16796000B59A0B /* LiveKitKrispNoiseFilter in Frameworks */,
5450
B5E1B90F2D14E9EC00A38CB6 /* LiveKitComponents in Frameworks */,
5551
B5E1B9122D14E9F500A38CB6 /* LiveKit in Frameworks */,
5652
ACBC177A2D8C34B5007EE71F /* AsyncAlgorithms in Frameworks */,
@@ -99,7 +95,6 @@
9995
B5E1B90E2D14E9EC00A38CB6 /* LiveKitComponents */,
10096
B5E1B9112D14E9F500A38CB6 /* LiveKit */,
10197
B577A9892D16796000B59A0B /* LiveKitKrispNoiseFilter */,
102-
B5E46D1F2D496CD200ACC2E6 /* LiveKitKrispNoiseFilter */,
10398
ACBC17792D8C34B5007EE71F /* AsyncAlgorithms */,
10499
ACFBA1DA2D8D5CBE0021202B /* Collections */,
105100
);
@@ -134,7 +129,6 @@
134129
packageReferences = (
135130
B5E1B90D2D14E9EC00A38CB6 /* XCRemoteSwiftPackageReference "components-swift" */,
136131
B5E1B9102D14E9F500A38CB6 /* XCRemoteSwiftPackageReference "client-sdk-swift" */,
137-
B5E46D1E2D496CD200ACC2E6 /* XCRemoteSwiftPackageReference "swift-krisp-noise-filter" */,
138132
ACBC17782D8C34B5007EE71F /* XCRemoteSwiftPackageReference "swift-async-algorithms" */,
139133
ACFBA1D92D8D5CBE0021202B /* XCRemoteSwiftPackageReference "swift-collections" */,
140134
);
@@ -312,8 +306,8 @@
312306
"INFOPLIST_KEY_UILaunchScreen_Generation[sdk=iphonesimulator*]" = YES;
313307
"INFOPLIST_KEY_UIStatusBarStyle[sdk=iphoneos*]" = UIStatusBarStyleDefault;
314308
"INFOPLIST_KEY_UIStatusBarStyle[sdk=iphonesimulator*]" = UIStatusBarStyleDefault;
315-
INFOPLIST_KEY_UISupportedInterfaceOrientations_iPad = "UIInterfaceOrientationPortrait UIInterfaceOrientationPortraitUpsideDown UIInterfaceOrientationLandscapeLeft UIInterfaceOrientationLandscapeRight";
316-
INFOPLIST_KEY_UISupportedInterfaceOrientations_iPhone = "UIInterfaceOrientationPortrait UIInterfaceOrientationLandscapeLeft UIInterfaceOrientationLandscapeRight";
309+
INFOPLIST_KEY_UISupportedInterfaceOrientations = UIInterfaceOrientationPortrait;
310+
INFOPLIST_KEY_UISupportedInterfaceOrientations_iPad = "UIInterfaceOrientationLandscapeLeft UIInterfaceOrientationLandscapeRight UIInterfaceOrientationPortrait UIInterfaceOrientationPortraitUpsideDown";
317311
IPHONEOS_DEPLOYMENT_TARGET = 18.2;
318312
LD_RUNPATH_SEARCH_PATHS = "@executable_path/Frameworks";
319313
"LD_RUNPATH_SEARCH_PATHS[sdk=macosx*]" = "@executable_path/../Frameworks";
@@ -355,8 +349,8 @@
355349
"INFOPLIST_KEY_UILaunchScreen_Generation[sdk=iphonesimulator*]" = YES;
356350
"INFOPLIST_KEY_UIStatusBarStyle[sdk=iphoneos*]" = UIStatusBarStyleDefault;
357351
"INFOPLIST_KEY_UIStatusBarStyle[sdk=iphonesimulator*]" = UIStatusBarStyleDefault;
358-
INFOPLIST_KEY_UISupportedInterfaceOrientations_iPad = "UIInterfaceOrientationPortrait UIInterfaceOrientationPortraitUpsideDown UIInterfaceOrientationLandscapeLeft UIInterfaceOrientationLandscapeRight";
359-
INFOPLIST_KEY_UISupportedInterfaceOrientations_iPhone = "UIInterfaceOrientationPortrait UIInterfaceOrientationLandscapeLeft UIInterfaceOrientationLandscapeRight";
352+
INFOPLIST_KEY_UISupportedInterfaceOrientations = UIInterfaceOrientationPortrait;
353+
INFOPLIST_KEY_UISupportedInterfaceOrientations_iPad = "UIInterfaceOrientationLandscapeLeft UIInterfaceOrientationLandscapeRight UIInterfaceOrientationPortrait UIInterfaceOrientationPortraitUpsideDown";
360354
IPHONEOS_DEPLOYMENT_TARGET = 18.2;
361355
LD_RUNPATH_SEARCH_PATHS = "@executable_path/Frameworks";
362356
"LD_RUNPATH_SEARCH_PATHS[sdk=macosx*]" = "@executable_path/../Frameworks";
@@ -427,14 +421,6 @@
427421
minimumVersion = 2.3.1;
428422
};
429423
};
430-
B5E46D1E2D496CD200ACC2E6 /* XCRemoteSwiftPackageReference "swift-krisp-noise-filter" */ = {
431-
isa = XCRemoteSwiftPackageReference;
432-
repositoryURL = "https://github.com/livekit/swift-krisp-noise-filter";
433-
requirement = {
434-
kind = upToNextMajorVersion;
435-
minimumVersion = 0.0.7;
436-
};
437-
};
438424
/* End XCRemoteSwiftPackageReference section */
439425

440426
/* Begin XCSwiftPackageProductDependency section */
@@ -462,11 +448,6 @@
462448
package = B5E1B9102D14E9F500A38CB6 /* XCRemoteSwiftPackageReference "client-sdk-swift" */;
463449
productName = LiveKit;
464450
};
465-
B5E46D1F2D496CD200ACC2E6 /* LiveKitKrispNoiseFilter */ = {
466-
isa = XCSwiftPackageProductDependency;
467-
package = B5E46D1E2D496CD200ACC2E6 /* XCRemoteSwiftPackageReference "swift-krisp-noise-filter" */;
468-
productName = LiveKitKrispNoiseFilter;
469-
};
470451
/* End XCSwiftPackageProductDependency section */
471452
};
472453
rootObject = B5B5E3AA2D124AE00099C9BE /* Project object */;

‎VoiceAssistant.xcodeproj/project.xcworkspace/xcshareddata/swiftpm/Package.resolved

+5-14
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,13 @@
11
{
2-
"originHash" : "427c44f75b04feb1c4e9a9f495419b817da0c8b3930e9f2d6d5e7539c8d3ca40",
2+
"originHash" : "b296e14c458c92b5325292dd05b50453c8b214c0b7935936aa5b8cf3967fbb60",
33
"pins" : [
44
{
55
"identity" : "client-sdk-swift",
66
"kind" : "remoteSourceControl",
77
"location" : "https://github.com/livekit/client-sdk-swift",
88
"state" : {
9-
"revision" : "4fba03e8934ec538e226c362275dd78692991424",
10-
"version" : "2.3.1"
9+
"revision" : "4f9db591da71bfd7c556589e43963e1d777cf337",
10+
"version" : "2.5.0"
1111
}
1212
},
1313
{
@@ -37,15 +37,6 @@
3737
"version" : "1.1.4"
3838
}
3939
},
40-
{
41-
"identity" : "swift-krisp-noise-filter",
42-
"kind" : "remoteSourceControl",
43-
"location" : "https://github.com/livekit/swift-krisp-noise-filter",
44-
"state" : {
45-
"revision" : "c12a41f2afc95ee67994faf2ea7c87cde6d65cc6",
46-
"version" : "0.0.8"
47-
}
48-
},
4940
{
5041
"identity" : "swift-log",
5142
"kind" : "remoteSourceControl",
@@ -69,8 +60,8 @@
6960
"kind" : "remoteSourceControl",
7061
"location" : "https://github.com/livekit/webrtc-xcframework.git",
7162
"state" : {
72-
"revision" : "d1a5661a412668829a22e44f875eeef109526c65",
73-
"version" : "125.6422.24"
63+
"revision" : "70bbd9b51f2c46333a91469c8a8be8303b0797b7",
64+
"version" : "125.6422.28"
7465
}
7566
}
7667
],

‎VoiceAssistant.xcodeproj/xcshareddata/xcschemes/VoiceAssistant.xcscheme

+78
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<Scheme
3+
LastUpgradeVersion = "1620"
4+
version = "1.7">
5+
<BuildAction
6+
parallelizeBuildables = "YES"
7+
buildImplicitDependencies = "YES"
8+
buildArchitectures = "Automatic">
9+
<BuildActionEntries>
10+
<BuildActionEntry
11+
buildForTesting = "YES"
12+
buildForRunning = "YES"
13+
buildForProfiling = "YES"
14+
buildForArchiving = "YES"
15+
buildForAnalyzing = "YES">
16+
<BuildableReference
17+
BuildableIdentifier = "primary"
18+
BlueprintIdentifier = "B5B5E3B12D124AE00099C9BE"
19+
BuildableName = "VoiceAssistant.app"
20+
BlueprintName = "VoiceAssistant"
21+
ReferencedContainer = "container:VoiceAssistant.xcodeproj">
22+
</BuildableReference>
23+
</BuildActionEntry>
24+
</BuildActionEntries>
25+
</BuildAction>
26+
<TestAction
27+
buildConfiguration = "Debug"
28+
selectedDebuggerIdentifier = "Xcode.DebuggerFoundation.Debugger.LLDB"
29+
selectedLauncherIdentifier = "Xcode.DebuggerFoundation.Launcher.LLDB"
30+
shouldUseLaunchSchemeArgsEnv = "YES"
31+
shouldAutocreateTestPlan = "YES">
32+
</TestAction>
33+
<LaunchAction
34+
buildConfiguration = "Debug"
35+
selectedDebuggerIdentifier = "Xcode.DebuggerFoundation.Debugger.LLDB"
36+
selectedLauncherIdentifier = "Xcode.DebuggerFoundation.Launcher.LLDB"
37+
launchStyle = "0"
38+
useCustomWorkingDirectory = "NO"
39+
ignoresPersistentStateOnLaunch = "NO"
40+
debugDocumentVersioning = "YES"
41+
debugServiceExtension = "internal"
42+
allowLocationSimulation = "YES">
43+
<BuildableProductRunnable
44+
runnableDebuggingMode = "0">
45+
<BuildableReference
46+
BuildableIdentifier = "primary"
47+
BlueprintIdentifier = "B5B5E3B12D124AE00099C9BE"
48+
BuildableName = "VoiceAssistant.app"
49+
BlueprintName = "VoiceAssistant"
50+
ReferencedContainer = "container:VoiceAssistant.xcodeproj">
51+
</BuildableReference>
52+
</BuildableProductRunnable>
53+
</LaunchAction>
54+
<ProfileAction
55+
buildConfiguration = "Release"
56+
shouldUseLaunchSchemeArgsEnv = "YES"
57+
savedToolIdentifier = ""
58+
useCustomWorkingDirectory = "NO"
59+
debugDocumentVersioning = "YES">
60+
<BuildableProductRunnable
61+
runnableDebuggingMode = "0">
62+
<BuildableReference
63+
BuildableIdentifier = "primary"
64+
BlueprintIdentifier = "B5B5E3B12D124AE00099C9BE"
65+
BuildableName = "VoiceAssistant.app"
66+
BlueprintName = "VoiceAssistant"
67+
ReferencedContainer = "container:VoiceAssistant.xcodeproj">
68+
</BuildableReference>
69+
</BuildableProductRunnable>
70+
</ProfileAction>
71+
<AnalyzeAction
72+
buildConfiguration = "Debug">
73+
</AnalyzeAction>
74+
<ArchiveAction
75+
buildConfiguration = "Release"
76+
revealArchiveInOrganizer = "YES">
77+
</ArchiveAction>
78+
</Scheme>

‎VoiceAssistant/Agent/AgentView.swift

+32
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
import LiveKit
2+
import LiveKitComponents
3+
import SwiftUI
4+
5+
struct AgentView: View {
6+
@EnvironmentObject private var room: Room
7+
@EnvironmentObject private var participant: Participant
8+
9+
private var worker: RemoteParticipant? {
10+
room.remoteParticipants.values.first { $0.kind == .agent && $0.attributes["lk.publish_on_behalf"] == participant.identity?.stringValue }
11+
}
12+
13+
private var cameraTrack: VideoTrack? {
14+
return participant.firstCameraVideoTrack ?? worker?.firstCameraVideoTrack
15+
}
16+
17+
private var micTrack: AudioTrack? {
18+
return participant.firstAudioTrack ?? worker?.firstAudioTrack
19+
}
20+
21+
var body: some View {
22+
ZStack {
23+
if let cameraTrack, !cameraTrack.isMuted {
24+
SwiftUIVideoView(cameraTrack)
25+
.clipShape(RoundedRectangle(cornerRadius: 8))
26+
} else {
27+
AgentBarAudioVisualizer(audioTrack: micTrack, agentState: participant.agentState, barColor: .primary, barCount: 5)
28+
}
29+
}
30+
.id("\(participant.identity?.stringValue ?? "none")-\(cameraTrack?.sid?.stringValue ?? "none")-\(micTrack?.sid?.stringValue ?? "none")")
31+
}
32+
}

‎VoiceAssistant/Chat/View/ChatView.swift

+7
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,13 @@ struct ChatView: View {
2727
}
2828
.listStyle(.plain)
2929
.scrollIndicators(.hidden)
30+
.mask(
31+
LinearGradient(
32+
gradient: Gradient(colors: [.clear, .black, .black]),
33+
startPoint: .top,
34+
endPoint: .init(x: 0.5, y: 0.2)
35+
)
36+
)
3037
}
3138
.animation(.default, value: viewModel.messages)
3239
.alert("Error while connecting to Chat", isPresented: .constant(viewModel.error != nil)) {

‎VoiceAssistant/Connect/StatusView.swift

-17
This file was deleted.

0 commit comments

Comments
 (0)