SDL3 Audio: pulseaudio takes twice as long to put recording data into AudioStream

[BAlzayat]

When recording audio from a device using pulseaudio, I found that it takes twice as long as the duration of the audio segment to receive the expected number of recorded bytes, all while the program's memory usage grows unreasonably large the longer it keeps recording.

My OS is Debian 12, SDL at the latest commit 1f6b5c681d788d3881a40c456bed5ce3e1a83aaa, using my phone's microphone forwarded to pulseaudio using https://github.com/gdzx/audiosource. Other recording software don't have the same issue.

The following program demonstrates this issue, it records 4 seconds of audio then writes the raw data into the file test.raw:

#include <SDL3/SDL.h>
#include <SDL3/SDL_main.h>

int main(int argc, char **argv){
	
	SDL_SetHint(SDL_HINT_AUDIO_DRIVER, "pulseaudio");
	SDL_Init(SDL_INIT_AUDIO);
	
	int recorder_count = 0;
	SDL_AudioDeviceID *recorders = SDL_GetAudioRecordingDevices(&recorder_count);
	if (!recorders){
		SDL_Log("Failed to get recorders: %s\n", SDL_GetError());
		return 1;
	}else if (recorder_count == 0){
		SDL_Log("No recorder found\n");
		return 1;
	}
	
	for (int i = 0; i < recorder_count; ++i){
		SDL_Log("Recorder %d: %s\n", recorders[i], SDL_GetAudioDeviceName(recorders[i]));
	}
	
	SDL_AudioStream *stream = SDL_OpenAudioDeviceStream(recorders[0], 0, 0, 0);
	if (!stream){
		SDL_Log("Failed to create stream %s\n", SDL_GetError());
		return 1;
	}
	
	SDL_free(recorders);
	
	SDL_AudioSpec spec = {0};	
	SDL_GetAudioStreamFormat(stream, 0, &spec);
	
	SDL_Log("Format: %s\n", SDL_GetAudioFormatName(spec.format));
	SDL_Log("Channels: %d\n", spec.channels);
	SDL_Log("Rate: %d\n", spec.freq);
	SDL_Log("Driver: %s\n", SDL_GetCurrentAudioDriver());	
	
	SDL_ResumeAudioStreamDevice(stream);
	
	const Uint32 record_duration = 4000;
	
	int total_bytes = SDL_AUDIO_BYTESIZE(spec.format) * spec.channels * spec.freq * record_duration / 1000;
	int record_bytes = 0;
	Uint32 record_start = 0;
	Uint32 record_end = 0;
	
	SDL_IOStream *file = SDL_IOFromFile("test.raw", "wb");	
	
	bool running = true;
	while (running){
		SDL_Event ev;
		while(SDL_PollEvent(&ev)){
		}
		
		static char buf[1024];
		int len = 0;
		
		do {
			len = SDL_GetAudioStreamData(stream, buf, sizeof(buf));
			
			if (len > 0){
				if (record_bytes == 0){
					record_start = SDL_GetTicks();
				}
				
				record_bytes += len;
				SDL_WriteIO(file, buf, len);
				
				if (record_bytes >= total_bytes){
					running = false;
					record_end = SDL_GetTicks();
				}
			}
		}while(len > 0);
		
		if (len == -1){
			SDL_Log("Failed to get stream data: %s\n", SDL_GetError());
			return 1;
		}
		
		SDL_Delay(1);
	}
	
	SDL_Log("Done recording in %dms\n", record_end - record_start);
	
	SDL_DestroyAudioStream(stream);
	SDL_CloseIO(file);
	
	SDL_Quit();
}

Here's the output of the program:

Recorder 18: Unix FIFO source /tmp/audiosource
Format: SDL_AUDIO_S16LE
Channels: 1
Rate: 44100
Driver: pulseaudio
Done recording in 7985ms

I found where I think the delay happens, which is in PULSEAUDIO_WaitRecordingDevice, as I observe the timing of the output of the following line:

index 69e8c1a84..d584ad827 100644
--- a/src/audio/pulseaudio/SDL_pulseaudio.c
+++ b/src/audio/pulseaudio/SDL_pulseaudio.c
@@ -502,7 +502,7 @@ static bool PULSEAUDIO_WaitRecordingDevice(SDL_AudioDevice *device)
PULSEAUDIO_pa_stream_drop(h->stream);  // drop this fragment.
} else {
	// store this fragment's data for use with RecordDevice
-                //SDL_Log("PULSEAUDIO: recorded %d new bytes", (int) nbytes);
+                SDL_Log("PULSEAUDIO: recorded %d new bytes", (int) nbytes);

I don't know the proper solution to this but I found the following change fixes the issue:

index 69e8c1a84..5f53a29ec 100644
--- a/src/audio/pulseaudio/SDL_pulseaudio.c
+++ b/src/audio/pulseaudio/SDL_pulseaudio.c
@@ -672,7 +672,7 @@ static bool PULSEAUDIO_OpenDevice(SDL_AudioDevice *device)
paspec.rate = device->spec.freq;

// Reduced prebuffering compared to the defaults.
-    paattr.fragsize = device->buffer_size;   // despite the name, this is only used for recording devices, according to PulseAudio docs!
+    paattr.fragsize = 2 * device->buffer_size;   // despite the name, this is only used for recording devices, according to PulseAudio docs!
paattr.tlength = device->buffer_size;
paattr.prebuf = -1;
paattr.maxlength = -1;

And here's the output of my test program after the change:

Recorder 18: Unix FIFO source /tmp/audiosource
Format: SDL_AUDIO_S16LE
Channels: 1
Rate: 44100
Driver: pulseaudio
Done recording in 4010ms