[BUG] [CRITICAL] Audio format mismatch leads to distorted sound playback #106

Activity

1. christallire

   added

   bugSomething isn't workingSomething isn't working

   on Mar 9, 2026
2. LSXPrime commented on Mar 19, 2026

   

   LSXPrime

   on Mar 19, 2026

   Owner

   More actions

   Hmm, interesting. I'm trying to reproduce this, but I'm getting different results on my end. For me, the audio seems to be playing back at the correct speed regardless of the sample rate mismatch; there's just a quality difference, which is what I'd expect from resampling.

   Here's what I tested:

   I took a standard 48kHz MP3 file that runs for 02:17 on my 48kHz speekers. It plays back correctly at 1x speed.

   Then, I converted it down to a 22.05kHz mono file using this exact ffmpeg command:

   ffmpeg -i source.mp3 -ac 1 -ar 22050 -c:a libmp3lame -q:a 9 out.mp3

   When I play this new out.mp3 file using the same code from your MRE, it also plays for the full 02:17 at 1x speed. There's no speed or pitch distortion—it just sounds like a lower-quality version of the original.

   Can you provide sample files?
3. christallire commented on Mar 19, 2026

   

   christallire

   on Mar 19, 2026

   Author

   More actions

   Oh, Hi, never thought I'd get answer so soon.
   I've wrote a test program

   using SoundFlow.Backends.MiniAudio;
   using SoundFlow.Components;
   using SoundFlow.Providers;
   using SoundFlow.Structs;
   
   // ── Initialize SoundFlow audio engine ───────────────────────────
   using var engine = new MiniAudioEngine();
   engine.UpdateAudioDevicesInfo();
   
   if (engine.PlaybackDevices.Length == 0)
   {
       Console.Error.WriteLine("No playback devices found.");
       return 1;
   }
   
   var devices = engine.PlaybackDevices;
   var defaultDevice = Array.Find(devices, d => d.IsDefault);
   if (defaultDevice.Name is null)
       defaultDevice = devices[0];
   
   Console.WriteLine($"Using device: {defaultDevice.Name}");
   
   using var playbackDevice1 = engine.InitializePlaybackDevice(defaultDevice, AudioFormat.StudioHq);
   using var playbackDevice2 = engine.InitializePlaybackDevice(defaultDevice, AudioFormat.DvdHq);
   playbackDevice1.Start();
   playbackDevice2.Start();
   
   // ── Create sound players ────────────────────────────────────────
   var highProvider = new StreamDataProvider(engine, File.OpenRead("test.wav"));
   var lowProvider = new StreamDataProvider(engine, File.OpenRead("test.wav"));
   
   var highPlayer = new SoundPlayer(engine, playbackDevice1.Format, highProvider);
   var lowPlayer = new SoundPlayer(engine, playbackDevice2.Format, lowProvider);
   
   playbackDevice1.MasterMixer.AddComponent(highPlayer);
   playbackDevice2.MasterMixer.AddComponent(lowPlayer);
   
   // ── Alternating playback ────────────────────────────────────────
   const int delayMs = 500; // simple pause between clicks
   
   Console.WriteLine();
   
   var beat = 0;
   var cts = new CancellationTokenSource();
   Console.CancelKeyPress += (_, e) =>
   {
       e.Cancel = true;
       cts.Cancel();
   };
   
   try
   {
       while (!cts.Token.IsCancellationRequested)
       {
           var isHigh = beat % 2 == 0;
           var player = isHigh ? highPlayer : lowPlayer;
           var label = isHigh ? "HIGH" : "LOW ";
   
           player.Seek(0);
           player.Play();
   
           Console.WriteLine($"  Beat {beat + 1:D4}  [{label}]");
           beat++;
   
           await Task.Delay(delayMs, cts.Token);
       }
   }
   catch (OperationCanceledException)
   {
       // Expected on Ctrl+C
   }
   
   Console.WriteLine();
   Console.WriteLine($"Stopped after {beat} beats.");
   return 0;

   test.wav

   2026-03-19.23-49-56.mp4

   You can clearly hear sounds different, but please note that both sounds different with original wav.
4. LSXPrime commented on Mar 20, 2026

   

   LSXPrime

   on Mar 20, 2026

   Owner

   More actions

   Oh, Hi, never thought I'd get answer so soon.

   After three months in the desert boot‑camp, I finally got back home and my phone was back in my hands. I turned it on, saw the issue notification, and felt happy that the library was still in use. I immediately opened my computer, started testing and taking notes - before even taking off my military clothes, haha.

   I've wrote a test program

   I ran the program on the bundled file and, honestly, I wouldn’t call it “distorted” – that word has been a pain point in earlier library releases. It’s more accurate to say the audio is “sped up.” The pitch shift comes from the fact that you initialized the engine with the default 48 kHz and 96 kHz presets, while the test file is 44.1 kHz.

   Luckily, the Audio Format API can auto‑detect the stream’s properties with AudioFormat.GetFormatFromStream(Stream stream).

   SoundPlayer's format has no effect on this. and API regarding the format is very confusing by itself.

   The AudioFormat API was added to centralize all the core audio metadata (format, sample rate, channels, etc.) that the underlying miniaudio backend and the SoundFlow editing/composition modules need, Before it existed, users still had to specify the initializing sample rate because the miniaudio backend required it.

   I get "normal" sound play whatever format I choose.

   The sample rate is what actually drives the playback speed. If you want to play a file at a different rate than its source, you’ll need to do time‑stretching to keep the sound natural. SoundFlow can do that, but it isn’t forced on you by default because time‑stretching can bite into performance. Most users just crank up a mismatched sample rate to speed things up, so the library leaves it up to you.

   ---

   Just a quick tip you might find handy, you can peek at what formats your device actually supports with the DeviceInfo.SupportedDataFormats API. For example:

   var supported = playbackDevice1.Info?.SupportedDataFormats;

   supported will be an array of NativeDataFormat objects, each telling you the sample format, channel count, and sample rate that the device can handle.
5. christallire commented on Mar 20, 2026

   

   christallire

   on Mar 20, 2026

   Last edited by christallire

   Author

   More actions

   Wow, congratulations on finishing your 3-month boot camp! :) I've been there, and I know how tough it can be. I hope it wasn't too exhausting for you. Thank you for your dedication to this great library.

   I understand your point about the audio being "sped up" rather than "distorted," and I agree with you 100%, However, I'd like to kindly discuss the API design and the expected behavior of a modern audio engine regarding this.

   1. Resampling vs. Time-Stretching
   You mentioned that keeping the sound natural requires "time-stretching" which impacts performance. I think there might be a slight confusion between time-stretching (changing speed without changing pitch, which is expensive) and resampling (converting a 44.1kHz stream to a 48kHz endpoint, which is standard and relatively cheap).
   The underlying miniaudio backend actually has excellent built-in resamplers. If a user plays a 44.1kHz test.wav on a 48kHz PlaybackDevice, the MasterMixer (or the player itself) should ideally auto-resample the stream to 48kHz so it plays at the normal 1x speed and original pitch.

   2. The Principle of Least Astonishment
   In almost all modern audio frameworks (FMOD, Wwise, Unity Audio, NAudio), if I load a 44.1kHz file and play it on my Windows default 48kHz device, it simply plays normally. Forcing the user to manually match the device's format to the file's format just to get normal playback is highly unusual. and very unintuitive. (at least to someone like me)
   What if I have a game or an app that needs to play a 44.1kHz background music track and a 48kHz sound effect simultaneously? I shouldn't have to create two separate PlaybackDevice instances for this. The mixer should handle differing source formats gracefully.

   3. Pitch/Speed Control Should Be Explicit

   Most users just crank up a mismatched sample rate to speed things up, so the library leaves it up to you.

   Relying on sample rate mismatches to change playback speed feels like a workaround. If users want to change the speed or pitch intentionally, wouldn't it be much better API design to expose a .Pitch or .Speed property on the SoundPlayer class instead?

   4. Handling Hardware Constraints (Low Sample Rate Devices)
   There is also a critical edge case regarding hardware limitations. What if the user's physical playback device only supports low sample rates?
   For example, a Bluetooth headset operating in Hands-Free Profile (HFP) might be locked to 8kHz or 16kHz. If my application needs to play a standard 48kHz or 44.1kHz audio file, I cannot "initialize the playback device to match the file's format" because the hardware/OS will simply reject the initialization.

   Without automatic resampling, playing that 48kHz file on a 16kHz device means it will play 3 times slower with a drastically lowered pitch. In this scenario, auto-downsampling the 48kHz stream to the device's native 16kHz format on the fly is the only way to guarantee the audio plays at the correct 1x speed. The user shouldn't have to manually pre-convert all their audio assets just to support different hardware endpoints.

   Suggestion:
   Could we consider enabling automatic resampling by default when SoundPlayer's stream format differs from the PlaybackDevice's format? This would make the API much more robust and intuitive for new users.

   Thank you again for your hard work on this library. I'd love to hear your thoughts on this!
6. LSXPrime commented on Mar 21, 2026

   

   LSXPrime

   on Mar 21, 2026

   Owner

   More actions

   I appreciate your analysis, but I think there's a bit of a mix-up. When I mention

   The pitch shift comes from the fact that you initialized the engine with the default 48 kHz and 96 kHz presets, while the test file is 44.1 kHz.

   I wasn't referring to the playback device, but rather the engine itself. The engine forces miniaudio's automatic resamplers to convert your 44.1kHz audio to either 48kHz or 96kHz, which is what's causing the pitch shifting.

   In the native API, specifically in Native\miniaudio-backend\library.c, there's a method called sf_allocate_device_config. This method is used to initialize the device and requires setting a target sample rate for the resamplers to work with, whether they need to upsample or downsample mismatched files.

   What if I have a game or an app that needs to play a 44.1kHz background music track and a 48kHz sound effect simultaneously? I shouldn't have to create two separate PlaybackDevice instances for this. The mixer should handle differing source formats gracefully.

   The mixer is still specific to each device, and each device should have a target sample rate. This is enforced by miniaudio's ma_device_config_init API.

   But wait, let me review the miniaudio documentation again:

       ma_device_config config = ma_device_config_init(ma_device_type_playback);
       config.playback.format   = ma_format_f32;   // Set to ma_format_unknown to use the device's native format.
       config.playback.channels = 2;               // Set to 0 to use the device's native channel count.
       config.sampleRate        = 48000;           // Set to 0 to use the device's native sample rate.

   I think I overlooked the trick of using the device's native rates. However, it doesn't really impact us because we have playbackDevice1.Info?.SupportedDataFormats; anyway. Even with this, both approaches still require specifying the target sample rate during initialization as it's going use the device default anyway for resamplers to work.

   Relying on sample rate mismatches to change playback speed feels like a workaround. If users want to change the speed or pitch intentionally, wouldn't it be much better API design to expose a .Pitch or .Speed property on the SoundPlayer class instead?

   To clarify, I was using that as an example. We already have a PlaybackSpeed property in SoundPlayerBase, which is inherited by both SoundPlayer and SurroundPlayer. This allows users to control the playback speed without affecting the pitch, thanks to the WSOLA algorithm working under the hood.

   There is also a critical edge case regarding hardware limitations. What if the user's physical playback device only supports low sample rates?

   Actually, miniaudio's built-in resamplers handle this directly. The pitch shifting occurs as expected because there's no time-stretching applied to preserve the pitch.

   ---

   Just trashed another 10 lines of garbage that veered off-topic. These past three months without a computer, phone, or any electrical devices, and no music, totally wiped my memory of what the library could do. Forget all that nonsense.

   The key is to initialize the decoder using the public StreamDataProvider(AudioEngine engine, AudioFormat format, Stream stream) provider, but don't stop at just the engine and sound player. You've got to include the target format too, like 16kHz or 96kHz.

   What if I have a game or an app that needs to play a 44.1kHz background music track and a 48kHz sound effect simultaneously? I shouldn't have to create two separate PlaybackDevice instances for this. The mixer should handle differing source formats gracefully.

   The mixer is still per device and device should have a target sample rate (this API is forced by miniaudio ma_device_config_init)

   To ditch this miniaudio limitation and let the provider and player use a different sample rate than the device (like 96kHz for provider/player and 16kHz for the device), while keeping the playback speed natural at 1X, I'd need to make SoundPlayerBase rate-aware. That means beefing up the SoundFlow built-in resample logic and relying on the time-stretcher to keep the pitch in check.

   But here's the catch, these changes will take some time, and I only have 2 days off before heading back to the desert for a 2-3 week deployment. After that, I should be able to take a monthly break to return home. I can't promise to tackle this limitation in just 2 days, my apologizes for that. For now, you'll still need to initialize the device, provider, and player with the same sample rate and format. The resamplers will handle the upsampling/downsampling to maintain that natural 1X speed and pitch.

   Actually, miniaudio's built-in resamplers handle this directly. The pitch shifting occurs as expected because there's no time-stretching applied to preserve the pitch.

   Actually, ignore that. If the user initializes the device, provider (decoder), and player with the same sample rate (like 16kHz), the resamplers will handle everything without any pitch shifting. No need to worry about it.
7. christallire commented on Mar 21, 2026

   

   christallire

   on Mar 21, 2026

   Author

   More actions

   Thank you for breaking it down for me. I completely understand the time crunch, and you definitely shouldn't spend your last 2 days at home rewriting core logic.

   Don't sweat the feature request right now. Enjoy your short time off, stay safe out there in the desert on your next deployment, and we can always revisit the auto-resampling idea whenever you have some proper downtime in the future.

   Thanks again for all your hard work on this!
8. simonhaines commented on Apr 7, 2026

   

   simonhaines

   on Apr 7, 2026

   Last edited by simonhaines

   More actions

   Let me add to @christallire and thank you for this excellent library. It's really first rate. Please stay safe during your military service.

   I think this bug is about playing audio files with differing formats through a playback device that has possibly a third format?

   If so, the Miniaudio codec factory already handles this, but you need to set it up in a different way (providing a target format hint). First you'll need two audio files with different formats. I found it difficult to hear the difference with the test.wav originally attached, so I created a sine wave that I found easier to test:
   ffmpeg -f lavfi -i "sine=frequency=1000:duration=0.25" sine.wav

   This created a WAV file with a format of S24, 1 channel and a sample rate of 44.1KHz. I then converted this to F32 format, with 2 channels and a sample rate of 48KHz:
   ffmpeg -i sine.wav -c:a pcm_f32le -ar 48000 -ac 2 sine2.wav

   I then used this modified code to create an output device with a third sample rate (96KHz StudioHq) and added both files.

   using SoundFlow.Backends.MiniAudio;
   using SoundFlow.Components;
   using SoundFlow.Providers;
   using SoundFlow.Structs;
   
   // ── Initialize SoundFlow audio engine ───────────────────────────
   using var engine = new MiniAudioEngine();
   engine.UpdateAudioDevicesInfo();
   
   if (engine.PlaybackDevices.Length == 0)
   {
       Console.Error.WriteLine("No playback devices found.");
       return 1;
   }
   
   var devices = engine.PlaybackDevices;
   var defaultDevice = Array.Find(devices, d => d.IsDefault);
   if (defaultDevice.Name is null)
       defaultDevice = devices[0];
   
   Console.WriteLine($"Using device: {defaultDevice.Name}");
   using var playbackDevice = engine.InitializePlaybackDevice(defaultDevice, AudioFormat.StudioHq);
   playbackDevice.Start();
   
   // ── Create sound players ────────────────────────────────────────
   await using var testStream = File.OpenRead("sine.wav");
   await using var test2Stream = File.OpenRead("sine2.wav");
   using var highProvider = new StreamDataProvider(engine, playbackDevice.Format, testStream);
   using var lowProvider = new StreamDataProvider(engine, playbackDevice.Format, test2Stream);
   
   var highPlayer = new SoundPlayer(engine, playbackDevice.Format, highProvider);
   var lowPlayer = new SoundPlayer(engine, playbackDevice.Format, lowProvider);
   
   playbackDevice.MasterMixer.AddComponent(highPlayer);
   playbackDevice.MasterMixer.AddComponent(lowPlayer);
   
   // ── Alternating playback ────────────────────────────────────────
   const int delayMs = 500; // simple pause between clicks
   
   Console.WriteLine();
   
   var beat = 0;
   var cts = new CancellationTokenSource();
   Console.CancelKeyPress += (_, e) =>
   {
       e.Cancel = true;
       cts.Cancel();
   };
   
   try
   {
       while (!cts.Token.IsCancellationRequested)
       {
           var isHigh = beat % 2 == 0;
           var player = isHigh ? highPlayer : lowPlayer;
           var label = isHigh ? "HIGH" : "LOW ";
   
           player.Seek(0);
           player.Play();
   
           Console.WriteLine($"  Beat {beat + 1:D4}  [{label}]");
           beat++;
   
           await Task.Delay(delayMs, cts.Token);
       }
   }
   catch (OperationCanceledException)
   {
       // Expected on Ctrl+C
   }
   
   Console.WriteLine();
   Console.WriteLine($"Stopped after {beat} beats.");
   return 0;

   On my linux-x64 system, this works as expected.

   ---

   Note though that there is an issue with the FFMpeg resampler. This can be heard by importing SoundFlow.Codecs.FFMpeg and adding the statement engine.RegisterCodecFactory(new FFmpegCodecFactory()); immediately after the engine is constructed in the code above. A fix for this issue is evolving on this branch.