Mixing A2DP with SPIFF sample, need help with buffering

[Nawor3565]

Hello! I'm trying to write a script that will take audio from A2DP and overlay a drum sample on top of it, then output to I2S. I'm currently storing the sample in SPIFFS but I don't think that matters. Right now, I have two BufferedStreams, one for the drum audio and one for the A2DP audio, both going to an OutputMixer like in this example. However, I'm getting stuttering on the output, like what happens when the mixer isn't getting two synchronized audio streams (writing the A2DP data to both buffers plays fine). I'm not sure how to appropriately buffer and send the decoded sample data to the mixer (I tried putting player.copy() in both the A2DP callback and the main loop), or whether an EncodedAudioStream or AudioPlayer instance would make more sense for this.

Side note: I haven't actually set up the SPIFFS samples yet, so I called player.begin() as inactive from the start. My understanding is that when the player is inactive it will just write all zeros as if the audio was silent, but please tell me if I'm missing something here.

Thank you!!

#include <BluetoothA2DPSink.h>
#include <AudioTools.h>
#include <AudioTools/AudioCodecs/CodecWAV.h>
#include <AudioTools/Disk/AudioSourceSPIFFS.h>

// — Config —
constexpr uint32_t buffer_size = 5000; // stupidly high value for testing
constexpr char *startFilePath="/";
constexpr char* ext="wav";

BluetoothA2DPSink a2dp_sink;
I2SStream i2s;
OutputMixer<int16_t> mixer(i2s, 2);
BufferedStream a2dp_buffer(buffer_size, mixer);  
BufferedStream drum_buffer(buffer_size, mixer);  
AudioSourceSPIFFS source(startFilePath, ext);
WAVDecoder wavDecoder;
//EncodedAudioStream streamPlayer(source, &wavDecoder);
AudioPlayer player(source, drum_buffer, wavDecoder);


AudioInfo fmt{44100, 2, 16};


// A2DP callback
void onA2dpData(const uint8_t* data, uint32_t len) {
  a2dp_buffer.write(data, len);

  //player.copy(buffer_size);
}


void setup() {
  Serial.begin(115200);
  AudioToolsLogger.begin(Serial, AudioToolsLogLevel::Info);

  // A2DP sink config
  a2dp_sink.set_stream_reader(onA2dpData, false);
  a2dp_sink.set_auto_reconnect(true);
  a2dp_sink.start("a2dp-i2s");

  auto cfg = i2s.defaultConfig(TX_MODE);
  cfg.sample_rate = 44100;
  cfg.bits_per_sample = fmt.bits_per_sample;
  cfg.channels = fmt.channels;
  cfg.pin_bck = 18;
  cfg.pin_ws = 22;
  cfg.pin_data = 19;
  i2s.begin(cfg);

  mixer.begin(buffer_size);
  player.begin(0, 0);
}

void loop() {
 player.copy(buffer_size); 
}

[pschatzmann]

I have improved the documentation of the OutputMixer. Please try to understand the API

Your sketch has many isssues:

1. The AudioPlayer only provides silence, if you switch this functionality on (with setSilenceOnInactive (bool active)): otherwise there is no output.
2. Your mixer uses AutoIndex mode, which means with each write it writes to the next mixing channel. When you use a BufferedStream as output, you end up splitting the content to the different output channels! So you must manage the index yourself by calling setAutoIndex (false) and setIndex() before each output to write(uint8_t, size_t).

However, it would be better to use the direct write API by indicating the mixing channel, when possible.

void onA2dpData(const uint8_t* data, uint32_t len) {
  // write to channel 0
  mixer.write(0, data, len);

  // write to channel 1
  mixer.setIndex(1);
  player.copy(mixer.available(0));

  // output mixing reslt
  mixer.flushMixer();
}

void setup() {
  ...
  // deactivate automatic index management
  mixer.setAutoIndex(false);
  ...
}

Also make sure that the buffer is big enough for the A2SP writes by calling resize(size);
If you want to mimimize the buffer size, you can send the A2DP output to a BufferedStream which forwards the output to a CallbackStream, and do the mixing in the CallbackStream (as described in this discussion).

[Nawor3565]

Thank you! I had to implement the CallbackStream as I was having the same problem that came up in that discussion. It seems to work great, except that for some reason, there is stuttering right after the sample finishes playing. It mixes the sample with A2DP fine, and plays back the A2DP fine on its own, but right after the sample ends there is about a second of stuttering. It looks like the AudioPlayer should be padding with silence if you try to copy out more bytes than are left in the sample, but maybe I'm not understanding it correctly.

#include <BluetoothA2DPSink.h>
#include <AudioTools.h>
#include <AudioTools/AudioCodecs/CodecWAV.h>        // WAV decoder
//#include <AudioTools/Disk/AudioSourceSPIFFS.h>
#include <AudioTools/Disk/AudioSourceLittleFS.h>

// — Config —
constexpr char *startFilePath="/";
constexpr char* ext=".wav";

BluetoothA2DPSink a2dp_sink;
I2SStream i2s;
OutputMixer<int16_t> mixer(i2s, 2);
CallbackStream cb;
BufferedStream buffered(cb);
AudioSourceLittleFS source(startFilePath, ext);
WAVDecoder wavDecoder;
//EncodedAudioStream player(source, &wavDecoder);
AudioPlayer player(source, mixer, wavDecoder);

AudioInfo fmt{44100, 2, 16};


void onA2dpData(const uint8_t* data, uint32_t len) {
  buffered.write(data, len);
}

// with each write: mix the data with data from file (or silence)
size_t onWrite(const uint8_t *data, size_t len){
  //LOGW("=>  onWrite: %u", len);
  mixer.write(0, data, len);
  mixer.setIndex(1);
  player.copy(len);
  mixer.flushMixer();
  return len;
}


void setup() {
  Serial.begin(115200);
  //AudioToolsLogger.begin(Serial, AudioToolsLogLevel::Debug);

  // A2DP sink config
  a2dp_sink.set_stream_reader(onA2dpData, /*no built-in I2S*/ false);
  a2dp_sink.set_auto_reconnect(true);
  a2dp_sink.start("a2dp-i2s");

  auto cfg = i2s.defaultConfig(TX_MODE);
  cfg.sample_rate = 44100;
  cfg.bits_per_sample = fmt.bits_per_sample;
  cfg.channels = fmt.channels;
  cfg.pin_bck = 18;
  cfg.pin_ws = 22;
  cfg.pin_data = 19;
  i2s.begin(cfg);

  mixer.setAutoIndex(false);
  mixer.begin(2000);
  player.setSilenceOnInactive(true);
  //player.setAutoFade(false);
  player.begin(0, false);

  cb.setWriteCallback(onWrite);
}

void loop() {
 player.setIndex(0);
 player.play();
 delay(7000);
}

[pschatzmann]

I think you are messing the processing up with the delay(). But I don't quite understand what you are doing in the loop...

[Nawor3565]

The loop is just for debugging, it should just play the 0.7s-long sample once every few seconds, overlaying it on top of the continuous A2DP audio. It does this, but regardless of how long the delay is, it still stutters immediately after the sample ends. Am I missing the proper way to gracefully end the AudioPlayer? (Adding player.setAutoNext(false) also didn't make a difference).

For context, my final goal is to have external triggers like buttons that when activated, play a drum sound(s) on top of the A2DP audio. I just wanted to get the audio pipeline working before adding in GPIO stuff.

Edit: Using a button to trigger the sample playback also seems to make no difference :/

void loop() {
  bool triggerState = digitalRead(TRIGGER_PIN);
  if (lastTriggerState == HIGH && triggerState == LOW) {
      player.setIndex(0);
      player.play();
    Serial.println("Trigger: Playing sample once");
  }
  lastTriggerState = triggerState;

  delay(10);
}