AAC audio mixing?

[arpit-softcircuits]

I need to mix AAC audio packets coming from 3 sources. So, there are 2 ways:

1. First is, to decode each aac packet from the 3 different sources separately, and then mix these and play on I2S speaker.
2. Or maybe we can mix 3 AAC packets , and decode the mixed output and play it. I am not sure whether it is possible to mix AAC packets or not.

So, my question is whether it is possible to go with the 2nd approach , because mixing aac packets will reduce the computational load as I will be decoding only once instead of 3 times?

[pschatzmann]

I don't think that 2 is an option and I have also my doubts if the processing power for option 1 is feasible on an ESP32.
The good news is that if it is not working out, you can just try and replace the Codec with some other alternatives.

If it is for music I suggest to look at

• sbc
• lc3
• opus

If it is for speech:

• gsm
• G.722
• codec2

In any case keep us updated about your findings...

[arpit-softcircuits]

Thanks @pschatzmann . I tried method 1 and as soon as I included the wifi code, I started getting error due to low memory. And I think, this method will also not be possible with ESP32 as you said.
So, since I am working with speech data, I am exploring GSM, G.722 and codec2. But when I see their directory & libraries, I am at a loss at how to incorporate it with Arduino. I couldn't find any sample code or arduino sketches. Could you please guide me on where should I start in order to understand how to include these libraries with arduino?

[pschatzmann]

Just read https://github.com/pschatzmann/arduino-audio-tools/wiki/Encoding-and-Decoding-of-Audio. In the list you find the class names to use. Just replace the class name that you used (AACDecoderHelix) in your current sketch with the new one: e.g. SBCDecoder to decode and SBCEncoder to encode.

The audio codecs usually expect a sample rate of 8000 with 1 channel.

Examples which test the codecs can be found int https://github.com/pschatzmann/arduino-audio-tools/tree/main/examples/tests

[pschatzmann]

the low audio output issue I mentioned in GSM incoding/decodig. I fixed that by removing /8 in file CodecGSM.h line number 196.

Hmm, I don't think this is a good solution: I extended the logic a bit so that you can pass a parameter in the constructor which deactivates the scaling - but does clipping instead..

[arpit-softcircuits]

hi @pschatzmann
I have to send GSM encoded packet to UDP. I am not able to understand how to read encoded GSM packet(33bytes). in your example you use EncodedAudioStream object.

[pschatzmann]

I am afraid your question does not make any sense to me and you don't need to read anything if you want to send.
In the EncodedAudioStream which does then encoding you just give a UDPStream object as parameter

[arpit-softcircuits]

hi @pschatzmann
actually earlier I was using *udp.broadcastTo((uint8_t )my_audio_buffer,num_result_bytes,1234); this function to send UDP broad cast on port 1234. so now i am confused how to use this?

[pschatzmann]

UDPStream udp
DecodedAudioStream(&udp, new GSMEncoder());
...
udp.begin(broadcast_ip, port);

https://pschatzmann.github.io/arduino-audio-tools/html/classaudio__tools_1_1_u_d_p_stream.html

[pschatzmann]

I just realized that I currently only support mixing on the input side. You however need to do it on the output side.

I just committed the OutputMixer class in AudioOutput.h. I haven't tested it however, so I suggest that you look at the implementation.
You would create an instance for the OutputMixer in your sketch e.g. with
OutputMixer<int_16_t> mixer(i2s, 3);
and in the 3 EncodedStreams you specify that the output goes to the mixer.

[taggie]

Hi! I was also looking to mix output channels onto I2S and tried your implementation, so far it seems i'm doing something wrong.

I'm attempting to use the code below.

When pointing the urlDecoder directly towards the I2S outputstream instead of the mixer, I get perfect sound, and Serial output information like:
16:45:33.390 -> [I] AudioCopy.h : 121 - StreamCopy::copy 1024 -> 1024 -> 1024 bytes - in 1 hops

However when I pont towards the mixer, I get no sound, and information like:

16:48:03.337 -> [I] AudioCopy.h : 121 - StreamCopy::copy 1024 -> 1024 -> 1024 bytes - in 1 hops
16:48:03.337 -> [I] AudioOutput.h : 442 - write 0: 4608
16:48:03.337 -> [I] AudioOutput.h : 442 - write 0: 4608
16:48:03.372 -> [I] AudioOutput.h : 442 - write 0: 4608

It seems the stream arrives okay, and is also handled by the Mixer's write function, but I think i'm not completing the journey towards the i2s stream correctly?

Your help would be very much appreciated! Also, thanks for the awesome library!

#include "AudioTools.h"
#include "AudioLibs/AudioKit.h"
#include "AudioCodecs/CodecMP3Helix.h"

#define _SSID       "xxxxxxx"
#define _PASS       "xxxxxxx"
#define _MP3_LINK   "http://www.linktosomfile.mp3"

#include <WiFi.h>
#include <WiFiClient.h>
WiFiClient streamClient;

AudioKitStream i2s; // final output of decoded stream
OutputMixer<int16_t> mixer(i2s, 1);

URLStream url(streamClient);  // or replace with ICYStream to get metadata
EncodedAudioStream urlDecoder(&mixer, new MP3DecoderHelix()); // Decoding stream
StreamCopy urlToDecoderCopier(urlDecoder, url); // copy url to decoder

void setup(void) {
  Serial.begin(115200);
  WiFi.begin(_SSID, _PASS);
  while (WiFi.status() != WL_CONNECTED) { Serial.print("."); delay(50); }
  
  AudioLogger::instance().begin(Serial, AudioLogger::Info);

  auto config = i2s.defaultConfig(TX_MODE);

  i2s.begin(config);
  i2s.setVolume(0.8);

  mixer.begin();
  
  urlDecoder.begin();

  url.begin(_MP3_LINK, "audio/mp3");

  Serial.println("setup complete");
}


void loop() 
{
  urlToDecoderCopier.copy();
}

[pschatzmann]

I committed a correction which should make your sketch work. However you will need to explicitly define the buffer size by providing the size in the begin. It seems that mp3 is submitting data chungs of 4608 bytes. So this is the min size

mixer.begin(4608);

[taggie]

Hi, thanks for the update!

The example now works! However when I set the number of streams to 2; the output becomes jittery, regardless of whether I'm actually adding a second stream to it. Is there a logical explanation for that?

OutputMixer<int16_t> mixer(i2s, 2);

I can imagine it's the limited resources of the ESP32, although it shouldn't actually have to do anything yet.

I have also tried increasing the buffer size, but that didn't have any effect.

If you have any ideas I'd love to hear it, and otherwise we'll just stick with single channel output.

Thanks!

[pschatzmann]

I think if you want to do mixing you can't use any resource intensitive encoded format. The best is to use PWM format or WAV which is basically PWM with a header.
If you have any impact on the files you can also consider to decrease the sampling rate and/or channels.

[taggie]

I have made a setup with a WAV file, and the result is similar.

If I set NUM_CHAN to 1, the audio plays (although a bit slower than its supposed to, I guess this is due to the larger file that has to be streamed).
More importantly, if i set NUM_CHAN to 2, the output jitters heavily. Also after a few seconds, I get this from the outputmixer class.

[W] AudioOutput.h : 451 - Available Buffer too small 373: requested: 512 -> increase the buffer size

I call mixer.begin with 16384 bytes, that should do right?

The complete code is below.

I also tried with adding an actual second wav stream from the SD card to the mixer (not just increasing the number in the constructor), and in that case the audio is also heavily distorted.

Do you maybe have a working example of the output mixer that I could start from?
I may also just be stretching whats possible, so maybe just use two boards and an external mixer.

Thanks!

#include "AudioTools.h"
#include "AudioLibs/AudioKit.h"

#define _SSID       "xxxxxxx"
#define _PASS       "xxxxxxx"
#define _WAV_LINK   "http://www.linktosomfile.wav"

#define NUM_CHAN 1

#include <WiFi.h>
#include <WiFiClient.h>
WiFiClient streamClient;

AudioKitStream i2s; // final output of decoded stream
OutputMixer<int16_t> mixer(i2s, NUM_CHAN);

URLStream url(streamClient);  // or replace with ICYStream to get metadata
EncodedAudioStream urlDecoder(&mixer, new WAVDecoder()); // Decoding stream
StreamCopy urlToDecoderCopier(urlDecoder, url); // copy url to decoder

void setup(void) {
  Serial.begin(115200);
  WiFi.begin(_SSID, _PASS);
  while (WiFi.status() != WL_CONNECTED) { Serial.print("."); delay(50); }
  
  AudioLogger::instance().begin(Serial, AudioLogger::Warning);

  auto config = i2s.defaultConfig(TX_MODE);

  i2s.begin(config);
  i2s.setVolume(0.8);

  mixer.begin(16384);
  
  urlDecoder.begin();

  url.begin(_WAV_LINK, "audio/wav");

  Serial.println("setup complete");
}


void loop() 
{
  urlToDecoderCopier.copy();
}

[pschatzmann]

Examples can be found in the documentation: https://github.com/pschatzmann/arduino-audio-tools/wiki/Splitting-and-Merging-Audio
Please not that the input mixer is much more efficient the mixing on the output side...

[taggie]

Working from the example and replacing 1 sine with a WAV file from SD, I am able to get a result. Two wav sources is too much.

How would audio file mixing on the input side look? I'm not sure how to use the InputMixer.add() functionality with a wav file.

Thanks!

[pschatzmann]

You could try to directly use File objects as input. In the add method you would pass the files.
If use use WAV files you will need to read (to ignore) the first 44 header bytes after calling open.