The NERTC SDK supports custom audio collection and rendering. You can provide custom audio input source data to the NERTC SDK and use a custom renderer, and then the NERTC SDK encodes and pushes the data.

Generally, the app uses the default device to collect audio data, which is usually the built-in microphone of the device. However, you may need to use a custom audio source in some scenarios. For example:

• The scenario where it is required to use custom sound effects or sound beautification library, or pre-processing library.
• The scenario where it is required to use an external audio source or use an external device to collect audio data, e.g. playing audio files in audio and video call or interactive live streaming events.
• The scenario where the app is unable to obtain control permission of the audio collection device. For example, the audio collection device is occupied by other services, or the default audio collection module of the hardware device is damaged.

Based on the above scenarios, the NERTC SDK enables developers to use custom audio sources or renderer to implement the related requirements in these business scenarios. This article describes how to implement the custom audio collection function.

You can enable the custom audio collection function through the setExternalAudioSource API of the NERTC SDK. Then, you can use the pushExternalAudioFrame API to push the external audio data frame to the NERTC SDK. The audio data format can be specified by the parameter NERtcAudioExternalFrame of the function pushExternalAudioFrame.

Considerations

• The setExternalAudioSource API must be set before you join a room, and such setting is still valid after a call ends. If you need to turn off this function, you need to call the API to disable the external audio data input function before the next call.

• The data sent to the SDK through pushExternalAudioFrame must be the uncompressed raw audio data in PCM format, and other compression formats are not supported.

• In the custom audio collection and rendering scenarios, the 3A algorithms (AEC, ANS, and AGC) for audio processing are disabled and cannot be manually enabled.

• You need to manage the processing logic of audio data collection and rendering by yourself.

Custom audio capture

1. Before joiningChannel, you can call setExternalAudioSource to enable the custom audio collection function and set the external audio collection parameters.
2. After local users successfully join a room, the self-collection module will be used to collect audio data. You need to manage the logic of audio data collection and processing by yourself.
3. After the audio data is processed, you can call pushExternalAudioFrame to push the external audio data frame to the NERTC SDK.

Custom audio rendering

1. Before joining a room, you can enable and set the external audio rendering by calling setExternalAudioRender.
2. After successfully joining the room, you can call pullExternalAudioFrame to pull the audio sent by the remote end.

Note: The buffer array parameter in pullExternalAudioFrame must be created by calling ByteBuffer.allocateDirect. It is recommended that the length of the audio data to be pulled is at least 10 milliseconds.

The calculation formula is: len = sampleRate/1000 × 2 × channels × audio data duration (milliseconds)

3. You need to render and play the pulled audio data by yourself.

Sample code

//Custom audio collection
NERtcAudioExternalFrame audioFrame = new
NERtcAudioExternalFrame();
audioFrame.audioData =  sampleData;
audioFrame.numberOfChannels = 1;
audioFrame.sampleRate =  48000;
audioFrame.samplesPerChannel = 48000  * 20 / 1000;
NERtcEx.getInstance().pushExternalAudioFrame(audioFrame);

//Custom audio rendering
NERtcEx.getInstance().setExternalAudioRender(true,sampleRate,channels);
int len = sampleRate/1000 × 2 × channels × 10;
ByteBuffer byteBuffer = ByteBuffer.allocateDirect(len);
NERtcEx.getInstance().pullExternalAudioFrame(byteBuffer,len);
byteBuffer.rewind();