A simple guide of starting use EasyRTC

Working on a Loongson(www.loongson.cn) PC, goal is to make it a Meeting Terminal. Because its CPU is MIPS arch, there could be lots of unexpected problems, so the first thought hit us is WebRTC.

This post is the first step to research into this topic.

Continue reading “A simple guide of starting use EasyRTC”

Open-Sourced H.264 Removes Barriers to WebRTC

Great news from Cisco: Cisco will open open-source their H.264 codec, and make H.264 free for use in WebRTC.

Here are the details, which came from blogs of Cisco by Rowan Trollope:
———————————————
When it comes to making collaboration technology such as high-definition video open and broadly available, it’s clear that the web browser plays an important role. The question is, how do you enable real-time video natively on the Web? It’s a question that folks are anxious to have answered.

WebRTC–a set of enhancements to HTML5–will address the issue head on. But, there is an important hurdle that must first be cleared, and that’s standardizing on a common video codec for real-time communications on the web – something the Internet Engineering Task Force (IETF) will decide next week.

The industry has been divided on the choice of a common video codec for some time, namely because the industry standard–H.264–requires royalty payments to MPEG LA. Today, I am pleased to announce Cisco is making a bold move to take concerns about these payments off the table.

We plan to open-source our H.264 codec, and to provide it as a binary module that can be downloaded for free from the Internet. Cisco will not pass on our MPEG LA licensing costs for this module, and based on the current licensing environment, this will effectively make H.264 free for use in WebRTC.

I’m also pleased that Mozilla has announced it will enable Firefox to utilize this module, bringing real-time H.264 support to their browser.

“It hasn’t been easy, but Mozilla has helped to lead the industry toward interoperable video on the Web,” said Brendan Eich, Mozilla Chief Technology Officer. “Cisco’s announcement helps us support H.264 in Firefox on most operating systems, and in downstream and other open source distributions using the Cisco H.264 binary module. We are excited to work with Cisco on advancing the state of interoperable Web video.”

Why is Cisco Doing This?

Many, including Cisco, have been backing H.264, the industry standard which today powers much of the video on the Internet. We strongly believe that interoperability is an essential goal of standards activities and that usage of H.264 by WebRTC means it will be able to interconnect, without transcoding, to a large set of existing clients from a multitude of vendors.

Regarding H.264, Jan Färjh, Vice President, Head of Standardization and Industry at Ericsson, states: “Support in WebRTC for H.264 enables interoperability with most video-capable mobile devices, which is great for industry acceptance.”

Finally, if you’ve read my blog or attended our recent annual Collaboration Summit, you will have heard our mission for Cisco Collaboration: Create rich collaboration technologies that are incredibly easy to use and make that technology broadly available to everyone in the world – from the largest companies to the smallest businesses. That’s what we would like to see happen with WebRTC, powered by an industry standard that is already prevalent in the market place.

We hope and believe that this step of open-sourcing our H.264 codec will enable the industry to move forward on WebRTC and have the best of all worlds: interoperability with the large installed base of H.264 endpoints, combined with an open‐source and effectively free codebase. This action also underscores our commitment to simplicity, for the greater benefit of developers, users, and vendors alike.

I’d love to start a dialogue on this which is why I’m inviting you to attend a TweetChat I’m hosting (@rowantrollope) later today, Wednesday, October 30 at 9:30 a.m. PDT. The hashtag is #collabchat. Cisco Fellow Cullen Jennings (@cfluffy), Cisco Collaboration CTO Jonathan Rosenberg (@jdrosen2) and Snorre Kjesbu (@KjesbuSnorre), Vice-President of Cisco’s Collaboration Endpoint Technology Group, will join me in the conversation. I also welcome your comments on this blog.

Reading codes of WebRTC: try to implement a common video module wrapper from webrtc

My goal here is only to test & research, practise and get to know more about WebRTC & its codes, so here we go.

Wrappering an independent Video Capture class from WebRTC’s video capture module Continue reading “Reading codes of WebRTC: try to implement a common video module wrapper from webrtc”

LINK : fatal error LNK1104: cannot open file ‘atlthunk.lib’

I was building webrtc on my notebook which OS is Win7 when I ran into this issue. I followed my previous steps which I suceeded to download and build the complete webrtc project:

  • Install Visual Studio 2010
  • Install DirectX SDK 2010(June)
  • Install Microsoft SDK v7.1

then downolad webrtc project including the project & settings for Visual Studio, and try to build it. You can view the detail steps here: http://rg4.net/archives/736.html.

“fatal error LNK1104: cannot open file’atlthunk.lib'”

First, I tried to add this pragma to the top of my header file. But problem remains.

#pragma comment(linker,”/NODEFAULTLIB:atlthunk.lib”)

Then I digged more into it.

Found out that atlthunk.lib belong to ATL7.1 , and ATL7.1 belongs to WinDDK. So what we need to do actually is download and install WinDDK. You can download it from this url:

http://download.microsoft.com/download/4/A/2/4A25C7D5-EFBE-4182-B6A9-AE6850409A78/GRMWDK_EN_7600_1.ISO

After downloaded and installed WinDDK, you still need to add this directory to VC’s library directories manually:

C:\WinDDK\7600.16385.1\lib\ATL\amd64

Then, re-open the webrtc’s all.sln, and rebuild the solution.

So the close note will be:

If you are building webrtc on WinXP, you need to install DirectX SDK 2010(June) and Microsoft SDK v7.1. But if you are building it on Win7, you need to install one more SDK, that is WinDDK v7.1.

WebRTC audio processing continuous: AGC effect optimizing (setting parameter adjustment)

After days of working,  all audio processing functions seems are working in the KdvMediaSDK now, and the next step will be getting a proper set for the audio processing algorithms to run under certain environments & circumstance.

I will start it by beginning with AGC module too. Here are the parameters of AGC algorithm. Continue reading “WebRTC audio processing continuous: AGC effect optimizing (setting parameter adjustment)”

Reading codes of WebRTC: Deep into WebRTC Voice Engine(draft)

Introduction of this document

This module includes software based acoustic echo cancellation (AEC), automatic gain control (AGC), noise reduction, noise suppression and hardware access and control across multiple platforms.

My ultimate goal will be wrapping an independent module out of WebRTC’s Voice Engine for our product, and the first task is to get AGC implemented base on current KdvMediaSDK implementations which is not so much the same in interfaces with WebRTC).

Keywords: WebRTC, audio processing, AEC, AGC, noise reduction, noise suppression.

Overall architecture of WebRTC

The overall architecture looks something like this:

Overall architecture of WebRTC
Overall architecture of WebRTC

(Image from http://webrtc.org)

WebRTC Voice Engine – AGC control workflow

WebRTC Voice Engine - AGC control workflow
WebRTC Voice Engine – AGC control workflow

You can download my original Visio file here:

http://rg4.net/p/webrtc/webrtc.voiceengine.agc.vsd

You can modify & distribute whatever you wish, but if you made any improvement for this chart, please send me a copy by mail. That’ll benefit a lot more people. Thank you.

Target/related source codes

Major source codes:

l  audio_device_wave_win.cc: %WEBRTC%\src\modules\audio_device\main\source\win\audio_device_wave_win.cc

l  audio_device_buffer.cc:

l  audio_device_utility.cc:

l  audio_mixer_manager_win.cc:

l  voe_base_impl.cc: %WEBRTC%\src\voice_engine\main\source\voe_base_impl.cc

l  transmit_mixer.cc : %WEBRTC%\src\voice_engine\main\source\transmit_mixer.cc

l  level_indicator.cc AudioLevel src\voice_engine\main\source\level_indicator.cc

Utility source codes:

event_win_wrapper.cc

thread_win_wrapper.cc

Detail interfaces & implementations

audio_device_wave_win.cc

It responsible for:

l  Audio capture

l  Get/Set Microphone Volume (I’m not sure what this volume means, hardware volume, or a virtual volume after audio processing, only because it is Get/Set through audio_device_mixer_manager.cc)

a. Audio capture.

Step 1: run audio capture in a thread named as ThreadProcess().

[cpp]bool AudioDeviceWindowsWave::ThreadProcess()

{

while ((nRecordedBytes = RecProc(recTime)) > 0)
{

}

}[/cpp]

Step 2: detail into RecProc() function, all capture parameters & the captured buffer will be saved to a variant |AudioDeviceBuffer* _ptrAudioBuffer|

[cpp]WebRtc_Word32 AudioDeviceWindowsWave::RecProc(LONGLONG& consumedTime)
{
……
// store the recorded buffer (no action will be taken if the #recorded samples is not a full buffer)

_ptrAudioBuffer->SetRecordedBuffer(_waveHeaderIn[bufCount].lpData, nSamplesRecorded);

// Check how large the playout and recording buffers are on the sound card.
// This info is needed by the AEC.

msecOnPlaySide = GetPlayoutBufferDelay(writtenSamples, playedSamples);
msecOnRecordSide = GetRecordingBufferDelay(readSamples, recSamples);

// If we use the alternative playout delay method, skip the clock drift compensation
// since it will be an unreliable estimate and might degrade AEC performance.

WebRtc_Word32 drift = (_useHeader > 0) ? 0 : GetClockDrift(playedSamples, recSamples);
_ptrAudioBuffer->SetVQEData(msecOnPlaySide, msecOnRecordSide, drift);


if (_AGC)
{
WebRtc_UWord32 newMicLevel = _ptrAudioBuffer->NewMicLevel();
if (newMicLevel != 0)
{
// The VQE will only deliver non-zero microphone levels when a change is needed.
WEBRTC_TRACE(kTraceStream, kTraceUtility, _id,”AGC change of volume: => new=%u”, newMicLevel);

// We store this outside of the audio buffer to avoid
// having it overwritten by the getter thread.
_newMicLevel = newMicLevel;
SetEvent(_hSetCaptureVolumeEvent);
}
}

}[/cpp]

b. Get/Set Microphone Volume

There are two other threads along with the major capture thread, they are

::DoGetCaptureVolumeThread()

::DoSetCaptureVolumeThread()

These threads will be always running waiting for a signal to Get or Set capture volume.

Things I’m still trying to figure out

There are so many definitions about microphone level or relevant. What I’m not sure is which volume is what volume? Here are some volume related definitions I confused with:

1.    class VoEBaseImpl(voe_base_impl.cc)

What’s the difference between currentVoEMicLevel and currentMicLevel in the codes below?

Which can be also compare to the variants _oldVoEMicLevel and _oldMicLevel defined in voe_base_impl.cc

WebRtc_UWord32 _oldVoEMicLevel

WebRtc_UWord32 _oldMicLevel

Where will set/change these variants values?

This code locates at voe_base_impl.cc

[cpp]WebRtc_Word32 VoEBaseImpl::RecordedDataIsAvailable(

const WebRtc_UWord32 currentMicLevel,
WebRtc_UWord32& newMicLevel)
{

// Will only deal with the volume in adaptive analog mode
if (isAnalogAGC)
{
// Scale from ADM to VoE level range
if (_audioDevicePtr->MaxMicrophoneVolume(&maxVolume) == 0)
{
if (0 != maxVolume)
{
currentVoEMicLevel = (WebRtc_UWord16) ((currentMicLevel
* kMaxVolumeLevel + (int) (maxVolume / 2))
/ (maxVolume));
}
}
// We learned that on certain systems (e.g Linux) the currentVoEMicLevel
// can be greater than the maxVolumeLevel therefore
// we are going to cap the currentVoEMicLevel to the maxVolumeLevel
// if it turns out that the currentVoEMicLevel is indeed greater
// than the maxVolumeLevel
if (currentVoEMicLevel > kMaxVolumeLevel)
{
currentVoEMicLevel = kMaxVolumeLevel;
}
}
// Keep track if the MicLevel has been changed by the AGC, if not,
// use the old value AGC returns to let AGC continue its trend,
// so eventually the AGC is able to change the mic level. This handles
// issues with truncation introduced by the scaling.
if (_oldMicLevel == currentMicLevel)
{
currentVoEMicLevel = (WebRtc_UWord16) _oldVoEMicLevel;
}
// Perform channel-independent operations
// (APM, mix with file, record to file, mute, etc.)
_transmitMixerPtr->PrepareDemux(audioSamples, nSamples, nChannels,
samplesPerSec,
(WebRtc_UWord16) totalDelayMS, clockDrift,
currentVoEMicLevel);
// Copy the audio frame to each sending channel and perform
// channel-dependent operations (file mixing, mute, etc.) to prepare
// for encoding.
_transmitMixerPtr->DemuxAndMix();
// Do the encoding and packetize+transmit the RTP packet when encoding
// is done.
_transmitMixerPtr->EncodeAndSend();
// Will only deal with the volume in adaptive analog mode
if (isAnalogAGC)
{
// Scale from VoE to ADM level range
newVoEMicLevel = _transmitMixerPtr->CaptureLevel();
if (newVoEMicLevel != currentVoEMicLevel)
{
// Add (kMaxVolumeLevel/2) to round the value
newMicLevel = (WebRtc_UWord32) ((newVoEMicLevel * maxVolume
+ (int) (kMaxVolumeLevel / 2)) / (kMaxVolumeLevel));
}
else
{
// Pass zero if the level is unchanged
newMicLevel = 0;
}
// Keep track of the value AGC returns
_oldVoEMicLevel = newVoEMicLevel;
_oldMicLevel = currentMicLevel;
}
return 0;
}[/cpp]

2.    class AudioDeviceWindowsWave

(audio_device_wave_win.cc):

[cpp]WebRtc_UWord32                          _newMicLevel;
WebRtc_UWord32                          _minMicVolume;
[/cpp]

WebRtc_UWord32 _newMicLevel

WebRtc_UWord32 _minMicVolume

Where will set/change these variants values?

_newMicLevel value: |_ptrAudioBuffer->NewMicLevel();| in AudioDeviceWindowsWave::RecProc while processing AGC

3.    class TransmitMixer(transmit_mixer.cc)

WebRtc_UWord32 _captureLevel;

Codes listed below is the key to the microphone level values, including the level before processing & and the level after processed.

[cpp]WebRtc_Word32 TransmitMixer::APMProcessStream(
const WebRtc_UWord16 totalDelayMS,
const WebRtc_Word32 clockDrift,
const WebRtc_UWord16 currentMicLevel)
{
WebRtc_UWord16 captureLevel(currentMicLevel);

if (_audioProcessingModulePtr->gain_control()->set_stream_analog_level(
captureLevel) == -1)
{
WEBRTC_TRACE(kTraceWarning, kTraceVoice, VoEId(_instanceId, -1),
“AudioProcessing::set_stream_analog_level(%u) => error”,
captureLevel);
}

captureLevel =
_audioProcessingModulePtr->gain_control()->stream_analog_level();
// Store new capture level (only updated when analog AGC is enabled)
_captureLevel = captureLevel;

return 0;
}[/cpp]

4.    class AudioDeviceBuffer

WebRtc_UWord32 _currentMicLevel;

WebRtc_UWord32 _newMicLevel;

5.    class

Functional implement flow:

audio_device_wave_win.cc

Summary

The major code of calling webrtc APM(audio processing manager) is in the function APMProcessStream() of TransmitMixer class.

For example: we will do the audio processing here for the input audio frame(AudioFrame _audioFrame), calculating the microphone levels in the same time, then output the processed frame and the new microphone level.

[cpp]WebRtc_Word32 TransmitMixer::APMProcessStream(
const WebRtc_UWord16 totalDelayMS,
const WebRtc_Word32 clockDrift,
const WebRtc_UWord16 currentMicLevel)
{
WebRtc_UWord16 captureLevel(currentMicLevel);

if (_audioProcessingModulePtr->gain_control()->set_stream_analog_level(
captureLevel) == -1)
{
WEBRTC_TRACE(kTraceWarning, kTraceVoice, VoEId(_instanceId, -1),
“AudioProcessing::set_stream_analog_level(%u) => error”,
captureLevel);
}

captureLevel =
_audioProcessingModulePtr->gain_control()->stream_analog_level();
// Store new capture level (only updated when analog AGC is enabled)
_captureLevel = captureLevel;

return 0;
}[/cpp]

And here are some customized log ouputs I added to the source code to log the detail processing and microphone level value change of AGC processing

[code]…

CRITICAL  ; ( 3: 7:13:562 |    4) AUDIO DEVICE:    1    99;      4776; TransmitMixer::PrepareDemux, Near-end Voice Quality Enhancement (APM) processing, currentMicLevel=18 before processing

CRITICAL  ; ( 3: 7:13:562 |    0)        VOICE:    1    99;      4776; TransmitMixer::APMProcessStream, AudioProcessing::set_stream_analog_level(18)

CRITICAL  ; ( 3: 7:13:562 |    1)        VOICE:    1    99;      4776; TransmitMixer::APMProcessStream, AudioProcessing::get_stream_analog_level after processed(17)

CRITICAL  ; ( 3: 7:13:562 |    0) AUDIO DEVICE:    1    99;      4776; TransmitMixer::PrepareDemux,Measure audio level of speech after APM processing, currentMicLevel=18, energy=-1

CRITICAL  ; ( 3: 7:13:562 |    0) AUDIO DEVICE:    1    99;      4776; AudioDeviceBuffer: _ptrCbAudioTransport->RecordedDataIsAvailable return newMicLevel=4369

CRITICAL  ; ( 3: 7:13:562 |    0)      UTILITY:    1    99;      4776; AudioDeviceWindowsWave::RecProc AGC change of volume: => new=4369

CRITICAL  ; ( 3: 7:13:562 |    3) AUDIO DEVICE:    1    99;      5672; AudioMixerManager::SetMicrophoneVolume volume=4369

…..

[/code]

First week in the new company

Phase 1:  Learn & master the existing MediaSDK & the sample programs.

—————————————————————————-
1. Add audio output device enumlation and selection option for playertester
2. Add audio input device enumlation and selection option for castertester
3. Add Mute setting for castertester
4. Add Microphone boost option for castertester.
5. UI adjustment for castertester to be more easier to be understand the working flow.
—————————————————————————-
Osp command: kdvmedianethelp
—————————————————————————-
Phase 2: WebRTC audio engine : audio device module & audio processing module research

—————————————————
Current status:
—————————————————
1. AGC settings and proccessing should be working now.
2. Microphone volume control still not undering coding.

—————————————————
How does AGC works?
—————————————————
Step 1: Audio capture(audio_device_wave_win.cc)
bool AudioDeviceWindowsWave::ThreadProcess()
Step 2: put into AudioBuffer and NewMicLevel()
WebRtc_Word32 AudioDeviceWindowsWave::RecProc(LONGLONG& consumedTime)
Step 3: Do audio processing/encoding/transmit tasks….
{
This is the key!!!
}
Step 4: Set new microphone volume thru _mixerManager(AudioMixerManager).
audio_mixer_manager_win.cc

—————————————————
Issues encounterred with
—————————————————
There are lot’s of differences between the Kdv audio modules and the WebRTC audio modules(audio device module and

maybe some others)

—————————————————
About microphone volume control(not relevant, curious only):
—————————————————
1. Client side audio mixing?
2. AGC may bring with side-effects with audio signal(audience only)?
3.

—————————————————

WebRTC音视频引擎研究(1)–整体架构分析

WebRTC: how to use audio process module?

WebRTC
WebRTC

My current job responsiblity is researching on WebRTC, and the first task is wrapping a class from WebRTC to process audio frames to implement functions of audio AEC, AGC, NS, High pass filter etc.

Information list below is from WebRTC.org, you can also view it by visiting http://www.webrtc.org, or it’s code.

Continue reading “WebRTC: how to use audio process module?”

Windows下编译WebRTC全步骤

1. 下载安装svn客户端,例如TortoiseSVN。安装完后,svn执行目录自动被添加入系统环境变量PATH中。

2. 下载并安装msysgit和Tortoisegit

Tortoisegit下载地址:http://code.google.com/p/tortoisegit/downloads/list

选择适合自己系统的版本,下载并安装(注:Tortoisegit只是一个gui,必须安装msysgit)
Tortoisegit安装时会找到git目录并自动配置好。把msysgit中bin目录手动添加到系统环境变量PATH中,比如我的目录是“D:/Program Files/Git/bin”
3. 下载并安装Python,建议安装Python2.6
下载地址:http://www.python.org/download/releases/2.6/。安装后Python执行目录自动被添加入系统环境变量PATH中。
4. 下载并配置depot_tools
    建立一个存放depot_tools的目录,command进入该目录
    svn co http://src.chromium.org/svn/trunk/tools/depot_tools
    下载后把depot_tools目录手动添加到系统环境变量PATH中
5. 建立WebRTC的源码目录
    比如E:/Developer/WebRTC/
6. 打开cmd,进入第5步建立的源码目录
7. 执行:  gclient config https://webrtc.googlecode.com/svn/trunk
        或者 gclient.bat config https://webrtc.googlecode.com/svn/trunk
8. 执行:  gclient sync –force (注意这里是两个- -,wordpress会把它变成一个全角的长—)
        或者 gclient.bat sync –force (注意这里是两个- -,wordpress会把它变成一个全角的长—)
9. 执行:  gclient runhooks –force (注意这里是两个- -,wordpress会把它变成一个全角的长—)
        或者 gclient.bat runhooks –force (注意这里是两个- -,wordpress会把它变成一个全角的长—)
10. 源码目录下应该已经含有webrtc.sln
  • webrtc 会用到Windows SDK 7.1,如果不想安装SDK,可以从这里下所依赖的文件:svn co http://vsfiltermod.googlecode.com/svn/trunk/src/BaseClasses,然后把文件放置在这个路 径:C:/Program Files/Microsoft SDKs/Windows/v7.1/Samples/multimedia/directshow/baseclasses (感谢乐得思蜀的方案)我的系统为Windows 7 32bit,不知为何在第8步只能执行gclient.bat sync –force才成功,直接执行gclient sync –force无法成功。
  • 如果你只安装了Visual Studio 2010,那么在gclient sync –force执行到最后会提示”Do not know how to convert MSVS attribute UseOfMFC”,可能对2010支持还不好,因为我系统里还有Visual Sduidio 2005,所以没有碰到这个问题,但是不用担心,因为最终的webrtc.sln照样会生成的,不影响看代码。
  • 如果遇到git –version return 1错误,应该是git目录没配置到PATH环境变量中,配置好后重启机器。
  • 如果你用的是Visual C++ express,那对不起,你还需要安装一下WinDDK。
  • 如果你碰到下面这个错误,我可以很高兴的告诉你,你的问题跟我碰到的一模一样,不一样的可能是,我花了N长时间才找出原因,你只要看到我这篇文章就可以搞定了。
[code]—— Build started: Project: peerconnection_client, Configuration: Debug Win32 ——
video_render_module.lib(video_render_windows_impl.obj) : error LNK2001: unresolved external symbol _CLSID_DxDiagProvider
video_render_module.lib(video_render_windows_impl.obj) : error LNK2001: unresolved external symbol _IID_IDxDiagProvider
video_render_module.lib(video_render_direct3d9.obj) : error LNK2019: unresolved external symbol _Direct3DCreate9@4 referenced in function "private: int __thiscall webrtc::VideoRenderDirect3D9::InitializeD3D(struct HWND__ *,struct _D3DPRESENT_PARAMETERS_ *)" (?InitializeD3D@VideoRenderDirect3D9@webrtc@@AAEHPAUHWND__@@PAU_D3DPRESENT_PARAMETERS_@@@Z)
E:\workspace\webtrc\trunk\Debug\peerconnection_client.exe : fatal error LNK1120: 3 unresolved externals</div>
<div>[/code]

错误原因:安装了错误版本的DirectX SDK,去下载安装Microsoft DirectX SDK (June 2010)