Just finished the MSE entrance exam of USTC. Hooray!!

After weeks of learning, finally I finished the entrance exam. The result will be revealed soon, maybe one and a half monthes later.

I was told not to be worry about the exam, now I’m sure, :D. Yeah!!!

But there are courses still remain to be attended.

Here are the rest courses & schedules:

课程名称

教材(名称、编者及出版社)

任课教师姓名职称

总学时

学习方式

上课时间

现代软件工程

教材:《面向对象软件工程》,Stephen R. Schach著,黄林鹏、徐小辉等译,机械工业出版社,2008版。

李俊

60/3

课堂授课

5月4,5 11,12 25,26

软件项目管理导论

教材:《软件项目管理案例教程》(第2版)韩万江,蒋立新著 机械工业出版社,2010年

王子磊

40/2

课堂授课

6月 15,16 22,23

软件质量保证与测试

教材:《软件质量保证.测试与评价》杨根兴等,清华大学出版社,2007年

关胜晓

40/2

课堂授课

7月6,7 13,14

知识产权

知识产权基础教程 清华大学出版社 作者:王兵

宋澜副教授

20/1

课堂授课

7月27,28

信息检索

电子信息检索与利用 机械工业出版社作者:刘绿茵

康宇副教授

20/1

课堂授课

8月10,11

人机界面技术

人机交互:以用户为中的设计和评估,董建明 等编著,清华大学出版社,2007年

郑全

30/1.5

课堂授课

8月17,18上课

软件体系结构与设计模式

教材:《设计模式:可复用面向对象软件的基础》,作者:Erich Gamma 等 翻译:李英军等 机械工业出版社2007年版

李俊

60/3

课堂授课

8月31, 9月1 , 7,8 14,15

网络服务系统

教材:《对等网络:结构,应用与设计》陈贵海,李振华著

清华大学出版社 2007.9

王雷

40/2

课堂授课

After all the courses are over, and passed, then comes with the biggest event, The Thesis. However, I can arrange my time more freely then.

Can’t wait to that.

Install DirectX SDK (June 2010) failed with error code S1023

I was trying to install Direct SDK (June 2010), everything went just fine, but when the installation started to do some post installation processes, it returned with an error like this:

install-directx-sdk-june-2010-failed-in-post-installation-steps

Setup failed.

Errors were encountered during installation of redistributable packages. Please close all open programs and try running setup again. If problems persist, contact DirectX Developer Support.

Error Code: S1023

I have tried looking in the log files in C:\Windows\Logs. There are 2 log files, DirectX.log and DirectX_SDK.log. I do not see any noticeable signs of “error” or “failure” of any kind.

In fact, the very last line in DirectX.log is:

dsetup32: Installation ended with value 0 = Installation succeeded

However the installation did fail. When I re-installed it for a second time, then finally found out that maybe has something to do with the VC 2010 redistributable run-time library. So I uninstalled it. and redo the installation for a third time, it installed perfectly.

So put it in a simple way, Solution for this issue:

go to Control Panel>Program & Features and uninstall programs like this:
Microsoft Visual C++ 2010 x86/x64 redistributable – 10.0.(number over 30319)

Coming event: entrance examination for postgraduate (May 26, 2013)

Finaly, it’s coming, after postponed for over 6 months. There will be three courses:

  1. Advanced Mathematics
  2. English
  3. Specialized course

And the preparations courses started at last weekend, I’m busy preparing for it now, although I don’t think it’ll be a problem for me after these monthes of study.

But still it’s an event I can not let down of my guard, I’d draw enough attention about it.

 

MP4 media file container with H.264 video and AAC audio

After all audio processing module has been full supported, I migrated the webrtc version to the latest version(update date: 5/9, 2013, svn code: 3988), I got a new assignment today, that is implement mp4 file container for the SDK.

During the discussion of the assignment, I offered my suggestion for the implementation solution, that was import ffmpeg for the SDK, because I was dealing with ffmpeg for over 8 years. By using ffmpeg, it really helps a lot in programming multimedia softwares, codec, container, scaling, streaming, so something else likewise.

However, I got a No, because “ffmpeg is too big for us”. True, this is something we shall concern.

So I can but only turn to mpeg4-ip/mp4v2.

And one more issue must be mentioned here is, the major audio codec of the SDK is G.722.1c. This means if we are intending to have a standard media file container for the SDK, transcode for the audio is required, otherwise, this version of MP4 container can support Windows only, and even more we need to add a DirectX filter for running on Windows.

Mission launched.

Preparations:

FAAC source code: http://sourceforge.net/projects/faac/ or

AAC implementation in libstagefright of android: http://ffmpeg.zeranoe.com/builds/source/external_libraries/vo-aacenc-0.1.2.tar.xz

libmp4v2 source code: http://code.google.com/p/mp4v2/

Directly to to codes

I must say some of the codes I got from my employer are getting really ugly:

  • No interface & variants comments
  • No sample programs
  • I can have only part of the source code.

So I digged into it myself for the first round of code study, trying to understand what these bouches of codes do, and how they work.

Current status in the previous two days

There are lots of stories during the code study, and after one-full-day’s try and try, I finally managed to write a simple program to run it up, but I still not sure whether it’s the right interfaces and right sequences I’m calling to run it up among the dozens of exported interfaces.

And when I trying to add libmp4v2 to it, I found an even big issue here which may turn the solution down:

We are using mono audio(encoded to G.722 1c) in our products, however, when I transcoded them to AAC and recorded to a MP4 container together with H.264 video, there will be a tremendous noise in the  audio.

I don’t know the exact reason.

So I did some test for it, writing a test program to record stereo audio to Mp4 by using libfaac & libmp4v2, everything works just fine.

Why???

Can someone tell me?

One suspecious phenomenon is the audio encoded by the FAAC always been showed as stereo when playing with VLC, whatever the audio format you inputed to FAAC, mono or stereo.

Code segment of encoding mono PCM audio to AAC

[cpp]
hEncoder = faacEncOpen(32000, 1,&samplesInput, &maxBytesOutput);

[/cpp]

Next step ( alternative solution for the problem )

After discussed with my supervisor, we decided to  not transcode the audio to AAC, but G.711 , for a temporary solution, but this means:

1. The audio quality will be poorer.

2. But loading for recording a MP4 media file container will be reduced.

I will do this in the next week.

LINK : fatal error LNK1104: cannot open file ‘xxx.map’

I was upgrading my IDE from VS 2008 to VS2010 when I located this issue.

After google it for a long time, I final found this webpage from Microsoft, telling me that it is a bug of VS2010, http://connect.microsoft.com/VisualStudio/feedback/details/665896.

And then I found that this bug has not been fixed yet per this post: https://connect.microsoft.com/VisualStudio/feedback/details/540975/lnk1104-intermittantly-when-building

However, it’s all about .map file for your library, and after discussed with the team leader, we decided let go of the .map file. and this issue no more bugs me.

LINK fatal error LNK1104-cannot open file xxx.map
Continue reading “LINK : fatal error LNK1104: cannot open file ‘xxx.map’”

WebRTC audio processing continuous: AGC effect optimizing (setting parameter adjustment)

After days of working,  all audio processing functions seems are working in the KdvMediaSDK now, and the next step will be getting a proper set for the audio processing algorithms to run under certain environments & circumstance.

I will start it by beginning with AGC module too. Here are the parameters of AGC algorithm. Continue reading “WebRTC audio processing continuous: AGC effect optimizing (setting parameter adjustment)”

Preparing for tomorrow’s Computer Networking(A Top-Down Approach) exam

Tomorrow morning will be my exam for Computer Networking(A Top-Down Approach).

I coded lots of applications related with netwroking, whatever for small embedded server or large-scale high concurrency server, whatever TCP based or UDP based, including but not limited to ICMP, HTTP, FTP, SMTP, RTSP, RTP, SIP protocols.

I was so sure about this course once, but after attended this course then I found out that this is really different than what I thought, especially when I got the exercise materials. Because what I did is only the TOP rather than TOP-DOWN, although I don’t think I gonna get into the DOWN level in the furture(This is at least I’m thinking now). And what I’m familiar with is network programming, theoretical stuffs? Oops, no!

So I have to do more for passing this course. Continue reading “Preparing for tomorrow’s Computer Networking(A Top-Down Approach) exam”

Reading codes of WebRTC: Deep into WebRTC Voice Engine(draft)

Introduction of this document

This module includes software based acoustic echo cancellation (AEC), automatic gain control (AGC), noise reduction, noise suppression and hardware access and control across multiple platforms.

My ultimate goal will be wrapping an independent module out of WebRTC’s Voice Engine for our product, and the first task is to get AGC implemented base on current KdvMediaSDK implementations which is not so much the same in interfaces with WebRTC).

Keywords: WebRTC, audio processing, AEC, AGC, noise reduction, noise suppression.

Overall architecture of WebRTC

The overall architecture looks something like this:

Overall architecture of WebRTC
Overall architecture of WebRTC

(Image from http://webrtc.org)

WebRTC Voice Engine – AGC control workflow

WebRTC Voice Engine - AGC control workflow
WebRTC Voice Engine – AGC control workflow

You can download my original Visio file here:

http://rg4.net/p/webrtc/webrtc.voiceengine.agc.vsd

You can modify & distribute whatever you wish, but if you made any improvement for this chart, please send me a copy by mail. That’ll benefit a lot more people. Thank you.

Target/related source codes

Major source codes:

l  audio_device_wave_win.cc: %WEBRTC%\src\modules\audio_device\main\source\win\audio_device_wave_win.cc

l  audio_device_buffer.cc:

l  audio_device_utility.cc:

l  audio_mixer_manager_win.cc:

l  voe_base_impl.cc: %WEBRTC%\src\voice_engine\main\source\voe_base_impl.cc

l  transmit_mixer.cc : %WEBRTC%\src\voice_engine\main\source\transmit_mixer.cc

l  level_indicator.cc AudioLevel src\voice_engine\main\source\level_indicator.cc

Utility source codes:

event_win_wrapper.cc

thread_win_wrapper.cc

Detail interfaces & implementations

audio_device_wave_win.cc

It responsible for:

l  Audio capture

l  Get/Set Microphone Volume (I’m not sure what this volume means, hardware volume, or a virtual volume after audio processing, only because it is Get/Set through audio_device_mixer_manager.cc)

a. Audio capture.

Step 1: run audio capture in a thread named as ThreadProcess().

[cpp]bool AudioDeviceWindowsWave::ThreadProcess()

{

while ((nRecordedBytes = RecProc(recTime)) > 0)
{

}

}[/cpp]

Step 2: detail into RecProc() function, all capture parameters & the captured buffer will be saved to a variant |AudioDeviceBuffer* _ptrAudioBuffer|

[cpp]WebRtc_Word32 AudioDeviceWindowsWave::RecProc(LONGLONG& consumedTime)
{
……
// store the recorded buffer (no action will be taken if the #recorded samples is not a full buffer)

_ptrAudioBuffer->SetRecordedBuffer(_waveHeaderIn[bufCount].lpData, nSamplesRecorded);

// Check how large the playout and recording buffers are on the sound card.
// This info is needed by the AEC.

msecOnPlaySide = GetPlayoutBufferDelay(writtenSamples, playedSamples);
msecOnRecordSide = GetRecordingBufferDelay(readSamples, recSamples);

// If we use the alternative playout delay method, skip the clock drift compensation
// since it will be an unreliable estimate and might degrade AEC performance.

WebRtc_Word32 drift = (_useHeader > 0) ? 0 : GetClockDrift(playedSamples, recSamples);
_ptrAudioBuffer->SetVQEData(msecOnPlaySide, msecOnRecordSide, drift);


if (_AGC)
{
WebRtc_UWord32 newMicLevel = _ptrAudioBuffer->NewMicLevel();
if (newMicLevel != 0)
{
// The VQE will only deliver non-zero microphone levels when a change is needed.
WEBRTC_TRACE(kTraceStream, kTraceUtility, _id,”AGC change of volume: => new=%u”, newMicLevel);

// We store this outside of the audio buffer to avoid
// having it overwritten by the getter thread.
_newMicLevel = newMicLevel;
SetEvent(_hSetCaptureVolumeEvent);
}
}

}[/cpp]

b. Get/Set Microphone Volume

There are two other threads along with the major capture thread, they are

::DoGetCaptureVolumeThread()

::DoSetCaptureVolumeThread()

These threads will be always running waiting for a signal to Get or Set capture volume.

Things I’m still trying to figure out

There are so many definitions about microphone level or relevant. What I’m not sure is which volume is what volume? Here are some volume related definitions I confused with:

1.    class VoEBaseImpl(voe_base_impl.cc)

What’s the difference between currentVoEMicLevel and currentMicLevel in the codes below?

Which can be also compare to the variants _oldVoEMicLevel and _oldMicLevel defined in voe_base_impl.cc

WebRtc_UWord32 _oldVoEMicLevel

WebRtc_UWord32 _oldMicLevel

Where will set/change these variants values?

This code locates at voe_base_impl.cc

[cpp]WebRtc_Word32 VoEBaseImpl::RecordedDataIsAvailable(

const WebRtc_UWord32 currentMicLevel,
WebRtc_UWord32& newMicLevel)
{

// Will only deal with the volume in adaptive analog mode
if (isAnalogAGC)
{
// Scale from ADM to VoE level range
if (_audioDevicePtr->MaxMicrophoneVolume(&maxVolume) == 0)
{
if (0 != maxVolume)
{
currentVoEMicLevel = (WebRtc_UWord16) ((currentMicLevel
* kMaxVolumeLevel + (int) (maxVolume / 2))
/ (maxVolume));
}
}
// We learned that on certain systems (e.g Linux) the currentVoEMicLevel
// can be greater than the maxVolumeLevel therefore
// we are going to cap the currentVoEMicLevel to the maxVolumeLevel
// if it turns out that the currentVoEMicLevel is indeed greater
// than the maxVolumeLevel
if (currentVoEMicLevel > kMaxVolumeLevel)
{
currentVoEMicLevel = kMaxVolumeLevel;
}
}
// Keep track if the MicLevel has been changed by the AGC, if not,
// use the old value AGC returns to let AGC continue its trend,
// so eventually the AGC is able to change the mic level. This handles
// issues with truncation introduced by the scaling.
if (_oldMicLevel == currentMicLevel)
{
currentVoEMicLevel = (WebRtc_UWord16) _oldVoEMicLevel;
}
// Perform channel-independent operations
// (APM, mix with file, record to file, mute, etc.)
_transmitMixerPtr->PrepareDemux(audioSamples, nSamples, nChannels,
samplesPerSec,
(WebRtc_UWord16) totalDelayMS, clockDrift,
currentVoEMicLevel);
// Copy the audio frame to each sending channel and perform
// channel-dependent operations (file mixing, mute, etc.) to prepare
// for encoding.
_transmitMixerPtr->DemuxAndMix();
// Do the encoding and packetize+transmit the RTP packet when encoding
// is done.
_transmitMixerPtr->EncodeAndSend();
// Will only deal with the volume in adaptive analog mode
if (isAnalogAGC)
{
// Scale from VoE to ADM level range
newVoEMicLevel = _transmitMixerPtr->CaptureLevel();
if (newVoEMicLevel != currentVoEMicLevel)
{
// Add (kMaxVolumeLevel/2) to round the value
newMicLevel = (WebRtc_UWord32) ((newVoEMicLevel * maxVolume
+ (int) (kMaxVolumeLevel / 2)) / (kMaxVolumeLevel));
}
else
{
// Pass zero if the level is unchanged
newMicLevel = 0;
}
// Keep track of the value AGC returns
_oldVoEMicLevel = newVoEMicLevel;
_oldMicLevel = currentMicLevel;
}
return 0;
}[/cpp]

2.    class AudioDeviceWindowsWave

(audio_device_wave_win.cc):

[cpp]WebRtc_UWord32                          _newMicLevel;
WebRtc_UWord32                          _minMicVolume;
[/cpp]

WebRtc_UWord32 _newMicLevel

WebRtc_UWord32 _minMicVolume

Where will set/change these variants values?

_newMicLevel value: |_ptrAudioBuffer->NewMicLevel();| in AudioDeviceWindowsWave::RecProc while processing AGC

3.    class TransmitMixer(transmit_mixer.cc)

WebRtc_UWord32 _captureLevel;

Codes listed below is the key to the microphone level values, including the level before processing & and the level after processed.

[cpp]WebRtc_Word32 TransmitMixer::APMProcessStream(
const WebRtc_UWord16 totalDelayMS,
const WebRtc_Word32 clockDrift,
const WebRtc_UWord16 currentMicLevel)
{
WebRtc_UWord16 captureLevel(currentMicLevel);

if (_audioProcessingModulePtr->gain_control()->set_stream_analog_level(
captureLevel) == -1)
{
WEBRTC_TRACE(kTraceWarning, kTraceVoice, VoEId(_instanceId, -1),
“AudioProcessing::set_stream_analog_level(%u) => error”,
captureLevel);
}

captureLevel =
_audioProcessingModulePtr->gain_control()->stream_analog_level();
// Store new capture level (only updated when analog AGC is enabled)
_captureLevel = captureLevel;

return 0;
}[/cpp]

4.    class AudioDeviceBuffer

WebRtc_UWord32 _currentMicLevel;

WebRtc_UWord32 _newMicLevel;

5.    class

Functional implement flow:

audio_device_wave_win.cc

Summary

The major code of calling webrtc APM(audio processing manager) is in the function APMProcessStream() of TransmitMixer class.

For example: we will do the audio processing here for the input audio frame(AudioFrame _audioFrame), calculating the microphone levels in the same time, then output the processed frame and the new microphone level.

[cpp]WebRtc_Word32 TransmitMixer::APMProcessStream(
const WebRtc_UWord16 totalDelayMS,
const WebRtc_Word32 clockDrift,
const WebRtc_UWord16 currentMicLevel)
{
WebRtc_UWord16 captureLevel(currentMicLevel);

if (_audioProcessingModulePtr->gain_control()->set_stream_analog_level(
captureLevel) == -1)
{
WEBRTC_TRACE(kTraceWarning, kTraceVoice, VoEId(_instanceId, -1),
“AudioProcessing::set_stream_analog_level(%u) => error”,
captureLevel);
}

captureLevel =
_audioProcessingModulePtr->gain_control()->stream_analog_level();
// Store new capture level (only updated when analog AGC is enabled)
_captureLevel = captureLevel;

return 0;
}[/cpp]

And here are some customized log ouputs I added to the source code to log the detail processing and microphone level value change of AGC processing

[code]…

CRITICAL  ; ( 3: 7:13:562 |    4) AUDIO DEVICE:    1    99;      4776; TransmitMixer::PrepareDemux, Near-end Voice Quality Enhancement (APM) processing, currentMicLevel=18 before processing

CRITICAL  ; ( 3: 7:13:562 |    0)        VOICE:    1    99;      4776; TransmitMixer::APMProcessStream, AudioProcessing::set_stream_analog_level(18)

CRITICAL  ; ( 3: 7:13:562 |    1)        VOICE:    1    99;      4776; TransmitMixer::APMProcessStream, AudioProcessing::get_stream_analog_level after processed(17)

CRITICAL  ; ( 3: 7:13:562 |    0) AUDIO DEVICE:    1    99;      4776; TransmitMixer::PrepareDemux,Measure audio level of speech after APM processing, currentMicLevel=18, energy=-1

CRITICAL  ; ( 3: 7:13:562 |    0) AUDIO DEVICE:    1    99;      4776; AudioDeviceBuffer: _ptrCbAudioTransport->RecordedDataIsAvailable return newMicLevel=4369

CRITICAL  ; ( 3: 7:13:562 |    0)      UTILITY:    1    99;      4776; AudioDeviceWindowsWave::RecProc AGC change of volume: => new=4369

CRITICAL  ; ( 3: 7:13:562 |    3) AUDIO DEVICE:    1    99;      5672; AudioMixerManager::SetMicrophoneVolume volume=4369

…..

[/code]

科达NVR亮相Secutech,专业、开放、易用博好评

2013年4月24日—25日,第十六届台北国际安全博览会(Secutech)在世贸南港展览馆隆重举行,作为国内最早最专业的NVR厂商,科达携全系列NVR产品参加了本次展会,并向现场观众重点展示及介绍了科达NVR在专业应用、开放性、可靠性这三方面的实力,得到专业观众的广泛认可。

image001

Continue reading “科达NVR亮相Secutech,专业、开放、易用博好评”

RTSPPlayer update: a fix for ONVIF supports provided by the previous version

2.0 (2013/5/1) svn 68
a. I got some feedback these days telling me that he/she can view nothing but a black screen after
requested a stream. this version is a fix for it.
b. I also added an asynchronous message handler
to accept NDK library’s socket connection status callback for RTSPPlayer. you can view the implements
from here: http://rg4.net/archives/762.html
c. Removed ONVIF support temperary because lack of device for test, but you can still try windows version
of ONVIFPlayer for test if you need. http://rg4.net/onvifplayer