Trouble shooting: step by step to analysis crashes

This post’s goal is to guide a starter to analysis a crash by reading into the assemble code.

But the example listed here is not a good one, because the crash point is not an obvious one, the real reason of the crash for this example is still remain uncovered.

My point here is, we can use a such kind of way to analysis some crash, and once you read this post, you can start the first step. If you run into any problems when analysis your crash, well, we can discuss wih them together here. Here we go. Continue reading “Trouble shooting: step by step to analysis crashes”

A common bug of HD3 series terminals

An issue of call establishment delay when conferencing with Polycom MCU RMX2000

The situation was

1. Meeting entities
1). Polycom MCU: Polycom RMX 2000, version ID: 8.3.0
2). Kedacom HD3 H600 SP4

2. Call scenario
HD3 joined a multi-point conference with RMX2000.
1) All the H.225 and H.245 processes were OK.
2) OLC request from both side returned with ACK.
3) The audio packets could be captured right after the OLC ACK.
4) The video packets from HD3 sent right after got the OLC ACK from the MCU.
5) HD3 could not receive any video packets from the MCU.
6) HD3 waiting for a terminalYouAreSeeing conferenceIndication from the MCU to switch the status to InConf…
7) 20 seconds later we finally got the terminalYouAreSeeing indication. Along with the terminalYouAreSeeing, we got the video.

Seems the MCU was waiting for a command to switch its status to an established-mode.
But we just don’t know what it is, even after tested lot’s of MTs from Polycom, Tendburg, HUAWEI, ZTE, which all of them just working fine.

What we got is only that it must be a HD3’s bug.

After a long long times comparison with the pcap file, the only difference was the H.224 channel.
We did not open the H.224(FECC) channel together with the audio, video and H.239 channel, this caused the RMX2000 to wait 20 seconds to send the terminalYouAreSeeing indication.
It’s a yet-another-long-existing bug, we survived a long time, but today we finally ran into the consequence.

PCAP file: a-common-bug-of-hd3-series-terminals.pcap

RTCP and AVPF related missing features

Most of the missing features are AVPF related, which is defined in RFC4585 and RFC5104.

RFC4585: Extended RTP Profile for Real-time Transport Control Protocol (RTCP)-Based Feedback (RTP/AVPF)
https://www.rfc-editor.org/rfc/rfc4585.txt
RFC5104:  Codec Control Messages in the RTP Audio-Visual Profile with Feedback (AVPF)
https://www.rfc-editor.org/rfc/rfc5104.txt

AVPF contains a mechanism for conveying such a message, but did not specify for which codec and according to which syntax the message should conform.  Recently, the ITU-T finalized Rec.H.271, which (among other message types) also includes a feedback message.  It is expected that this feedback message will fairly quickly enjoy wide support.  Therefore, a mechanism to convey feedback messages according to H.271 appears to be desirable.

RTCP Receiver Report Extensions
1. CCM – Codec Control Message
2. FIR – Full Intra Request Command
A Full Intra Request (FIR) Command, when received by the designated
media sender, requires that the media sender sends a Decoder Refresh
Point (see section 2.2) at the earliest opportunity.  The evaluation
of such an opportunity includes the current encoder coding strategy
and the current available network resources.

FIR is also known as an “instantaneous decoder refresh request”,
“fast video update request” or “video fast update request”.

3. TMMBR – Temporary Maximum Media Stream Bit Rate Request
4. TMMBN – Temporary Maximum Media Stream Bit Rate Notification

Example from RFC5104:

Receiver A: TMMBR_max total BR = 35 kbps, TMMBR_OH = 40 bytes
Receiver B: TMMBR_max total BR = 40 kbps, TMMBR_OH = 60 bytes

For a given packet rate (PR), the bit rate available for media
payloads in RTP will be:

Max_net media_BR_A =
TMMBR_max total BR_A – PR * TMMBR_OH_A * 8 … (1)

Max_net media_BR_B =
TMMBR_max total BR_B – PR * TMMBR_OH_B * 8 … (2)

For a PR = 20, these calculations will yield a Max_net media_BR_A =
28600 bps and Max_net media_BR_B = 30400 bps, which suggests that
receiver A is the limiting one for this packet rate.  However, at a
certain PR there is a switchover point at which receiver B becomes
the limiting one.  The switchover point can be identified by setting
Max_media_BR_A equal to Max_media_BR_B and breaking out PR:

TMMBR_max total BR_A – TMMBR_max total BR_B
PR =  ——————————————- … (3)
8*(TMMBR_OH_A – TMMBR_OH_B)

5. TSTR – Temporal-Spatial Trade-off Request

5. TSTN – Temporal-Spatial Trade-off Request

6. VBCM – H.271 Video Back Channel Message

7. RTT – Round Trip Time
A receiver that receives a request closely after
sending a decoder refresh point — within 2 times the longest round
trip time (RTT) known, plus any AVPF-induced RTCP packet sending
delays — should await a second request message to ensure that the
media receiver has not been served by the previously delivered
decoder refresh point.  The reason for the specified delay is to
avoid sending unnecessary decoder refresh points.

8. NACK
8a. PLI – Picture Loss Indication
8b. SLI – Slice Loss Indication
8c. RPSI – Reference Picture Selection Indication

Here’s a sample INVITE command relaying from FreeSWITCH:

INVITE sip:1009@10.21.1.80:5060;transport=tcp SIP/2.0
Via: SIP/2.0/TCP 192.168.100.96;branch=z9hG4bK6p37yQX86QXar
Route: <sip:1009@10.21.1.80:5060>;transport=tcp
Max-Forwards: 69
From: "Extension 1008" <sip:1008@192.168.100.96>;tag=DKK4FpBB3ptSS
To: <sip:1009@10.21.1.80:5060;transport=tcp>
Call-ID: 0199ec1f-9e53-1233-8583-000c29f7d152
CSeq: 77747697 INVITE
Contact: <sip:mod_sofia@192.168.100.96:5060;transport=tcp>
User-Agent: FreeSWITCH-mod_sofia/1.7.0+git~20150614T062551Z~a647b42910~64bit
Allow: INVITE, ACK, BYE, CANCEL, OPTIONS, MESSAGE, INFO, UPDATE, REGISTER, REFER, NOTIFY, PUBLISH, SUBSCRIBE
Supported: timer, path, replaces
Allow-Events: talk, hold, conference, presence, as-feature-event, dialog, line-seize, call-info, sla, include-session-description, presence.winfo, message-summary, refer
Content-Type: application/sdp
Content-Disposition: session
Content-Length: 495
X-FS-Support: update_display,send_info
Remote-Party-ID: "Extension 1008" <sip:1008@192.168.100.96>;party=calling;screen=yes;privacy=off

v=0
o=FreeSWITCH 1436150633 1436150634 IN IP4 192.168.100.96
s=FreeSWITCH
c=IN IP4 192.168.100.96
t=0 0
m=audio 16890 RTP/AVP 96 0 8 101
a=rtpmap:96 opus/48000/2
a=fmtp:96 useinbandfec=1; stereo=0; sprop-stereo=0
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-16
a=ptime:20
m=video 22404 RTP/AVP 96
b=AS:1024
a=rtpmap:96 H264/90000
a=fmtp:96 profile-level-id=42801F
a=rtcp-fb:96 ccm fir tmmbr
a=rtcp-fb:96 nack
a=rtcp-fb:96 nack pli

Sample SDP of WebRTC for Firefox

GET /socket.io/1/websocket/GgKg1qt9TCXtfPb6n2g0 HTTP/1.1
Host: 172.16.203.1:2013
User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:36.0) Gecko/20100101 Firefox/36.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate
Sec-WebSocket-Version: 13
Origin: http://172.16.203.1:2013
Sec-WebSocket-Key: pPQe97SI5k09yaPnVLa2RQ==
Connection: keep-alive, Upgrade
Pragma: no-cache
Cache-Control: no-cache
Upgrade: websocket

HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: Lm5/9dDv4pjyphQQeswS+V+AiKc=

..1::..|.'kI..Q..I
...Q^.U...BK......IIP.F....Q'.A...z..T5:::{"name":"log","args":[[">>> Message from server: ","Room foo has 1 client(s)"]]}.`5:::{"name":"log","args":[[">>> Message from server: ","Request to create or join room","foo"]]}.$5:::{"name":"joined","args":["foo"]}.B5:::{"name":"emit(): client GgKg1qt9TCXtfPb6n2g0 joined room foo"}..$.N...t _. {I.l ..+iW.)...l{V.=8..l}K.noW.<:I.*sE..g.Z5:::{"name":"log","args":[[">>> Message from server: ","Got message: ","got user media"]]}.~.x5:::{"name":"message","args":[{"type":"offer","sdp":"
v=0
o=mozilla...THIS_IS_SDPARTA-38.0 2820695485956467000 0 IN IP4 0.0.0.0
s=-
t=0 0
a=fingerprint:sha-256 7C:7B:AE:C2:AE:ED:14:39:A4:7A:EE:4B:FB:FE:90:90:E8:A1:0B:C1:50:FC:C8:9C:FA:28:68:22:EE:1C:F6:97
a=group:BUNDLE sdparta_0 sdparta_1
a=ice-options:trickle
a=msid-semantic:WMS *
m=audio 9 RTP/AVP 109 9 0 8
c=IN IP4 0.0.0.0
a=sendrecv
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=ice-pwd:db746395f7bf11179b1cddabf3ef6359
a=ice-ufrag:1c858a90
a=mid:sdparta_0
a=msid:{69b2b229-1dc0-4291-a703-aafe505d477b} {ebc6bb1c-8525-4a70-9601-354b53c5c103}
a=rtcp-mux
a=rtpmap:109 opus/48000/2
a=rtpmap:9 G722/8000/1
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=setup:actpass
a=ssrc:4051396866 cname:{f0f8a3ab-8c54-4694-872a-98dd14f0c821}
m=video 9 RTP/AVP 126 97
c=IN IP4 0.0.0.0
a=sendrecv
a=fmtp:120 max-fs=12288;max-fr=60
a=fmtp:126 profile-level-id=42e01f;level-asymmetry-allowed=1;packetization-mode=1
a=fmtp:97 profile-level-id=42e01f;level-asymmetry-allowed=1
a=ice-pwd:db746395f7bf11179b1cddabf3ef6359
a=ice-ufrag:1c858a90
a=mid:sdparta_1
a=msid:{69b2b229-1dc0-4291-a703-aafe505d477b} {34f61d33-9fe4-42ff-8e2b-ef9c465c6f67}
a=rtcp-fb:120 nack
a=rtcp-fb:120 nack pli
a=rtcp-fb:120 ccm fir
a=rtcp-fb:126 nack
a=rtcp-fb:126 nack pli
a=rtcp-fb:126 ccm fir
a=rtcp-fb:97 nack
a=rtcp-fb:97 nack pli
a=rtcp-fb:97 ccm fir
a=rtcp-mux
a=rtpmap:126 H264/90000
a=rtpmap:97 H264/90000
a=setup:actpass
a=ssrc:3993721606 cname:{f0f8a3ab-8c54-4694-872a-98dd14f0c821}

 

 

Restore blogroll function for your WordPress

Found some features were missing after upgraded my blog to the latest version, WordPress v4.0, such as blogrolls.
Tried to find it all over the dashboard, but got no luck.
So I turned to the almighty Google, finally got the solution, and it’s really easy. All you need to do is adding the following code to the end of functions.php of your theme:

add_filter('pre_option_link_manager_enabled','__return_true');

 

 

About H.235 encryption algorithms

[20150822 Update] For the record, I didn’t find the right ITU-REC document when I wrote this post, and misguided by a claim of HUAWEI VP9650 which says it has AES256 supported, but when I sending out a call from VP9650, it showed a new DH group DH1536, so I made my conclusion arbitary, DH1536 means AES256, obviously, it’s a terrible mistake.

Planning to upgrade to H.235 encryption from AES128 to AES256, but don’t know where to start.

Did not find a short way to achieve that.

We wished to get some informations about AES256 by capturing some pcap files of some other Video Conference solution providers.

But did not find a device which actually supports H.235 + AES256 feature.

So we returned to the ITU-REC document for more details.  The right ITU-REC should be T-REC-H.235.6-201401-I!!PDF-E.pdf.

Some key steps of implementing H.235.

1. SETUP: Caller send a public key token DHSet of in H.225 SETUP, which includes:

1) halfkey: contains the random public key of one party

2)modsize: contains the DH-prime

3)generator: contains the DH-group

1. Public key token DHSet of in H.225 SETUP
1. Public key token DHSet of in H.225 SETUP

2. CONNECT: Callee generate a private key token by using the public key of caller, and send it back to the caller in its H.225 CONNECT, also including halfkey, modsize and generator.

2. Private key token of DHSet in H.225 CONNECT
2. Private key token of DHSet in H.225 CONNECT

3. TCS: Both caller and callee send their H.245 TCS with H.235 capabilities.

3. Sample H.245 TCS with H.235 capability
3. Sample H.245 TCS with H.235 capability

4. MasterSlave determination

5. Master generate a media key which will be used to encrypt/decrypt the media.

5a. OLC(from master): Open a logical channel with a specified H.235 media, and send to the slaver

H235Key in H.245 OLC
H235Key in H.245 OLC

5b. OLC ACK(from master): Reply the media key to the OLC requester(Slaver)

H235Key in H.245 OLC ACK
H235Key in H.245 OLC ACK

6. Some other H.245 request/indication messages, such as encryptionUpdateRequest, encryptionUpdate.

H.235 encryption related node definitions:

1. The tokenOID in H.225 SETUP and CONNECT message:
1a. H600’s tokenOID
Item0: 0.0.8.235.0.1.5
Item0: 0.0.8.235.0.1.5
Item2: 0.0.8.235.0.3.24
Item3: 0.0.8.235.0.3.43

1b. Huawei MCU’s tokenOID
ProductId: VP9650, versionId: V200R001C02B018SP07 Apr 28 2014 16:15:31+08
1. Public key token DHSet of in H.225 SETUP - HUAWEI-MCU-VP9650 3. Sample H.245 TCS with H.235 capability - HUAWEI-MCU-VP9650
Item0: 0.0.8.235.0.1.5
Item1: 0.0.8.235.0.3.44
Item2: 0.0.8.235.0.3.43
Item3: 0.0.8.235.0.3.24

1c. Huawei TE40’s tokenOID
ProductId: TEx0, versionId: Release 1.1.24.5
Item0: 0.0.8.235.0.3.43
Item1: 0.0.8.235.0.3.24

http://www.oid-info.com/get/0.0.8.235.0.1.5
http://www.oid-info.com/get/0.0.8.235.0.3.24
http://www.oid-info.com/get/0.0.8.235.0.3.43
http://www.oid-info.com/get/0.0.8.235.0.3.44

“T”      {itu-t (0) recommendation (0) h (8) 235 version (0) 2 5} {itu-t (0) recommendation (0) h (8) 235 version (0) 1 5} Used in Procedures I and IA as the baseline ClearToken for the message authentication and replay protection and optionally also for Diffie-Hellman key management as described in D.7.1.
“DH1024” {itu-t (0) recommendation (0) h (8) 235 version (0) 2 43} 1024-bit DH group
“DH1536” {itu-t (0) recommendation (0) h (8) 235 version (0) 3 44} 1536-bit DH group
From chapter D.11 List of object identifiers of <T-REC-H.235-200308-S!!PDF-E.pdf>, P.70-71

[20150822 Update]

Earlier when I wrote this post, I didn’t find the right ITU/T standard(T-REC-H.235.6-201401-I!!PDF-E.pdf).

The information I got was that HUAWEI VP9650 supports AES256, and when I tried to sending out a call on VP9650, I got  to know it supports following DH groups:

Item0: 0.0.8.235.0.1.5
Item1: 0.0.8.235.0.3.44
Item2: 0.0.8.235.0.3.43
Item3: 0.0.8.235.0.3.24

So I made my conclusion arbitarty that DH1536 is our goal: AES256, but turned out I was terribly wrong. (I don’t know why VP9650 sending out max to DH1536 while it claims having AES 256 supported)

DH group - DH1536

2. Media encryption algorithm definitions on H.245 TCS, OLC, OLC ACK,etc:

The most frequently used/seem types are:
2a. AES 128 bit CBC: 2.16.840.1.101.3.4.1.2
2b. DES 56 bit CBC(Voice encryption using DES in CBC mode and 512-bit DH-group): 1.3.14.3.2.7

2.16.840.1.101.3.4.1.1 – id-aes128-ECB
2.16.840.1.101.3.4.1.2 – id-aes128-CBC
2.16.840.1.101.3.4.1.3 – id-aes128-OFB
2.16.840.1.101.3.4.1.4 – id-aes128-CFB
2.16.840.1.101.3.4.1.6 – id-aes128-GCM
2.16.840.1.101.3.4.1.7 – id-aes-CCM
2.16.840.1.101.3.4.1.21 – id-aes192-ECB
2.16.840.1.101.3.4.1.22 – id-aes192-CBC
2.16.840.1.101.3.4.1.23 – id-aes192-OFB
2.16.840.1.101.3.4.1.24 – id-aes192-CFB
2.16.840.1.101.3.4.1.26 – id-aes192-GCM
2.16.840.1.101.3.4.1.27 – id-aes192-CCM
2.16.840.1.101.3.4.1.41 – id-aes256-ECB
2.16.840.1.101.3.4.1.42 – id-aes256-CBC
2.16.840.1.101.3.4.1.43 – id-aes256-OFB
2.16.840.1.101.3.4.1.44 – id-aes256-CFB
2.16.840.1.101.3.4.1.46 – id-aes256-GCM
2.16.840.1.101.3.4.1.47 – id-aes256-CCM

Source : http://www.alvestrand.no/objectid/2.16.840.1.101.3.4.1.html

About DH key exchange:
1. http://baike.baidu.com/view/551692.htm
2. http://www.rosoo.net/a/201507/17349.html
3. Diffie-Hellman, http://www.cryptopp.com/wiki/Diffie-Hellman
4. rfc3526: More Modular Exponential (MODP) Diffie-Hellman groups for Internet Key Exchange (IKE), https://www.ietf.org/rfc/rfc3526.txt
5. Huawei VP9650(which claims having AES256 supported): http://e.huawei.com/cn/related-page/products/enterprise-network/telepresence-video-conferencing/infrastructure/vp9600/TPVC_MCU_VP9600

Add compilation time cost for each source file in make file

Encountered an issue of extreme long time compiling days ago, so we tried to add a time cost output in the Makefile to locate what on earth happened during the compilation.

There are two Makefiles need to be modified, one is Linux based Makefile, another is Android based Makefile

Linux Makefile:

##------------------------------------------------------------------------
## Rules
## Suffix rules
$(SRC_DIR)/%.o: $(SRC_DIR)/%.s
    $(CC) -c -o $@ $(CFLAGS) $<
$(SRC_DIR)/%.o: $(SRC_DIR)/%.cpp
    @now=`date`; echo "==========compile at $${now}"
    $(CC) -c -o $@ $(CFLAGS) $<

or

@echo "Compilation begin at `date`"
@echo Compilation in progress, please wait ...
@sleep 1
@now=`date`; echo "Compilation end at $${now}"

Test Makefile: http://rg4.net/p/tools/add-time-output-to-makefile/common.mk

Android Makefile:
You need to modify NDK common make file, definitions.mk, to archive this, it’s locating at /path-to-ndk/android-ndk-r7c/build/core/definitions.mk

_CC   := $$(NDK_CCACHE) $$($$(my)CXX)

# Jacky, add timestamp to the compile output
# 1) for linux
# NOW := `date`
# 2) for windows
NOW        := %TIME:~0,10%

_TEXT := "Compile++ $$(call get-src-file-text,$1), start at: $$(NOW)"

$$(eval $$(call ev-build-source-file))
endef

Test Makefile: http://rg4.net/p/tools/add-time-output-to-makefile/definitions.mk