This post’s goal is to guide a starter to analysis a crash by reading into the assemble code.
But the example listed here is not a good one, because the crash point is not an obvious one, the real reason of the crash for this example is still remain uncovered.
My point here is, we can use a such kind of way to analysis some crash, and once you read this post, you can start the first step. If you run into any problems when analysis your crash, well, we can discuss wih them together here. Here we go.
As you know, you can get the call stack by adding a backtrace support for your app.
You can compile your app by adding -rdynamic, when the crash happened, you can use addr2line to locate the exact line of your source code which caused the crash. http://rg4.net/archives/1656.html
But when a formal product is released, we will not compile our components with -rdynamic, then we need to analysis the source code ourself.
Here is an example:
The crash report log which contains the call stack when the crash happened.
2015-7-17 16:37:28 ---------------------------------pid=31603,tid=32136<STACKAPP>------------------- ----------------SIGSEGV------------------- SI code = 1 (Address not mapped to object) Fault addr = 0x7 ----------------stack------------------- h323sadp() [0x806b9de] h323sadp() [0x806bc30] [0x2d5410] /opt/mcu/pas/libkdv323adapter.so(raAdd+0x3b) [0x51dbdb] /opt/mcu/pas/libkdv323adapter.so(+0x1436cf) [0x5206cf] /opt/mcu/pas/libkdv323adapter.so(rtAddBrother+0x35) [0x520765] /opt/mcu/pas/libkdv323adapter.so(+0x1107ef) [0x4ed7ef] /opt/mcu/pas/libkdv323adapter.so(pvtAdd+0x43) [0x4edac3] /opt/mcu/pas/libkdv323adapter.so(perDecodeChoice+0x2ad) [0x4bcf4d] /opt/mcu/pas/libkdv323adapter.so(perDecNode+0x557) [0x4bb1d7] /opt/mcu/pas/libkdv323adapter.so(perDecodeSequece+0x16d) [0x4bf2cd] /opt/mcu/pas/libkdv323adapter.so(perDecNode+0x50f) [0x4bb18f] /opt/mcu/pas/libkdv323adapter.so(perDecodeSequeceOF+0x1c0) [0x4bfe30] /opt/mcu/pas/libkdv323adapter.so(perDecNode+0x4e5) [0x4bb165] /opt/mcu/pas/libkdv323adapter.so(perDecodeSequece+0x1a1) [0x4bf301] /opt/mcu/pas/libkdv323adapter.so(perDecNode+0x50f) [0x4bb18f] /opt/mcu/pas/libkdv323adapter.so(perDecodeChoice+0x3b9) [0x4bd059] /opt/mcu/pas/libkdv323adapter.so(perDecNode+0x557) [0x4bb1d7] /opt/mcu/pas/libkdv323adapter.so(perDecodeChoice+0x318) [0x4bcfb8] /opt/mcu/pas/libkdv323adapter.so(perDecNode+0x557) [0x4bb1d7] /opt/mcu/pas/libkdv323adapter.so(+0xde634) [0x4bb634] /opt/mcu/pas/libkdv323adapter.so(emDecode+0xa2) [0x4ba962] /opt/mcu/pas/libkdv323adapter.so(cmEmDecode+0x5c) [0x4ba7cc] /opt/mcu/pas/libkdv323adapter.so(cmEvTransNewRawMessage+0x64) [0x4a4d24] /opt/mcu/pas/libkdv323adapter.so(decodeIncomingMessage+0x1b7) [0x5096a7] /opt/mcu/pas/libkdv323adapter.so(transH245Handler+0x32a) [0x50affa] /opt/mcu/pas/libkdv323adapter.so(tpktEvent+0x12a) [0x507d6a] /opt/mcu/pas/libkdv323adapter.so(Rv2SelectHandleEPollFds+0xe6) [0x527716] /opt/mcu/pas/libkdv323adapter.so(Rv2SelectWaitAndBlock+0x1bd) [0x527b1d] /opt/mcu/pas/libkdv323adapter.so(seliSelectUntil+0x44) [0x4e9a84] /opt/mcu/pas/libkdv323adapter.so(_Z15kdvCheckMessagev+0x32) [0x446462] h323sadp() [0x80687d5] /opt/mcu/nm/libosp.so(OspAppEntry+0x62c) [0x88648c] /opt/mcu/nm/libosp.so(+0x28e2d) [0x894e2d] /lib/libpthread.so.0() [0xd26b39] /lib/libc.so.6(clone+0x5e) [0x6dfd6e] ----------------context struct---------------- ct->uc_flags = 0 ct->uc_link = 0x(nil) ct->uc_sigmask = 00 ct->uc_stack.ss_sp = 0x(nil) ct->uc_stack.ss_flags = 0x2 ct->uc_stack.ss_size = 0x0
Get the libkdv323adapter.so and disassemble it, here is the code we get.
00140ba0 <raAdd>:
140ba0: 55 push %ebp
140ba1: 89 e5 mov %esp,%ebp
140ba3: 57 push %edi
140ba4: 56 push %esi
140ba5: 53 push %ebx
140ba6: 83 ec 1c sub $0x1c,%esp
140ba9: 8b 5d 08 mov 0x8(%ebp),%ebx //ra
140bac: 85 db test %ebx,%ebx //if ( ra == NULL )
140bae: 0f 84 ef 01 00 00 je 140da3 <raAdd+0x203>
140bb4: 8b 43 64 mov 0x64(%ebx),%eax //ra->threadSafe
140bb7: 85 c0 test %eax,%eax //if (ra->threadSafe)
140bb9: 0f 85 a1 00 00 00 jne 140c60 <raAdd+0xc0>
140bbf: 8b 73 24 mov 0x24(%ebx),%esi //ra->firstVacantElement
140bc2: 85 f6 test %esi,%esi //if (ra->firstVacantElement == NULL)
140bc4: 0f 84 7b 01 00 00 je 140d45 <raAdd+0x1a5>
140bca: 8b 46 08 mov 0x8(%esi),%eax //find variant allocatedElement
140bcd: 89 43 24 mov %eax,0x24(%ebx) //allocatedElement = ra->firstVacantElement;
140bd0: 8b 43 28 mov 0x28(%ebx),%eax //ra->firstVacantElement = (RAElement)((vacantNode *)allocatedElement)->next;
140bd3: 85 c0 test %eax,%eax //if (((vacantNode *)ra->workingSetElement) == NULL)
140bd5: 0f 84 aa 00 00 00 je 140c85 <raAdd+0xe5>
-------------------------raAdd+0x3b----------------------------------
140bdb: 8b 40 08 mov 0x8(%eax),%eax //((vacantNode *)ra->workingSetElement)->next
---------------------------------------------------------------------
140bde: 85 c0 test %eax,%eax //if (((vacantNode *)ra->workingSetElement) && ((vacantNode *)ra->workingSetElement)->next != NULL)
140be0: 0f 84 da 00 00 00 je 140cc0 <raAdd+0x120>
140be6: 89 43 28 mov %eax,0x28(%ebx) //ra->workingSetElement = (RAElement)(((vacantNode *)ra->workingSetElement)->next);
140be9: 8b 43 24 mov 0x24(%ebx),%eax //ra->firstVacantElement
140bec: 85 c0 test %eax,%eax //if (ra->firstVacantElement == NULL)
140bee: 0f 84 12 01 00 00 je 140d06 <raAdd+0x166>
140bf4: c7 40 04 00 00 00 00 movl $0x0,0x4(%eax) //((vacantNode *)ra->firstVacantElement)->prev = NULL;
140bfb: 8b 7e 0c mov 0xc(%esi),%edi //((vacantNode *)allocatedElement)->nodeIndex;
140bfe: b8 80 00 00 00 mov $0x80,%eax
140c03: 89 fa mov %edi,%edx
140c05: 89 f9 mov %edi,%ecx
140c07: c1 ea 03 shr $0x3,%edx
140c0a: 83 e1 07 and $0x7,%ecx
140c0d: d3 f8 sar %cl,%eax
140c0f: 08 44 13 6c or %al,0x6c(%ebx,%edx,1)
140c13: 8b 43 38 mov 0x38(%ebx),%eax
140c16: 83 c0 01 add $0x1,%eax
140c19: 3b 43 3c cmp 0x3c(%ebx),%eax
140c1c: 89 43 38 mov %eax,0x38(%ebx)
140c1f: 7f 5f jg 140c80 <raAdd+0xe0>
140c21: 8b 43 58 mov 0x58(%ebx),%eax
140c24: 85 c0 test %eax,%eax
140c26: 74 18 je 140c40 <raAdd+0xa0>
140c28: c7 44 24 04 08 00 00 movl $0x8,0x4(%esp)
140c2f: 00
140c30: 89 04 24 mov %eax,(%esp)
140c33: e8 fc ff ff ff call 140c34 <raAdd+0x94>
140c38: 85 c0 test %eax,%eax
140c3a: 0f 85 90 00 00 00 jne 140cd0 <raAdd+0x130>
140c40: 8b 43 64 mov 0x64(%ebx),%eax
140c43: 85 c0 test %eax,%eax
140c45: 75 29 jne 140c70 <raAdd+0xd0>
140c47: 8b 45 0c mov 0xc(%ebp),%eax
140c4a: 85 c0 test %eax,%eax
140c4c: 74 05 je 140c53 <raAdd+0xb3>
140c4e: 8b 45 0c mov 0xc(%ebp),%eax
140c51: 89 30 mov %esi,(%eax)
140c53: 83 c4 1c add $0x1c,%esp
140c56: 89 f8 mov %edi,%eax
140c58: 5b pop %ebx
140c59: 5e pop %esi
140c5a: 5f pop %edi
140c5b: 5d pop %ebp
140c5c: c3 ret
140c5d: 8d 76 00 lea 0x0(%esi),%esi
140c60: c7 43 68 00 00 00 00 movl $0x0,0x68(%ebx)
140c67: e9 53 ff ff ff jmp 140bbf <raAdd+0x1f>
140c6c: 8d 74 26 00 lea 0x0(%esi),%esi
140c70: c7 43 68 00 00 00 00 movl $0x0,0x68(%ebx)
140c77: eb ce jmp 140c47 <raAdd+0xa7>
140c79: 8d b4 26 00 00 00 00 lea 0x0(%esi),%esi
140c80: 89 43 3c mov %eax,0x3c(%ebx)
140c83: eb 9c jmp 140c21 <raAdd+0x81>
140c85: 8b 43 58 mov 0x58(%ebx),%eax
140c88: 85 c0 test %eax,%eax
140c8a: 74 18 je 140ca4 <raAdd+0x104>
140c8c: c7 44 24 04 02 00 00 movl $0x2,0x4(%esp)
140c93: 00
140c94: 89 04 24 mov %eax,(%esp)
140c97: e8 fc ff ff ff call 140c98 <raAdd+0xf8>
140c9c: 85 c0 test %eax,%eax
140c9e: 0f 85 85 00 00 00 jne 140d29 <raAdd+0x189>
140ca4: 89 5c 24 04 mov %ebx,0x4(%esp)
140ca8: c7 04 24 7c 8b 1b 00 movl $0x1b8b7c,(%esp)
140caf: e8 fc ff ff ff call 140cb0 <raAdd+0x110>
140cb4: 8b 43 28 mov 0x28(%ebx),%eax
140cb7: 85 c0 test %eax,%eax
140cb9: 0f 85 1c ff ff ff jne 140bdb <raAdd+0x3b>
140cbf: 90 nop
140cc0: 83 6b 30 01 subl $0x1,0x30(%ebx)
140cc4: e9 20 ff ff ff jmp 140be9 <raAdd+0x49>
140cc9: 8d b4 26 00 00 00 00 lea 0x0(%esi),%esi
140cd0: 89 7c 24 14 mov %edi,0x14(%esp)
140cd4: 89 74 24 10 mov %esi,0x10(%esp)
140cd8: 8b 43 38 mov 0x38(%ebx),%eax
140cdb: 89 5c 24 08 mov %ebx,0x8(%esp)
140cdf: c7 44 24 04 f0 8b 1b movl $0x1b8bf0,0x4(%esp)
140ce6: 00
140ce7: 89 44 24 0c mov %eax,0xc(%esp)
140ceb: 8b 43 58 mov 0x58(%ebx),%eax
140cee: 89 04 24 mov %eax,(%esp)
140cf1: e8 fc ff ff ff call 140cf2 <raAdd+0x152>
140cf6: 8b 43 64 mov 0x64(%ebx),%eax
140cf9: 85 c0 test %eax,%eax
140cfb: 0f 84 46 ff ff ff je 140c47 <raAdd+0xa7>
140d01: e9 6a ff ff ff jmp 140c70 <raAdd+0xd0>
140d06: 89 5c 24 04 mov %ebx,0x4(%esp)
140d0a: c7 04 24 c4 8b 1b 00 movl $0x1b8bc4,(%esp)
140d11: e8 fc ff ff ff call 140d12 <raAdd+0x172>
140d16: c7 43 28 00 00 00 00 movl $0x0,0x28(%ebx)
140d1d: c7 43 30 00 00 00 00 movl $0x0,0x30(%ebx)
140d24: e9 d2 fe ff ff jmp 140bfb <raAdd+0x5b>
140d29: 89 5c 24 08 mov %ebx,0x8(%esp)
140d2d: c7 44 24 04 34 8b 1b movl $0x1b8b34,0x4(%esp)
140d34: 00
140d35: 8b 43 58 mov 0x58(%ebx),%eax
140d38: 89 04 24 mov %eax,(%esp)
140d3b: e8 fc ff ff ff call 140d3c <raAdd+0x19c>
140d40: e9 5f ff ff ff jmp 140ca4 <raAdd+0x104>
140d45: 8b 43 58 mov 0x58(%ebx),%eax
140d48: 85 c0 test %eax,%eax
140d4a: 74 14 je 140d60 <raAdd+0x1c0>
140d4c: c7 44 24 04 02 00 00 movl $0x2,0x4(%esp)
140d53: 00
140d54: 89 04 24 mov %eax,(%esp)
140d57: e8 fc ff ff ff call 140d58 <raAdd+0x1b8>
140d5c: 85 c0 test %eax,%eax
140d5e: 75 59 jne 140db9 <raAdd+0x219>
140d60: 8b 43 34 mov 0x34(%ebx),%eax
140d63: 89 5c 24 04 mov %ebx,0x4(%esp)
140d67: c7 04 24 0c 8b 1b 00 movl $0x1b8b0c,(%esp)
140d6e: 89 44 24 08 mov %eax,0x8(%esp)
140d72: e8 fc ff ff ff call 140d73 <raAdd+0x1d3>
140d77: 8b 45 0c mov 0xc(%ebp),%eax
140d7a: 85 c0 test %eax,%eax
140d7c: 74 09 je 140d87 <raAdd+0x1e7>
140d7e: 8b 45 0c mov 0xc(%ebp),%eax
140d81: c7 00 00 00 00 00 movl $0x0,(%eax)
140d87: 8b 43 64 mov 0x64(%ebx),%eax
140d8a: bf ff ff ff ff mov $0xffffffff,%edi
140d8f: 85 c0 test %eax,%eax
140d91: 0f 84 bc fe ff ff je 140c53 <raAdd+0xb3>
140d97: c7 43 68 00 00 00 00 movl $0x0,0x68(%ebx)
140d9e: e9 b0 fe ff ff jmp 140c53 <raAdd+0xb3>
140da3: c7 04 24 56 8a 1b 00 movl $0x1b8a56,(%esp)
140daa: bf ff ff ff ff mov $0xffffffff,%edi
140daf: e8 fc ff ff ff call 140db0 <raAdd+0x210>
140db4: e9 9a fe ff ff jmp 140c53 <raAdd+0xb3>
140db9: 8b 43 34 mov 0x34(%ebx),%eax
140dbc: 89 5c 24 08 mov %ebx,0x8(%esp)
140dc0: c7 44 24 04 e4 8a 1b movl $0x1b8ae4,0x4(%esp)
140dc7: 00
140dc8: 89 44 24 0c mov %eax,0xc(%esp)
140dcc: 8b 43 58 mov 0x58(%ebx),%eax
140dcf: 89 04 24 mov %eax,(%esp)
140dd2: e8 fc ff ff ff call 140dd3 <raAdd+0x233>
140dd7: eb 87 jmp 140d60 <raAdd+0x1c0>
140dd9: 90 nop
140dda: 8d b6 00 00 00 00 lea 0x0(%esi),%esi
The source code of the crash point
***********************************
typedef struct vacantNode_tag vacantNode;
struct vacantNode_tag
{
void* unused; /* This unused pointer is here to protect the first parameter people
will put in structs inside RA even when the element is vacant.
It is used by EMA. */
vacantNode* prev; /* Pointer to the previous vacant element. NULL if this one is the first */
vacantNode* next; /* Pointer to the next vacant element. NULL if this one is the last */
int nodeIndex; /* Current node's index. This is used for faster calculation of the raAdd() function. */
};
int raAdd(IN HRA raH, OUT RAElement *pOutElem)
{
raHeader *ra = (raHeader *)raH;
RAElement allocatedElement;
int vLocation;
if ( ra == NULL )
{
printf("raAdd raH is NULL!\n");
return RV_ERROR_UNKNOWN;
}
if (ra->threadSafe)
Rv2LockGet(&ra->lock, ra->logMgr);
/* See if there's any place in this RA */
if (ra->firstVacantElement == NULL)
{
RvLogError(ra->pLog,
(ra->pLog, "raAdd (%s): Array full (%d elements)", ra->name, ra->maxNumOfElements));
Rv2Printf("raAdd (%s): Array full (%d elements)\n", ra->name, ra->maxNumOfElements);
if (pOutElem != NULL) *pOutElem = NULL;
if (ra->threadSafe)
Rv2LockRelease(&ra->lock, ra->logMgr);
return RV_ERROR_UNKNOWN;
}
allocatedElement = ra->firstVacantElement;
/* Get the element from list of vacant elements and fix that list */
ra->firstVacantElement = (RAElement)((vacantNode *)allocatedElement)->next;
if (((vacantNode *)ra->workingSetElement) == NULL)
{
RvLogError(ra->pLog, (ra->pLog, "raAdd (%s):[test][raAdd] ((vacantNode *)ra->workingSetElement)==NULL", ra->name));
Rv2Printf("raAdd (%s): [test][raAdd] ((vacantNode *)ra->workingSetElement)==NULL\n", ra->name);
}
if (((vacantNode *)ra->workingSetElement) && ((vacantNode *)ra->workingSetElement)->next != NULL)
ra->workingSetElement = (RAElement)(((vacantNode *)ra->workingSetElement)->next);
else
ra->workingSetDistance--;
/* Let's see what we have to do with our free list */
if (ra->firstVacantElement == NULL)
{
Rv2Printf("raAdd (%s): ra->firstVacantElement ==NULL\n", ra->name);
/* No more free elements - update the working set information */
ra->workingSetElement = NULL;
ra->workingSetDistance = 0;
}
else
{
/* Since we chopped off the head of the free list - we should set the new head
of the free list as the first element: i.e - prev=NULL */
((vacantNode *)ra->firstVacantElement)->prev = NULL;
}
......
return vLocation;
}
Analysis steps:
1. Disassemble the library/excutable.
objdump -D libkdv323adapter.so > libkdv323adapter.so.asm
2. Locate the crash point
Use the crash backtrace information to locate the crash point.
/opt/mcu/pas/libkdv323adapter.so(raAdd+0x3b) [0x51dbdb]
Find raAdd in the asm code, the address is 00140ba0, crash point is raAdd+0x3b, that is 00143765
Now we got the crash point which code is:
140bdb: 8b 40 08 mov 0x8(%eax),%eax
Seems it was happend when reading/assigning a value.
3. Translate the asm code to C code.
If see into the definition of struct vacantNode_tag, we can get that 0x8(%eax) means ra->workingSetElement->next, so the crash was happened in this line
if (((vacantNode *)ra->workingSetElement) && ((vacantNode *)ra->workingSetElement)->next != NULL)
ra->workingSetElement = (RAElement)(((vacantNode *)ra->workingSetElement)->next);
else
ra->workingSetDistance--;
I’ve matched and listed the C code together with the asm code in the previous asm contents.
Done!
Like what I said at the beginning, this is really not a good example to explain crash analysis by reading the asm code. Because we did not find the real reason of the crash for this example.
And if you read the C code carefully, you can find a really weired thing, how could this crash happen in such a code?
if (((vacantNode *)ra->workingSetElement) && ((vacantNode *)ra->workingSetElement)->next != NULL)
{
...
}
————————-raAdd+0x3b———————————-
140bdb: 8b 40 08 mov 0x8(%eax),%eax //((vacantNode *)ra->workingSetElement)->next
———————————————————————
140bde: 85 c0 test %eax,%eax //if (((vacantNode *)ra->workingSetElement) && ((vacantNode *)ra->workingSetElement)->next != NULL)
Allow me to explain it later.
There’s certainly a great deal to find out about this
issue. I love all the points you have made.