Trouble shooting: step by step to analysis crashes 1


This post’s goal is to guide a starter to analysis a crash by reading into the assemble code.

But the example listed here is not a good one, because the crash point is not an obvious one, the real reason of the crash for this example is still remain uncovered.

My point here is, we can use a such kind of way to analysis some crash, and once you read this post, you can start the first step. If you run into any problems when analysis your crash, well, we can discuss wih them together here. Here we go.

As you know, you can get the call stack by adding a backtrace support for your app.

You can compile your app by adding -rdynamic, when the crash happened, you can use addr2line to locate the exact line of your source code which caused the crash. http://rg4.net/archives/1656.html

But when a formal product is released, we will not compile our components with -rdynamic, then we need to analysis the source code ourself.

Here is an example:

The crash report log which contains the call stack when the crash happened.

2015-7-17 16:37:28
---------------------------------pid=31603,tid=32136<STACKAPP>-------------------
----------------SIGSEGV-------------------
SI code = 1 (Address not mapped to object)
Fault addr = 0x7
----------------stack-------------------
h323sadp() [0x806b9de]
h323sadp() [0x806bc30]
[0x2d5410]
/opt/mcu/pas/libkdv323adapter.so(raAdd+0x3b) [0x51dbdb]
/opt/mcu/pas/libkdv323adapter.so(+0x1436cf) [0x5206cf]
/opt/mcu/pas/libkdv323adapter.so(rtAddBrother+0x35) [0x520765]
/opt/mcu/pas/libkdv323adapter.so(+0x1107ef) [0x4ed7ef]
/opt/mcu/pas/libkdv323adapter.so(pvtAdd+0x43) [0x4edac3]
/opt/mcu/pas/libkdv323adapter.so(perDecodeChoice+0x2ad) [0x4bcf4d]
/opt/mcu/pas/libkdv323adapter.so(perDecNode+0x557) [0x4bb1d7]
/opt/mcu/pas/libkdv323adapter.so(perDecodeSequece+0x16d) [0x4bf2cd]
/opt/mcu/pas/libkdv323adapter.so(perDecNode+0x50f) [0x4bb18f]
/opt/mcu/pas/libkdv323adapter.so(perDecodeSequeceOF+0x1c0) [0x4bfe30]
/opt/mcu/pas/libkdv323adapter.so(perDecNode+0x4e5) [0x4bb165]
/opt/mcu/pas/libkdv323adapter.so(perDecodeSequece+0x1a1) [0x4bf301]
/opt/mcu/pas/libkdv323adapter.so(perDecNode+0x50f) [0x4bb18f]
/opt/mcu/pas/libkdv323adapter.so(perDecodeChoice+0x3b9) [0x4bd059]
/opt/mcu/pas/libkdv323adapter.so(perDecNode+0x557) [0x4bb1d7]
/opt/mcu/pas/libkdv323adapter.so(perDecodeChoice+0x318) [0x4bcfb8]
/opt/mcu/pas/libkdv323adapter.so(perDecNode+0x557) [0x4bb1d7]
/opt/mcu/pas/libkdv323adapter.so(+0xde634) [0x4bb634]
/opt/mcu/pas/libkdv323adapter.so(emDecode+0xa2) [0x4ba962]
/opt/mcu/pas/libkdv323adapter.so(cmEmDecode+0x5c) [0x4ba7cc]
/opt/mcu/pas/libkdv323adapter.so(cmEvTransNewRawMessage+0x64) [0x4a4d24]
/opt/mcu/pas/libkdv323adapter.so(decodeIncomingMessage+0x1b7) [0x5096a7]
/opt/mcu/pas/libkdv323adapter.so(transH245Handler+0x32a) [0x50affa]
/opt/mcu/pas/libkdv323adapter.so(tpktEvent+0x12a) [0x507d6a]
/opt/mcu/pas/libkdv323adapter.so(Rv2SelectHandleEPollFds+0xe6) [0x527716]
/opt/mcu/pas/libkdv323adapter.so(Rv2SelectWaitAndBlock+0x1bd) [0x527b1d]
/opt/mcu/pas/libkdv323adapter.so(seliSelectUntil+0x44) [0x4e9a84]
/opt/mcu/pas/libkdv323adapter.so(_Z15kdvCheckMessagev+0x32) [0x446462]
h323sadp() [0x80687d5]
/opt/mcu/nm/libosp.so(OspAppEntry+0x62c) [0x88648c]
/opt/mcu/nm/libosp.so(+0x28e2d) [0x894e2d]
/lib/libpthread.so.0() [0xd26b39]
/lib/libc.so.6(clone+0x5e) [0x6dfd6e]
----------------context struct----------------
ct->uc_flags = 0
ct->uc_link = 0x(nil)
ct->uc_sigmask = 00
ct->uc_stack.ss_sp = 0x(nil)
ct->uc_stack.ss_flags = 0x2
ct->uc_stack.ss_size = 0x0

Get the libkdv323adapter.so and disassemble it, here is the code we get.

00140ba0 <raAdd>:
  140ba0:    55                       push   %ebp
  140ba1:    89 e5                    mov    %esp,%ebp
  140ba3:    57                       push   %edi
  140ba4:    56                       push   %esi
  140ba5:    53                       push   %ebx
  140ba6:    83 ec 1c                 sub    $0x1c,%esp
  140ba9:    8b 5d 08                 mov    0x8(%ebp),%ebx                    //ra
  140bac:    85 db                    test   %ebx,%ebx                            //if ( ra == NULL )
  140bae:    0f 84 ef 01 00 00        je     140da3 <raAdd+0x203>
  140bb4:    8b 43 64                 mov    0x64(%ebx),%eax                //ra->threadSafe
  140bb7:    85 c0                    test   %eax,%eax                            //if (ra->threadSafe)
  140bb9:    0f 85 a1 00 00 00        jne    140c60 <raAdd+0xc0>
  140bbf:    8b 73 24                 mov    0x24(%ebx),%esi                //ra->firstVacantElement
  140bc2:    85 f6                    test   %esi,%esi                            //if (ra->firstVacantElement == NULL)
  140bc4:    0f 84 7b 01 00 00        je     140d45 <raAdd+0x1a5>
  140bca:    8b 46 08                 mov    0x8(%esi),%eax                    //find variant allocatedElement
  140bcd:    89 43 24                 mov    %eax,0x24(%ebx)                //allocatedElement = ra->firstVacantElement;
  140bd0:    8b 43 28                 mov    0x28(%ebx),%eax                //ra->firstVacantElement = (RAElement)((vacantNode *)allocatedElement)->next;
 
    140bd3:    85 c0                    test   %eax,%eax                            //if (((vacantNode *)ra->workingSetElement) == NULL)
  140bd5:    0f 84 aa 00 00 00        je     140c85 <raAdd+0xe5>
-------------------------raAdd+0x3b----------------------------------
  140bdb:    8b 40 08                 mov    0x8(%eax),%eax                    //((vacantNode *)ra->workingSetElement)->next
---------------------------------------------------------------------
  140bde:    85 c0                    test   %eax,%eax                            //if (((vacantNode *)ra->workingSetElement) && ((vacantNode *)ra->workingSetElement)->next != NULL)
  140be0:    0f 84 da 00 00 00        je     140cc0 <raAdd+0x120>
  140be6:    89 43 28                 mov    %eax,0x28(%ebx)                //ra->workingSetElement = (RAElement)(((vacantNode *)ra->workingSetElement)->next);
  140be9:    8b 43 24                 mov    0x24(%ebx),%eax                //ra->firstVacantElement
  140bec:    85 c0                    test   %eax,%eax                            //if (ra->firstVacantElement == NULL)
  140bee:    0f 84 12 01 00 00        je     140d06 <raAdd+0x166>
  140bf4:    c7 40 04 00 00 00 00     movl   $0x0,0x4(%eax)                    //((vacantNode *)ra->firstVacantElement)->prev = NULL;
  140bfb:    8b 7e 0c                 mov    0xc(%esi),%edi                    //((vacantNode *)allocatedElement)->nodeIndex;
  140bfe:    b8 80 00 00 00           mov    $0x80,%eax
  140c03:    89 fa                    mov    %edi,%edx
  140c05:    89 f9                    mov    %edi,%ecx
  140c07:    c1 ea 03                 shr    $0x3,%edx
  140c0a:    83 e1 07                 and    $0x7,%ecx
  140c0d:    d3 f8                    sar    %cl,%eax
  140c0f:    08 44 13 6c              or     %al,0x6c(%ebx,%edx,1)
  140c13:    8b 43 38                 mov    0x38(%ebx),%eax
  140c16:    83 c0 01                 add    $0x1,%eax
  140c19:    3b 43 3c                 cmp    0x3c(%ebx),%eax
  140c1c:    89 43 38                 mov    %eax,0x38(%ebx)
  140c1f:    7f 5f                    jg     140c80 <raAdd+0xe0>
  140c21:    8b 43 58                 mov    0x58(%ebx),%eax
  140c24:    85 c0                    test   %eax,%eax
  140c26:    74 18                    je     140c40 <raAdd+0xa0>
  140c28:    c7 44 24 04 08 00 00     movl   $0x8,0x4(%esp)
  140c2f:    00
  140c30:    89 04 24                 mov    %eax,(%esp)
  140c33:    e8 fc ff ff ff           call   140c34 <raAdd+0x94>
  140c38:    85 c0                    test   %eax,%eax
  140c3a:    0f 85 90 00 00 00        jne    140cd0 <raAdd+0x130>
  140c40:    8b 43 64                 mov    0x64(%ebx),%eax
  140c43:    85 c0                    test   %eax,%eax
  140c45:    75 29                    jne    140c70 <raAdd+0xd0>
  140c47:    8b 45 0c                 mov    0xc(%ebp),%eax
  140c4a:    85 c0                    test   %eax,%eax
  140c4c:    74 05                    je     140c53 <raAdd+0xb3>
  140c4e:    8b 45 0c                 mov    0xc(%ebp),%eax
  140c51:    89 30                    mov    %esi,(%eax)
  140c53:    83 c4 1c                 add    $0x1c,%esp
  140c56:    89 f8                    mov    %edi,%eax
  140c58:    5b                       pop    %ebx
  140c59:    5e                       pop    %esi
  140c5a:    5f                       pop    %edi
  140c5b:    5d                       pop    %ebp
  140c5c:    c3                       ret    
  140c5d:    8d 76 00                 lea    0x0(%esi),%esi
  140c60:    c7 43 68 00 00 00 00     movl   $0x0,0x68(%ebx)
  140c67:    e9 53 ff ff ff           jmp    140bbf <raAdd+0x1f>
  140c6c:    8d 74 26 00              lea    0x0(%esi),%esi
  140c70:    c7 43 68 00 00 00 00     movl   $0x0,0x68(%ebx)
  140c77:    eb ce                    jmp    140c47 <raAdd+0xa7>
  140c79:    8d b4 26 00 00 00 00     lea    0x0(%esi),%esi
  140c80:    89 43 3c                 mov    %eax,0x3c(%ebx)
  140c83:    eb 9c                    jmp    140c21 <raAdd+0x81>
  140c85:    8b 43 58                 mov    0x58(%ebx),%eax
  140c88:    85 c0                    test   %eax,%eax
  140c8a:    74 18                    je     140ca4 <raAdd+0x104>
  140c8c:    c7 44 24 04 02 00 00     movl   $0x2,0x4(%esp)
  140c93:    00
  140c94:    89 04 24                 mov    %eax,(%esp)
  140c97:    e8 fc ff ff ff           call   140c98 <raAdd+0xf8>
  140c9c:    85 c0                    test   %eax,%eax
  140c9e:    0f 85 85 00 00 00        jne    140d29 <raAdd+0x189>
  140ca4:    89 5c 24 04              mov    %ebx,0x4(%esp)
  140ca8:    c7 04 24 7c 8b 1b 00     movl   $0x1b8b7c,(%esp)
  140caf:    e8 fc ff ff ff           call   140cb0 <raAdd+0x110>
  140cb4:    8b 43 28                 mov    0x28(%ebx),%eax
  140cb7:    85 c0                    test   %eax,%eax
  140cb9:    0f 85 1c ff ff ff        jne    140bdb <raAdd+0x3b>
  140cbf:    90                       nop    
  140cc0:    83 6b 30 01              subl   $0x1,0x30(%ebx)
  140cc4:    e9 20 ff ff ff           jmp    140be9 <raAdd+0x49>
  140cc9:    8d b4 26 00 00 00 00     lea    0x0(%esi),%esi
  140cd0:    89 7c 24 14              mov    %edi,0x14(%esp)
  140cd4:    89 74 24 10              mov    %esi,0x10(%esp)
  140cd8:    8b 43 38                 mov    0x38(%ebx),%eax
  140cdb:    89 5c 24 08              mov    %ebx,0x8(%esp)
  140cdf:    c7 44 24 04 f0 8b 1b     movl   $0x1b8bf0,0x4(%esp)
  140ce6:    00
  140ce7:    89 44 24 0c              mov    %eax,0xc(%esp)
  140ceb:    8b 43 58                 mov    0x58(%ebx),%eax
  140cee:    89 04 24                 mov    %eax,(%esp)
  140cf1:    e8 fc ff ff ff           call   140cf2 <raAdd+0x152>
  140cf6:    8b 43 64                 mov    0x64(%ebx),%eax
  140cf9:    85 c0                    test   %eax,%eax
  140cfb:    0f 84 46 ff ff ff        je     140c47 <raAdd+0xa7>
  140d01:    e9 6a ff ff ff           jmp    140c70 <raAdd+0xd0>
  140d06:    89 5c 24 04              mov    %ebx,0x4(%esp)
  140d0a:    c7 04 24 c4 8b 1b 00     movl   $0x1b8bc4,(%esp)
  140d11:    e8 fc ff ff ff           call   140d12 <raAdd+0x172>
  140d16:    c7 43 28 00 00 00 00     movl   $0x0,0x28(%ebx)
  140d1d:    c7 43 30 00 00 00 00     movl   $0x0,0x30(%ebx)
  140d24:    e9 d2 fe ff ff           jmp    140bfb <raAdd+0x5b>
  140d29:    89 5c 24 08              mov    %ebx,0x8(%esp)
  140d2d:    c7 44 24 04 34 8b 1b     movl   $0x1b8b34,0x4(%esp)
  140d34:    00
  140d35:    8b 43 58                 mov    0x58(%ebx),%eax
  140d38:    89 04 24                 mov    %eax,(%esp)
  140d3b:    e8 fc ff ff ff           call   140d3c <raAdd+0x19c>
  140d40:    e9 5f ff ff ff           jmp    140ca4 <raAdd+0x104>
  140d45:    8b 43 58                 mov    0x58(%ebx),%eax
  140d48:    85 c0                    test   %eax,%eax
  140d4a:    74 14                    je     140d60 <raAdd+0x1c0>
  140d4c:    c7 44 24 04 02 00 00     movl   $0x2,0x4(%esp)
  140d53:    00
  140d54:    89 04 24                 mov    %eax,(%esp)
  140d57:    e8 fc ff ff ff           call   140d58 <raAdd+0x1b8>
  140d5c:    85 c0                    test   %eax,%eax
  140d5e:    75 59                    jne    140db9 <raAdd+0x219>
  140d60:    8b 43 34                 mov    0x34(%ebx),%eax
  140d63:    89 5c 24 04              mov    %ebx,0x4(%esp)
  140d67:    c7 04 24 0c 8b 1b 00     movl   $0x1b8b0c,(%esp)
  140d6e:    89 44 24 08              mov    %eax,0x8(%esp)
  140d72:    e8 fc ff ff ff           call   140d73 <raAdd+0x1d3>
  140d77:    8b 45 0c                 mov    0xc(%ebp),%eax
  140d7a:    85 c0                    test   %eax,%eax
  140d7c:    74 09                    je     140d87 <raAdd+0x1e7>
  140d7e:    8b 45 0c                 mov    0xc(%ebp),%eax
  140d81:    c7 00 00 00 00 00        movl   $0x0,(%eax)
  140d87:    8b 43 64                 mov    0x64(%ebx),%eax
  140d8a:    bf ff ff ff ff           mov    $0xffffffff,%edi
  140d8f:    85 c0                    test   %eax,%eax
  140d91:    0f 84 bc fe ff ff        je     140c53 <raAdd+0xb3>
  140d97:    c7 43 68 00 00 00 00     movl   $0x0,0x68(%ebx)
  140d9e:    e9 b0 fe ff ff           jmp    140c53 <raAdd+0xb3>
  140da3:    c7 04 24 56 8a 1b 00     movl   $0x1b8a56,(%esp)
  140daa:    bf ff ff ff ff           mov    $0xffffffff,%edi
  140daf:    e8 fc ff ff ff           call   140db0 <raAdd+0x210>
  140db4:    e9 9a fe ff ff           jmp    140c53 <raAdd+0xb3>
  140db9:    8b 43 34                 mov    0x34(%ebx),%eax
  140dbc:    89 5c 24 08              mov    %ebx,0x8(%esp)
  140dc0:    c7 44 24 04 e4 8a 1b     movl   $0x1b8ae4,0x4(%esp)
  140dc7:    00
  140dc8:    89 44 24 0c              mov    %eax,0xc(%esp)
  140dcc:    8b 43 58                 mov    0x58(%ebx),%eax
  140dcf:    89 04 24                 mov    %eax,(%esp)
  140dd2:    e8 fc ff ff ff           call   140dd3 <raAdd+0x233>
  140dd7:    eb 87                    jmp    140d60 <raAdd+0x1c0>
  140dd9:    90                       nop    
  140dda:    8d b6 00 00 00 00        lea    0x0(%esi),%esi

 

The source code of the crash point

***********************************

typedef struct vacantNode_tag vacantNode;
struct vacantNode_tag
{
    void*       unused; /* This unused pointer is here to protect the first parameter people
                           will put in structs inside RA even when the element is vacant.
                           It is used by EMA. */
    vacantNode* prev; /* Pointer to the previous vacant element. NULL if this one is the first */
    vacantNode* next; /* Pointer to the next vacant element. NULL if this one is the last */
    int         nodeIndex; /* Current node's index. This is used for faster calculation of the raAdd() function. */
};

int raAdd(IN HRA raH, OUT RAElement *pOutElem)
{    
    raHeader *ra = (raHeader *)raH;
    RAElement allocatedElement;
    int vLocation;

    if ( ra == NULL )
    {
        printf("raAdd raH is NULL!\n");
        return RV_ERROR_UNKNOWN;
    }

    if (ra->threadSafe)
        Rv2LockGet(&ra->lock, ra->logMgr);

    /* See if there's any place in this RA */
    if (ra->firstVacantElement == NULL)
    {
        RvLogError(ra->pLog,
            (ra->pLog, "raAdd (%s): Array full (%d elements)", ra->name, ra->maxNumOfElements));
        Rv2Printf("raAdd (%s): Array full (%d elements)\n", ra->name, ra->maxNumOfElements);
        if (pOutElem != NULL) *pOutElem = NULL;
        if (ra->threadSafe)
            Rv2LockRelease(&ra->lock, ra->logMgr);
        return RV_ERROR_UNKNOWN;
    }

    allocatedElement = ra->firstVacantElement;

    /* Get the element from list of vacant elements and fix that list */
    ra->firstVacantElement = (RAElement)((vacantNode *)allocatedElement)->next;

    if (((vacantNode *)ra->workingSetElement) == NULL)
    {
        RvLogError(ra->pLog, (ra->pLog, "raAdd (%s):[test][raAdd] ((vacantNode *)ra->workingSetElement)==NULL", ra->name));
        Rv2Printf("raAdd (%s): [test][raAdd] ((vacantNode *)ra->workingSetElement)==NULL\n", ra->name);
    }

    if (((vacantNode *)ra->workingSetElement) && ((vacantNode *)ra->workingSetElement)->next != NULL)
        ra->workingSetElement = (RAElement)(((vacantNode *)ra->workingSetElement)->next);
    else
        ra->workingSetDistance--;

    /* Let's see what we have to do with our free list */
    if (ra->firstVacantElement == NULL)
    {
            Rv2Printf("raAdd (%s): ra->firstVacantElement ==NULL\n", ra->name);
        /* No more free elements - update the working set information */
        ra->workingSetElement = NULL;
        ra->workingSetDistance = 0;
    }
    else
    {
        /* Since we chopped off the head of the free list - we should set the new head
           of the free list as the first element: i.e - prev=NULL */
        ((vacantNode *)ra->firstVacantElement)->prev = NULL;
    }

    ......
 
    return vLocation;
}

Analysis steps:

1. Disassemble the library/excutable.

objdump -D libkdv323adapter.so > libkdv323adapter.so.asm

2. Locate the crash point

Use the crash backtrace information to locate the crash point.

/opt/mcu/pas/libkdv323adapter.so(raAdd+0x3b) [0x51dbdb]

Find raAdd in the asm code, the address is 00140ba0, crash point is raAdd+0x3b, that is 00143765

Now we got the crash point which code is:

  140bdb:    8b 40 08                 mov    0x8(%eax),%eax

Seems it was happend when reading/assigning a value.

3. Translate the asm code to C code.

If see into the definition of struct vacantNode_tag, we can get that 0x8(%eax) means ra->workingSetElement->next, so the crash was happened in this line

    if (((vacantNode *)ra->workingSetElement) && ((vacantNode *)ra->workingSetElement)->next != NULL)
        ra->workingSetElement = (RAElement)(((vacantNode *)ra->workingSetElement)->next);
    else
        ra->workingSetDistance--;

I’ve matched and listed the C code together with the asm code in the previous asm contents.

Done!

Like what I said at the beginning, this is really not a good example to explain crash analysis by reading the asm code. Because we did not find the real reason of the crash for this example.
And if you read the C code carefully, you can find a really weired thing, how could this crash happen in such a code?

if (((vacantNode *)ra->workingSetElement) && ((vacantNode *)ra->workingSetElement)->next != NULL)
{
    ...
}

————————-raAdd+0x3b———————————-
  140bdb:    8b 40 08                 mov    0x8(%eax),%eax                    //((vacantNode *)ra->workingSetElement)->next
———————————————————————
  140bde:    85 c0                    test   %eax,%eax                            //if (((vacantNode *)ra->workingSetElement) && ((vacantNode *)ra->workingSetElement)->next != NULL)

Allow me to explain it later.


Leave a comment

Your email address will not be published. Required fields are marked *

One thought on “Trouble shooting: step by step to analysis crashes