As you know, there are LIMIT settings in Linux OS which may have something to do with some frequent-happen-issue. Such as stack size, open files, core file generation, etc.

Here are some tips about ulimit, core, and some debugging tricks relevant.

1. ulimit comamnd

——————————

To view your OS limitations, you can use command:

ulimit -a

It will show you the current OS limitations for your environment like this:

[weiguohua@localhost ~]$ ulimit -a

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
pending signals                 (-i) 1024
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 16384
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

2. Enable core file
——————————

If you wish to generate a core file when your program crashed, you can use a command like this:

ulimit -c 123456789

The number following ulimit -c is the max core file size, you can set a number as you wish.

And there are rules for core file name, they are saved in:

1) /proc/sys/kernel/core_pattern

Used to control the core file name format
Default value: core
You can change it to “/home/weiguohua/corefile/core-%e-%p-%t” which means all the core file will be saved to the folder “/home/weiguohua/corefile”, and the core file name will be “core-[program name]-[pid]-[crash-time]”.

echo /home/weiguohua/corefile/core-%e-%p-%t > /proc/sys/kernel/core_pattern

 2) /proc/sys/kernel/core_uses_pid

0: Default setting, means the core file will not suffixed with a PID.
1: Add a PID suffix to the core file, such as “core.9721”

You can change it by command:

echo 1 > /proc/sys/kernel/core_uses_pid

You can man core to reference the naming rule of core dump files.

3. Debug a core file by GDB
——————————

Debugging a core file in some way is nothing different with debugging the program in a normal way.
Assume your program name is segment.out, and the generated core file name is core.9721, you can start your debug by command:

gdb ./segment.out core.9721

Then your can do anything you want for the core file, such as bt, p, etc.

P.S. If a core dump happened, and you can not locate the function by gdb backtrace, you can:
1) Use ldd to check out the so files(dynamic link libraries) this program depends on.
2) Use nm to list the symbols of the so files, and search the addresses in the core file.
3) Locate the functions by comparing the addresses in core file and the symbol map listed by nm command.

4. Memory maps for a running program
——————————

You can view the address map & range for a running program by simply cat the maps file in the /proc/[pid] folder.
It will print out the detail memory map of all the symbols the program loaded, including the so files it depended on.

[weiguohua@localhost 9721]# pwd
/proc/9721
[weiguohua@localhost 9721]# cat maps

00110000-00113000 r-xp 00000000 fd:00 677857     /lib/libdl-2.12.so
00113000-00114000 r–p 00002000 fd:00 677857     /lib/libdl-2.12.so
00114000-00115000 rw-p 00003000 fd:00 677857     /lib/libdl-2.12.so
00115000-00127000 r-xp 00000000 fd:00 677856     /lib/libz.so.1.2.3
00127000-00128000 rw-p 00011000 fd:00 677856     /lib/libz.so.1.2.3
00128000-00256000 r-xp 00000000 fd:00 1967906    /usr/lib/mysql/libmysqlclient.so.16.0.0
00256000-0029d000 rw-p 0012e000 fd:00 1967906    /usr/lib/mysql/libmysqlclient.so.16.0.0
0029d000-002c5000 r-xp 00000000 fd:00 677847     /lib/libm-2.12.so
002c5000-002c6000 r–p 00027000 fd:00 677847     /lib/libm-2.12.so
002c6000-002c7000 rw-p 00028000 fd:00 677847     /lib/libm-2.12.so
002c7000-00439000 r-xp 00000000 fd:00 1867809    /usr/lib/libcrypto.so.1.0.0
00439000-0044d000 rw-p 00172000 fd:00 1867809    /usr/lib/libcrypto.so.1.0.0
0044d000-00450000 rw-p 00000000 00:00 0
00450000-00457000 r-xp 00000000 fd:00 677864     /lib/libcrypt-2.12.so
00457000-00458000 r–p 00007000 fd:00 677864     /lib/libcrypt-2.12.so
00458000-00459000 rw-p 00008000 fd:00 677864     /lib/libcrypt-2.12.so
00459000-00480000 rw-p 00000000 00:00 0
00480000-0054a000 r-xp 00000000 fd:00 677879     /lib/libkrb5.so.3.3
0054a000-00550000 rw-p 000ca000 fd:00 677879     /lib/libkrb5.so.3.3
00550000-00595000 r-xp 00000000 fd:00 677863     /lib/libfreebl3.so
00595000-00596000 rw-p 00045000 fd:00 677863     /lib/libfreebl3.so
00596000-0059a000 rw-p 00000000 00:00 0
0059f000-005bd000 r-xp 00000000 fd:00 677835     /lib/ld-2.12.so
005bd000-005be000 r–p 0001d000 fd:00 677835     /lib/ld-2.12.so
005be000-005bf000 rw-p 0001e000 fd:00 677835     /lib/ld-2.12.so
005bf000-005c8000 r-xp 00000000 fd:00 677876     /lib/libkrb5support.so.0.1
005c8000-005c9000 rw-p 00008000 fd:00 677876     /lib/libkrb5support.so.0.1
005c9000-005e6000 r-xp 00000000 fd:00 677860     /lib/libselinux.so.1
005e6000-005e7000 r–p 0001c000 fd:00 677860     /lib/libselinux.so.1
005e7000-005e8000 rw-p 0001d000 fd:00 677860     /lib/libselinux.so.1
005f2000-00628000 r-xp 00000000 fd:00 1967263    /usr/libexec/postfix/pickup
00629000-0062a000 r–p 00036000 fd:00 1967263    /usr/libexec/postfix/pickup
0062a000-0062b000 rw-p 00037000 fd:00 1967263    /usr/libexec/postfix/pickup
0062b000-0062c000 rw-p 00000000 00:00 0
00634000-00678000 r-xp 00000000 fd:00 1867822    /usr/lib/libldap-2.4.so.2.5.2
00678000-00679000 rw-p 00044000 fd:00 1867822    /usr/lib/libldap-2.4.so.2.5.2
00679000-00685000 r-xp 00000000 fd:00 655000     /lib/libnss_files-2.12.so
00685000-00686000 r–p 0000b000 fd:00 655000     /lib/libnss_files-2.12.so
00686000-00687000 rw-p 0000c000 fd:00 655000     /lib/libnss_files-2.12.so
006c1000-006e2000 rw-p 00000000 00:00 0          [heap]
006f9000-0086c000 r-xp 00000000 fd:00 655098     /lib/libdb-4.7.so
0086c000-0086f000 rw-p 00173000 fd:00 655098     /lib/libdb-4.7.so
00888000-008a0000 r-xp 00000000 fd:00 1867808    /usr/lib/libsasl2.so.2.0.23
008a0000-008a1000 rw-p 00018000 fd:00 1867808    /usr/lib/libsasl2.so.2.0.23
009ff000-00a02000 r-xp 00000000 fd:00 677878     /lib/libcom_err.so.2.1
00a02000-00a03000 rw-p 00002000 fd:00 677878     /lib/libcom_err.so.2.1
00b40000-00b6f000 r-xp 00000000 fd:00 655464     /lib/libpcre.so.0.0.1
00b6f000-00b70000 rw-p 0002e000 fd:00 655464     /lib/libpcre.so.0.0.1
00bb5000-00bb7000 r-xp 00000000 fd:00 677875     /lib/libkeyutils.so.1.3
00bb7000-00bb8000 rw-p 00001000 fd:00 677875     /lib/libkeyutils.so.1.3
00c30000-00c56000 r-xp 00000000 fd:00 677877     /lib/libk5crypto.so.3.1
00c56000-00c57000 rw-p 00026000 fd:00 677877     /lib/libk5crypto.so.3.1
00c71000-00c72000 r-xp 00000000 00:00 0          [vdso]
00cac000-00ce2000 r-xp 00000000 fd:00 677880     /lib/libgssapi_krb5.so.2.2
00ce2000-00ce3000 rw-p 00036000 fd:00 677880     /lib/libgssapi_krb5.so.2.2
00cfb000-00d12000 r-xp 00000000 fd:00 677844     /lib/libpthread-2.12.so
00d12000-00d13000 r–p 00016000 fd:00 677844     /lib/libpthread-2.12.so
00d13000-00d14000 rw-p 00017000 fd:00 677844     /lib/libpthread-2.12.so
00d14000-00d16000 rw-p 00000000 00:00 0
00dab000-00dc1000 r-xp 00000000 fd:00 655424     /lib/libnsl-2.12.so
00dc1000-00dc2000 r–p 00016000 fd:00 655424     /lib/libnsl-2.12.so
00dc2000-00dc3000 rw-p 00017000 fd:00 655424     /lib/libnsl-2.12.so
00dc3000-00dc5000 rw-p 00000000 00:00 0
00e4f000-00ea2000 r-xp 00000000 fd:00 1867810    /usr/lib/libssl.so.1.0.0
00ea2000-00ea6000 rw-p 00052000 fd:00 1867810    /usr/lib/libssl.so.1.0.0
00efc000-00f09000 r-xp 00000000 fd:00 1843537    /usr/lib/liblber-2.4.so.2.5.2
00f09000-00f0a000 rw-p 0000c000 fd:00 1843537    /usr/lib/liblber-2.4.so.2.5.2
00f3b000-00f50000 r-xp 00000000 fd:00 677859     /lib/libresolv-2.12.so
00f50000-00f51000 r–p 00014000 fd:00 677859     /lib/libresolv-2.12.so
00f51000-00f52000 rw-p 00015000 fd:00 677859     /lib/libresolv-2.12.so
00f52000-00f54000 rw-p 00000000 00:00 0
b762c000-b7630000 rw-p 00000000 00:00 0
b7630000-b77b5000 r-xp 00000000 fd:00 677841     /lib/libc-2.12.so
b77b5000-b77b6000 —p 00185000 fd:00 677841     /lib/libc-2.12.so
b77b6000-b77b8000 r–p 00185000 fd:00 677841     /lib/libc-2.12.so
b77b8000-b77b9000 rw-p 00187000 fd:00 677841     /lib/libc-2.12.so
b77b9000-b77be000 rw-p 00000000 00:00 0
b77d3000-b77d4000 rw-p 00000000 00:00 0
bfdde000-bfdf3000 rw-p 00000000 00:00 0          [stack]