Wednesday, November 6, 2013

C++ on Linux

Prerequesits

# to build
yum -y install gcc-c++

# to remote debug (needed on run-time box)
yum -y install gdb-gdbserver

# to debug
yum -y install gdb

Compile

g++ hello.cpp -o hello.exe
./hello.exe

Core dump, Segmentation Fault, Segfault

echo "0" > /proc/sys/kernel/core_uses_pid 
ulimit -c unlimited
/tmp/crasher.exe
cd /tmp
gdb crasher.exe core
pstack core

Remote Debug

On the run-time box you use gdbserver and then from the source-code box you use gdb and connect to the gdbserver. The gdbserver can either launch a binary or attach to an existing binary, in which case the existing binary is suspended until gdb connects.

In any case you must specify the name of the gdb machine and the port on which to listen for it.

# launch binary (port is arbitrary)
gdbserver sourceCodeBox:2345 yourBinary
 Process yourBinary created; pid = 26991
 Listening on port 2345

# get the pid of an existing binary
ps -ef | grep [y]ourBinary | awk '{ print $2 }'
 18883

# attach to existing binary via pid from above
gdbserver --attach sourceCodeBox:2345 18883
 Attached; pid = 18883
 Listening on port 2345

Warning: if it doesn't say "Listening" after your gdbserver command then you won't be able to connect with gdb.

Once the gdbserver is running we will launch gdb on the source-code box and connect.

# open your binary with gdb
cd /your/source/output/folder
gdb yourBinary

# tell gdb where your source is located
dir /your/source/folder/one:/your/source/folder/two

# connect to the gdbserver
target remote runTimeBox:2345

# set some breakpoints
break ClassName::functionName

# allow execution to proceed
continue

# when you're finished
quit

If the gdb and gdbserver versions don't match, you're in for errors like the following.

Malformed packet(b) (missing colon): re:0;
Packet: 'T0505:587ed8bf;04:7c7cd8bf;08:24546500;thread:5d06;core:0;'
readchar: Got EOF
Remote side has terminated connection.

gdb is available on gnu.org and linuxtopia provides some good build/install instructions.

The first thing you need is to determine what versions you have.

gdbserver --version

GNU gdbserver (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
gdbserver is free software, covered by the GNU General Public License.
This gdbserver was configured as "i686-redhat-linux-gnu"

gdb --version

GNU gdb Fedora (6.8-37.el5)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu".

I believe what we that both the version and configuration must match. Following the instructions from linuxtopia, I did the following on the sourceCodeBox.

su - root
cd /opt
wget http://ftp.gnu.org/gnu/gdb/gdb-7.6.tar.gz
tar -xf gdb-7.6.tar.gz
cd /opt/gdb-7.6/
./configure i686-redhat-linux-gnu
make
/opt/gdb-7.6/gdb/gdb --version
GNU gdb (GDB) 7.2
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.

That solved the Malformed packet failure but didn't successfully connect. Now I have Connection timed out. That's because my gdbserver was Attached but not Listenting which, come to think of it, was probably the root of my Malformed packet failure.

We can get extra info about the session with the --debug option. Here's an example of a successful session.

gdbserver --debug --attach sourceCodeBox:2345 27465
 my_waitpid (27470, 0x0)
 my_waitpid (27470, 0x0): status(137f), 27470
 my_waitpid (27470, 0x0)
 my_waitpid (27470, 0x0): status(1057f), 27470
 my_waitpid (27471, 0x0)
 my_waitpid (27471, 0x0): status(137f), 27471
 my_waitpid (27471, 0x0)
 my_waitpid (27471, 0x0): status(9), 27471
 my_waitpid (27470, 0x0)
 my_waitpid (27470, 0x0): status(117f), 27470
 my_waitpid (27470, 0x0)
 my_waitpid (27470, 0x0): status(9), 27470
 Attached; pid = 27465
 linux_wait: [Process 27465]
 linux_wait_for_lwp: <all threads>
 my_waitpid (-1, 0x40000000)
 blocking
 sigchld_handler
 my_waitpid (-1, 0x1): status(137f), 27465
 Got an event from 27465 (137f)
 pc is 0x63e424
 stop pc is 0x63e424
 linux_wait_for_lwp: pc is 0x63e424
 Expected stop.
 Hit a non-gdbserver trap event.
 wait_for_sigstop: LWP 27465 already stopped
 Checking whether LWP 27465 needs to move out of the jump pad...no
 linux_wait ret = LWP 27465.27465, 1, 0
 Listening on port 2345

netstat -anp | grep 2345
 tcp        0      0 0.0.0.0:2345                0.0.0.0:*                   LISTEN      27469/gdbserver

Here's an example of a session that fails to listen.

gdbserver --debug --attach sourceCodeBox:2345 23814
 my_waitpid (27590, 0x0)
 my_waitpid (27590, 0x0): status(137f), 27590
 my_waitpid (27590, 0x0)
 my_waitpid (27590, 0x0): status(1057f), 27590
 my_waitpid (27591, 0x0)
 my_waitpid (27591, 0x0): status(137f), 27591
 my_waitpid (27591, 0x0)
 my_waitpid (27591, 0x0): status(9), 27591
 my_waitpid (27590, 0x0)
 my_waitpid (27590, 0x0): status(117f), 27590
 my_waitpid (27590, 0x0)
 my_waitpid (27590, 0x0): status(9), 27590
 Attached; pid = 23814
 linux_wait: [Process 23814]
 linux_wait_for_lwp: <all threads>
 my_waitpid (-1, 0x40000000)
 blocking

netstat -anp | grep 2345
 -- no result

netstat -anp | grep gdb
 -- no result

So it looks like gdbserver is unable to successfully attach to my problem program because it asks the threads of my program to suspend and waits on that forever. In my case, I gave up and figured out how to launch my binary from gdbserver instead of attaching to it.

gdb Quick Reference

[enter]    repeat the last command
c          continue to next breakpoint
n          execute next command
l          show (list) source near current position
s          step into the next line

info breakpoints    show you yoru breakpoints
bt                  show backtrace
frame 0             specify the active frame for commands like list
ptype StrInfoType   see a variable's type
print numargs       see a variable's value
set numargs = 2     changes a variables' value

Useful info commands

info address
info breakpoints
info frame
info functions
info line
info program
info sharedlibrary
info signals
info source 
info sources
info stack
info symbol
info threads

Debug into a Shared Library

A bunch of posts suggested that using gdb to step into a shared library (.so) would just work, or that you just needed to set LD_LIBRARY_PATH, but none of that worked for me. The info command showed that no symbols were loaded for my .so, and setting the solib-search-path did fix that, but I'm still unable to set breakpoints or see source. I had to additionally force it to load the symbols. This is visible in the info colum "Syms Read"

info sharedlibrary
show solib-search-path
set solib-search-path /your/path
sharedlibrary yourLib.so

For example, this was my original info.

From        To          Syms Read   Shared Object Library
                        No          /your/path/yourLib.so

After specifying the path with solib-search-path, it added the address range.

From        To          Syms Read   Shared Object Library
0x001313e0  0x00173ed4  No          /your/path/yourLib.so

After forcing a load with sharedlibrary, it toggled the syms read.

From        To          Syms Read   Shared Object Library
0x001313e0  0x00173ed4  Yes         /your/path/yourLib.so
{ "loggedin": false, "owner": false, "avatar": "", "render": "nothing", "trackingID": "UA-36983794-1", "description": "", "page": { "blogIds": [ 385 ] }, "domain": "holtstrom.com", "base": "\/michael", "url": "https:\/\/holtstrom.com\/michael\/", "frameworkFiles": "https:\/\/holtstrom.com\/michael\/_framework\/_files.4\/", "commonFiles": "https:\/\/holtstrom.com\/michael\/_common\/_files.3\/", "mediaFiles": "https:\/\/holtstrom.com\/michael\/media\/_files.3\/", "tmdbUrl": "http:\/\/www.themoviedb.org\/", "tmdbPoster": "http:\/\/image.tmdb.org\/t\/p\/w342" }