Become a GDB Power User

If you’re writing C++ for anything other than Windows, chances are that you occasionally break out GDB. This session presents some of the lesser known features of GDB that can change the way you debug. GDB has come a long way in the last few years and does so much more than break, print, step and continue. Reversible debugging; Non-Stop Mode; Multi-process Debugging; and Dynamic Printf are but some of its best features, and its built-in Python scripting is particularly powerful. Join Undo Software co-founder and CEO, Greg Law, as he takes you through a series of demos to show some amazing tricks with GDB, and powerful new (and not-so-new) features that you may not have heard of.

Learning C with gdb

“Coming from a background in higher-level languages like Ruby, Scheme, or Haskell, learning C can be challenging. In addition to having to wrestle with C’s lower-level features like manual memory management and pointers, you have to make do without a REPL. Once you get used to exploratory programming in a REPL, having to deal with the write-compile-run loop is a bit of a bummer.

It occurred to me recently that I could use gdb as a pseudo-REPL for C. I’ve been experimenting with using gdb as a tool for learning C, rather than merely debugging C, and it’s a lot of fun.

My goal in this post is to show you that gdb is a great tool for learning C. I’ll introduce you to a few of my favorite gdb commands, and then I’ll demonstrate how you can use gdb to understand a notoriously tricky part of C: the difference between arrays and pointers…”

Recovering Redis data with GDB

“We recently had an incident where we had a Redis instance blocked on writing to disk, hung inside the kernel.

Through a variety of other circumstances this meant the only up to date copy of business critical data was in memory on a single machine, multiple independent replicas and backups were unavailable, and that machine could crash at any moment.

The problem was eventually solved without data loss of any kind due to our Techops team being utterly brilliant, but during the “incident” a number of ideas that would otherwise be dismissed as absolutely crazy were discussed.

One of the ideas was based on the simple idea that redis was an in-memory database. We had access to the machine. And the repeated question came up of just copying all the memory off of the machine, or at least out of the process, and wrest our data from it’s grasp.

Although mostly not serious, at least to start with, as options lessened the idea of taking a core-dump and picking it apart looked more and more like a straw worth clutching at.

It ended up not being needed, which is a good thing, but I figured out that it was at least possible, if you are really really really desperate, and have some time on your hands. On the off-chance anyone is really that desperate in the future, I thought I’d at least list out what direction you would need to take, and wish you the best of luck.

(Past this point I’m going to assume an exceptional knowledge of C, a good knowledge of GDB, ELF loading, symbol lookup, the amd64 ABI used by Linux, memory layout and generally how the dynamic linker works. If interested check out the wiki, especially on linking, ELF loaders etc and – Or just fake it and read on)

Given you’re at point of desperation, you’re probably going to be limited in either time or what you can do.  At the very least you are going to want to get a core dump of the redis process, taking into account that it’s going to have to write to somewhere and if you can’t just do a redis SAVE, then there is a chance that you can’t write to disk.

If you have to write to /tmp using tmpfs, realise that you might run out of memory at some point when writing it, and although the OOM killer probably won’t kick in, be very very sure first.  You might be able to use the remote gdb stub/server to dump out the core over the network rather than locally in a pinch…”