Saturday, October 26, 2013

QtCreator's Python Debug Visualizers

Peter Lohrmann wrote QtCreator debug visualizers in Python for some key classes used by the Linux OpenGL debugger project we've been working on together. He recently blogged about the details here.

So far we've got visualizers for our dynamic_string and vector classes. (Like many/most game devs, we use our own custom containers to minimize our reliance on the C++ runtime and "standard" libraries, but that's another story.) Before, to visualize the contents of vectors in QtCreator, I've had to muck around in the mud with the watch window and type in the object's name, followed by the pointer and the # of elements to view. Our dynamic_string class uses the small string optimization (not the super optimized version that Thatcher describes here, just something basic to get the job done). So it's been a huge pain to visualize strings, or basically anything in the watch/locals window.

The below pic shows the new debug visualizers in action on a vector of vectors containing dynamic_strings.  Holy shit, it just works!

I'm not a big fan of Python, but this is valuable and cool enough to make it worth my while to learn it.


Here's the code. Almost all of this is Peter's work, I've just tweaked the vector dumper to fix some things. I'm a total Python newbie so it's possible I screwed something up here, but this is working much better than I expected already. It's amazing how something simple like this on Linux can make me so happy.

You can find a bunch of QtCreator's debug visualizer code here: ~/qtcreator-2.8.0/share/qtcreator/dumper

In my ~/.gdbinit file:

python
execfile('/home/richg/dev/raddebugger/src/crnlib/gdb-dumpers.py')
end

And here's my /home/richg/dev/raddebugger/src/crnlib/gdb-dumpers.py file:

#!/usr/bin/python

# This file contains debug dumpers / helpers / visualizers so that certain crnlib

# classes can be more easily inspected by gdb and QtCreator.

def qdump__crnlib__dynamic_string(d, value):

    dyn = value["m_dyn"]
    small = value["m_small"]
    len = value["m_len"]
    small_flag = small["m_flag"]
    d.putAddress(value.address)
    buf = dyn["m_pStr"]
    if small_flag == 1:
        buf = small["m_buf"]
    p = buf.cast(lookupType("unsigned char").pointer())
    strPrefix = "[%d] " % int(len)
    str = "'" + p.string(length=len) + "'"
    d.putValue(strPrefix + str)
    d.putNumChild(3)
    with Children(d):
        d.putSubItem("m_len", len)
        with SubItem(d, "m_small"):
            d.putValue( str if small_flag == 1 else "<ignored>")
            d.putNumChild(2)
            with Children(d):
                d.putSubItem("m_flag", small_flag)
                with SubItem(d, "m_buf"):
                    d.putValue(str if small_flag == 1 else "<ignored>")
        with SubItem(d, "m_dyn"):
            d.putValue("<ignored>" if small_flag == 1 else str)
            d.putNumChild(2)
            with Children(d):
                with SubItem(d, "m_buf_size"):
                    d.putValue("<ignored>" if small_flag == 1 else dyn["m_buf_size"])
                with SubItem(d, "m_pStr"):
                    d.putValue("<ignored>" if small_flag == 1 else str)

def qdump__crnlib__vector(d, value):

    size = value["m_size"]
    capacity = value["m_capacity"]
    data = value["m_p"]
    maxDisplayItems = 100
    innerType = d.templateArgument(value.type, 0)
    p = gdb.Value(data.cast(innerType.pointer()))
    d.putValue( 'Size: {} Capacity: {} Data: {}'.format(size, capacity, data ) )
    d.putNumChild(size)
    numDisplayItems = min(maxDisplayItems, size)
    if d.isExpanded():
         with Children(d, size, maxNumChild=numDisplayItems, childType=innerType, addrBase=p, addrStep=p.dereference().__sizeof__):
             for i in range(0,numDisplayItems):
                 d.putSubItem(i, p.dereference())
                 p += 1

Saturday, October 19, 2013

A Shout-Out to QtCreator 2.8.x on Linux

So this is a little post about C/C++ IDE's, which apart from the browser is the key piece of software I live in most of the day. I know a lot of Windows-centric developers who swear by Visual Studio, and up until recently I used to be one of the VS faithful. I'm going to try and sell you on trying something else, especially if you develop on Linux or OSX but it's available for Windows too.

I think I've finally found a reasonable cross platform VS alternative for C/C++ development that doesn't require shelling out hundreds (or thousands) of dollars every time MS tweaks (or totally screws up) the UI or adds some compiler options. I've been using QtCreator full-time now for 6 months and I think it's awesome. I would buy it in a heartbeat, but it's a free download and it's even open source.

A bit of the background behind my need for a VS alternative: For more than a decade I've been using Visual Studio (since VC5 I think), and various other IDE's from Borland/Watcom/MS before that. When I started working on a new Linux OpenGL debugger (about 6 months ago) all the Linux devs around me where using text editors, cgdb, etc. There was no way in hell I was going back to only a text editor (even the goodness that is Sublime) for editing, gdb cmd line for debugging, and another command line for compiling, etc. It's been a long time since my DOS development days and I'm just too old to do that again on the PC. (On embedded platforms I can tolerate crappy or no IDE's, but not on a full-blown modern desktop!) So I began an exhaustive, and somewhat desperate search for a real Linux IDE with a useful debugger that doesn't suck.

I experimented with a bunch of packages (such as CodeBlocks, CodeLite, Eclipse, KDevelop, etc.) and even some stand-alone debuggers (like ddd, cgdb) on some of my open source projects and settled on the amazing QtCreator 2.8.x. It's a full blown C/C++ IDE with surprisingly few rough edges. It's got all the usual stuff you would expect: editor, project manager (with optional support for things like cmake), integrated source control, an Intellisense-equivalent that just works and doesn't randomly slow the IDE to a crawl like in VS, C/C++ refactoring, and nice gdb/lldb frontends that don't require you to know anything about obscure gdb commands. I've been using it to compile with either clang v3.3 (using Mike Sartain's instructions that make it trivial to switch between clang vs. gcc), and with gcc v4.6. The whole product is super polished, and I find myself happier using it than VS and its fleet of unreliable (but pretty much necessary on real projects) 3rd party plugins like Visual Assist, Incredibuild, etc. that make the whole thing a buggy and unstable mess.

QtCreator's name can be misleading. It's not just for Qt stuff, although it's obviously designed to be great for Qt dev/debugging too. I use it to debug command line and OpenGL apps, either starting them from within QtCreator or attaching to the process remotely. It's got built-in support for Mercurial (hg), Git, Perforce, SVN, etc. although I've only used its hg and p4 support.


Visual Studio since 2012 has apparently gone almost completely batshit, so I've been delaying upgrading for as long as possible even before my VS divorce. I was hoping the saner and more tasteful hands at MS would reign in the "modern app" idiocy and fix things, but I've lost hope. Although with Ballmer (who's obviously been completely out of touch) being finally put to pasture maybe they can turn the ship around.

Here are a few more screenshots of QtCreator Linux in action. I'm using KDE Plasma desktop installed under Ubuntu v12.04 x64. (If you've just installed Ubuntu for the first time and have no Linux desktop preferences yet, do yourself a favor and just go reinstall Kubuntu.) If you want to try it out, be sure to download the version from Qt's website (not the Ubuntu software center - it's really outdated the last time I checked). Also check if your distro requires disabling ptrace hardening before you debug anything. Also, I had to change the default terminal used for running/debugging apps to something else, so under Tools->Environment->Terminal: "/usr/bin/xterm -sl 1999999 -fg white -bg black -geometry 200x60 -e"

We've also just added custom debug visualizers for our most important container classes to QtCreator, but I've not had a chance to play with this stuff yet.

Source control configuration:


Debugging:


More debugging:



Sunday, October 13, 2013

The big miniz zip64 merge

Currently merging miniz v1.15 into my in-progress zip64 branch which is being used/tested in the Linux OpenGL debugger engine project I've been working on. The left pane is v1.15, right is the new version with zip64.


Fun times. The new version still needs to be C-ified in a few places. I'm actually liking the purity of C for some strange reason, it's amazing how fast it compiles vs. C++. I'm so used to glacially slow compiles that when I used TCC (Tiny C Compiler) again it appeared to complete so fast that I thought it had silently crashed.

miniz.c: Finally added zip64 support, other fixes/improvements incoming

I finally needed Zip64 support for the Linux OpenGL debugger I've been working on at work. GL traces and state snapshots can be huge (~4-10GB of data for 3-4 minute game runs is not uncommon), and can consist of tons of binary "blob" files for VB's/IB's/shaders/etc. I looked at some other C archive libraries (libzip, minizip, etc.) but they where either ugly/huge messes with a zillion C/H files, or they didn't fully abstract their file I/O, or they didn't support in-memory archives for both reading/writing, or their licenses sucked, or they weren't thread safe (!), or I just didn't trust them, etc.

So screw it, I'll bite the bullet and do this myself. It's certainly possible I missed a really good library out there. I prefer C for this kind of stuff because most C++ libs I find in the wild use features I can't live with for various reasons, such as C++ exceptions, stl, heavy use of heap memory, Boost, or have tons of other lib dependencies, etc.

The original ancient zip file format was OK and kinda elegant for what is was, and the code to parse and write the original headers was nice and easy. But once you add zip64 it becomes an ugly mess full of conditionals, and copying zip header/archive data from one zip to another can be a big pain because you can't just blindly copy the zip64 extended data fields from the source to destination zip. (You've got to kill the old one from the extended data block and add a new one, etc.) Zip64 is now "done", and I've been running a bunch of automated testing on the new code paths, but I'm worried I've bit off more than I can chew given the very limited time I have to work on this feature for the debugger.

I've also renamed a lot of the zip "reader" API's so they can be used in both reading and writing mode. There's no reason why you can't locate files in the central directory, or get file stats while writing, for example, because the entire zip central directory is kept in memory during writing.

I've added full error codes to all zip archive handling functions. You can get the last error, clear the last error, peek at the last error, etc. I went through the entire thing and made sure the errors are set appropriately, so now you can get more info than just MZ_FALSE when something goes wrong. This change alone took several hours.

In zip64 mode I only support a total of UINT_MAX (2^32-1) files in the central dir, and central dirs are limited to a total of UINT_MAX bytes. These are huge increases from the previous limits, so this should be fine. I'm not writing a hard disk backup utility here after all, so I'm not going to support archives that big right now.

Bugwise, the only major bug I'm worried about in the current public release (miniz.c v1.14) that really worries me is the MZ_ZIP_FLAG_DO_NOT_SORT_CENTRAL_DIRECTORY flag (Issue #11 on the Google Code Issue tracker). I doubt anybody really used this flag, so I'm not worried about that, but a few internal API's used it to speed up loading a little and they could fail. It's bad enough that I'm going to patch v1.14 to fix this tonight.

Also, it's definitely time to split up miniz.c into at least two source files. One for Deflate/Inflate/zlib API emulation/crc32/etc. The other file will be for ZIP archive handling and be (of course) optional.

I'll also try to merge in all the fixes/improvements people have either placed on github, on miniz's Google code bug tracker, or have sent to me privately, if time permits.

Let me know if you are dying to try this version out and I can send you a private copy for testing.