Debugging Voxel Globe (Developer)

Debugging

There are many moving pieces in Voxel Globe to debug, such as

• The client
• The web server
• The Job Queue (known as Rabbit MQ)
• The database
• Each individual processing job
• And more...

How to get a debugging into each stage will be covered in the following sections

Debugging Javascript on the Client

See firebug/firefox build in Development tools. Sometimes using Chrome is also useful

Debugging CSS on the client

TODO

Debugging the webserver

TODO - VIP_DJANGO_DEBUG, VIP_DJANGO_TEMPLATE_DEBUG, VIP_HTTPD_DEBUG_INDEXES

Debugging the job queue

TODO - Flower, rabbitmq http interface, and basic CLI commands, and maybe python amqp tricks?

Debugging the database

TODO - Django admin portal and pgadmin3

Debugging celery tasks

Debugging a celery task can be hard at first. Some tools have been included to ease this process.

vsi.tools.vdb - Debugging in Python

Some VSI Debugging (vdb) tools have been written to get a debugger into the actual processing task. The typical debugging approach won't work for a number of reasons

1. You can not attach a debugging to a python process and wait, as celery tasks will start up new python instances on demand. It's fairly difficult to attach to an already running python session too.
2. pdb.set_trace() is only useful when you have direct stdin/stdout access. A celery task is buried deep in the celery worker daemon and is piped to log files
3. Calling tasks synchronously (celery.task.apply) becomes the only way left to call a task, and involves duplicating what voxel globe already does for you each time.

Two different debuggers are wrapped in vsi.tools.vdb to give you debugging access to a python program over the network.

1. rpdb - The rpdb library is wrapped in vsi.tools.vdb_rpdb to give your telnet access to the original pdb library. The default is to server on port 4444. It's a bit crude, but works for simple debugging
2. rpdb2 - The rpdb2 library is wrapped in vsi.tools.vdb_rpdb2 to give you network access to a debugger simliar to pdb, but is actually running in pythons built in profiler instead. This has some speed benefits and can also handle multiple threads and can follow (one process only, but you get to pick with process) multi process forking. You can debug the program using either the rpdb2 command line tool or the winpdb GUI tool.

Example

import vsi.tools.vdb_rpdb2 as vdb
vdb.set_trace(_rpdb2_pwd="synsepalum dulcificum")

Start up winpdb, using the same password, and attach to your program where the set_trace left off.

---

To automatically break on the start of a task, set the environment variable VIP_CELERY_DBSTOP_ON_START to a regular expression for the task name (.* for all tasks).

Sometime set_trace isn't convenient. Another useful trick is vsi.tools.vdb_rpdb.dbstop_if_error/vsi.tools.vdb_rpdb2.dbstop_if_error which will automatically attach a post mortem debugger after an uncaught python exception. This is useful for only entering the debugger for failed tasks (albeit after the program has crashed and can no longer be recovered).

Celery has an on_failure method that is called when ANY celery task raises an uncaught exception. To automatically trigger a rpdb post mortem debugger on failure, set the environment variable VIP_CELERY_DBSTOP_IF_ERROR to 1. This is typically set in the local_vip.env file. Setting to 1 will start a pm debugger on any failed task, without having to change any code. This is good for when you aren't expecting any crashes, but are ready to debug them. You shouldn't leave this on all the time, as failed tasks will pile up waiting to be debugged and never end.

Running a task locally

It is possible to run a task in a python session, using the apply function to run a task synchronously. A task function should never be run directly! All voxel globe tasks are meant to be bound to a celery task and running it unbound (directly) will result in error. The /apps/task/status/{taskid} endpoint can be used to view status on synchronous task calls.

Example

#Normally handled by celery-django
import django
django.setup()

#Normally run by celery.py
import boxm2_register
boxm2_register.smart_register = True

#Create a service object for the database
import voxel_globe.meta.models as models
si = models.ServiceInstance(serviceName='debug')
si.save()

#Setup logging you you see all the logging messages
import celery.utils.log
celery.utils.log.base_logger.setLevel(celery.utils.log.logging.DEBUG)

t = tasks.generate_error_point_cloud.apply(args=(voxel_world_id, threshold, None), task_id=str(si.id))

Debugging C++ (Currently Notebook/Celery workers only)

It's a little harder to get gdb attached, but there are a number of methods

Method 1

1. Add export VIP_GDBSERVER=1 to your local_vip.env file
2. Add a python set_trace (See above). This is optional, just to determine the PID and pause the python before the C++ starts to set a breakpoint. If using notebook, this is less necessary
3. Wait for python breakpoint to pause, determine pid of process (either winpdb attach tells you, or use os.getpid())
4. Run program, wait for it to pause, and run ./just gdbserver, choose the container from the list, then the pid# from the list
5. Connect your favorite gdb debugger to connect to remote gdb, using localhost:{port#} printed

Method 2

Less Preferred

1. Modify C. Add

    #include <signals.h> //At top of file
   
    printf("%d\n", getpid());
    raise(SIGSTOP); // Where you want the debugger to attach
   

   Or c++

    #include <csignal> //At top of file
   
    cout << getpid() << endl; 
    raise(SIGSTOP); // Where you want the debugger to attach
   

2. Run ./just vxl restart celery

3. Run program, wait for it to pause, and run ./just gdbserver, choose the container from the list, then the pid# from the list

4. Connect your favorite gdb debugger to connect to remote gdb, using localhost:{port#} printed

Attaching a debugger

For example Eclipse, because it's the only one I know how to

1. Start Eclipse (with CDT installed)
2. Run-> Debug Configuration...
3. Create New, under debugger Tab, open connection sub-tab, and enter localhost and port number from previous section
4. Under Source Tab, Click Add... File System Directory (or similar), enter the external directory of voxel_globe (not the vxl_src dir)
5. Apply, Debug