Skip to content

Debugging Voxel Globe (Developer)

Jump to bottom
Andy Neff edited this page Mar 15, 2017 · 33 revisions

There are many moving pieces in Voxel Globe to debug, such as

  • The client
  • The web server
  • The Job Queue (known as Rabbit MQ)
  • The database
  • Each individual processing job
  • And more...

How to get a debugging into each stage will be covered in the following sections

Debugging Javascript on the Client

See firebug/firefox build in Development tools. Sometimes using Chrome is also useful

Debugging CSS on the client

TODO

Debugging the webserver

TODO - VIP_DJANGO_DEBUG, VIP_DJANGO_TEMPLATE_DEBUG, VIP_HTTPD_DEBUG_INDEXES

Debugging the job queue

TODO - Flower, rabbitmq http interface, and basic CLI commands, and maybe python amqp tricks?

Debugging the database

TODO - Django admin portal and pgadmin3

Debugging celery tasks

Debugging a celery task can be hard at first. Some tools have been included to ease this process.

vsi.tools.vdb - Debugging in Python

Some VSI Debugging (vdb) tools have been written to get a debugger into the actual processing task. The typical debugging approach won't work for a number of reasons

  1. You can not attach a debugging to a python process and wait, as celery tasks will start up new python instances on demand. It's fairly difficult to attach to an already running python session too.
  2. pdb.set_trace() is only useful when you have direct stdin/stdout access. A celery task is buried deep in the celery worker daemon and is piped to log files
  3. Calling tasks synchronously (celery.task.apply) becomes the only way left to call a task, and involves duplicating what voxel globe already does for you each time.

Two different debuggers are wrapped in vsi.tools.vdb to give you debugging access to a python program over the network.

  1. rpdb - The rpdb library is wrapped in vsi.tools.vdb_rpdb to give your telnet access to the original pdb library. The default is to server on port 4444. It's a bit crude, but works for simple debugging
  2. rpdb2 - The rpdb2 library is wrapped in vsi.tools.vdb_rpdb2 to give you network access to a debugger simliar to pdb, but is actually running in pythons built in profiler instead. This has some speed benefits and can also handle multiple threads and can follow (one process only, but you get to pick with process) multi process forking. You can debug the program using either the rpdb2 command line tool or the winpdb GUI tool.

Example

import vsi.tools.vdb_rpdb2 as vdb
vdb.set_trace(_rpdb2_pwd="synsepalum dulcificum")

Start up winpdb, using the same password, and attach to your program where the set_trace left off.

just build_winpdb
just winpdb

To automatically break on the start of a task, set the environment variable VIP_CELERY_DBSTOP_ON_START to a regular expression for the task name (.* for all tasks). The regex matches the task name, for example

voxel_globe.create_site.tasks.create_site
voxel_globe.event_trigger.tasks.event_trigger
voxel_globe.filter_number_observations.tasks.filter_number_observations

This will dump you into the celery prerun section. Step up celery/app/trace.py and step in the -*- TRACE -*- try section where fun is called, then step in self.run in __protected_call__. At this point, you should be in your celery task.

Sometime set_trace isn't convenient. Another useful trick is vsi.tools.vdb_rpdb.dbstop_if_error/vsi.tools.vdb_rpdb2.dbstop_if_error which will automatically attach a post mortem debugger after an uncaught python exception. This is useful for only entering the debugger for failed tasks (albeit after the program has crashed and can no longer be recovered).

Celery has an on_failure method that is called when ANY celery task raises an uncaught exception. To automatically trigger a rpdb post mortem debugger on failure, set the environment variable VIP_CELERY_DBSTOP_IF_ERROR to 1. This is typically set in the local_vip.env file. Setting to 1 will start a pm debugger on any failed task, without having to change any code. This is good for when you aren't expecting any crashes, but are ready to debug them. You shouldn't leave this on all the time, as failed tasks will pile up waiting to be debugged and never end.

Running a task locally

It is possible to run a task in a python session, using the apply function to run a task synchronously. A task function should never be run directly! All voxel globe tasks are meant to be bound to a celery task and running it unbound (directly) will result in error. The /apps/task/status/{taskid} endpoint can be used to view status on synchronous task calls.

Example

#Normally handled by celery-django
import django
django.setup()

#Normally run by celery.py
import boxm2_register
boxm2_register.smart_register = True

#Create a service object for the database
import voxel_globe.meta.models as models
si = models.ServiceInstance(serviceName='debug')
si.save()

#Setup logging you you see all the logging messages
import celery.utils.log
celery.utils.log.base_logger.setLevel(celery.utils.log.logging.DEBUG)

t = tasks.generate_error_point_cloud.apply(args=(voxel_world_id, threshold, None), task_id=str(si.id))

Debugging C++ (Currently Notebook/Celery workers only)

It's a little harder to get gdb attached, but there are a number of methods

Method 1 - Visual Studio Code

Initial setup

  1. Download and install Visual Studio Code
  2. Start VS Code
  3. Go to extensions, and type in cpptools, and make sure you select the Microsoft plugin and install.
  4. File -> Open Folder, and select the voxel_globe directory

Attaching to process in celery/notebook

  1. In your local_vip.env set VIP_VXL_BUILD_TYPE to Debug or RelWithDebInfo and build vxl (just vxl)
  2. (Optional) Set your breakpoints in vxl
  3. Edit your .vscode/launch.json file to include the following configuration (Don't forget to substitute {container name}). I suggest making a new configuration and naming it differently for each container name
{
    "name": "C++ Attach",
    "type": "cppdbg",
    "request": "attach",
    "program": "/usr/bin/python",
    "processId": "${command:pickRemoteProcess}",
    "pipeTransport": {
        "pipeCwd": "${workspaceRoot}",
        "pipeProgram": "docker",
        "pipeArgs": [
            "exec", 
            "-i", 
            "{container name}", 
            "sh",
            "-c"
        ],
        "debuggerPath": "/usr/bin/gdb"
    },
    "sourceFileMap": {
            "/opt/vip":"${workspaceRoot}",
            "/vxl_src": "${workspaceRoot}/external/vxl_src"
    },
    "linux": {
        "MIMode": "gdb",
        "setupCommands": [
            {
                "description": "Enable pretty-printing for gdb",
                "text": "-enable-pretty-printing",
                "ignoreFailures": true
            },
            {
                "description": "Auto load symbols when loading an .so file",
                "text": "set auto-solib-add",
                "ignoreFailures": false
            }
        ]
    },
    "osx": {
        "MIMode": "gdb",
        "setupCommands": [
            {
                "description": "Enable pretty-printing for gdb",
                "text": "-enable-pretty-printing",
                "ignoreFailures": true
            },
            {
                "description": "Auto load symbols when loading an .so file",
                "text": "set auto-solib-add",
                "ignoreFailures": false
            }
        ]
    },
    "windows": {
        "MIMode": "gdb",
        "setupCommands": [
            {
                "description": "Enable pretty-printing for gdb",
                "text": "-enable-pretty-printing",
                "ignoreFailures": true
            },
            {
                "description": "Auto load symbols when loading an .so file",
                "text": "set auto-solib-add",
                "ignoreFailures": false
            }
        ]
    }
}
  1. Make sure export VIP_GDB=1 is set in your local_vip.env file, and restart celery/notebook
  2. Start/Run your process in celery/notebook
  3. Run Launch configuration in VS Code
  4. Choose the pid inside the container to attach to
  5. Debug like it's 1999!

Launching a program

Launching is very similar to attaching explained above. The container must already be running. The container is not launched, only the process is launched in the container.

  1. This time, edit your launch.json file to include (don't forget to fill in {container name}, {program}, and {program args}
{
    "name": "C++ Launch",
    "type": "cppdbg",
    "request": "launch",
    "program": "/bin/bash",
    "pipeTransport": {
        "pipeCwd": "${workspaceRoot}",
        "pipeProgram": "docker",
        "pipeArgs": [
            "exec", 
            "-i", 
            "{container name}", 
            "sh",
            "-c"
        ],
        "debuggerPath": "/usr/bin/gdb"
    },
    "sourceFileMap": {
            "/opt/vip": "${workspaceRoot}",
            "/vxl_src": "${workspaceRoot}/external/vxl_src"
    },
    "args": [
        "/opt/vip/wrap", {program}, {comma separated program args}
        ],
    "stopAtEntry": false,
    "cwd": "/opt/vip",
    "environment": [],
    "externalConsole": true,
    "linux": {
        "MIMode": "gdb",
        "setupCommands": [
            {
                "description": "Enable pretty-printing for gdb",
                "text": "-enable-pretty-printing",
                "ignoreFailures": true
            },
            {
                "description": "Auto load symbols when loading an .so file",
                "text": "set auto-solib-add",
                "ignoreFailures": false
            }
        ]
    },
    "osx": {
        "MIMode": "gdb",
        "setupCommands": [
            {
                "description": "Enable pretty-printing for gdb",
                "text": "-enable-pretty-printing",
                "ignoreFailures": true
            },
            {
                "description": "Auto load symbols when loading an .so file",
                "text": "set auto-solib-add",
                "ignoreFailures": false
            }
        ]
    },
    "windows": {
        "MIMode": "gdb",
        "setupCommands": [
            {
                "description": "Enable pretty-printing for gdb",
                "text": "-enable-pretty-printing",
                "ignoreFailures": true
            },
            {
                "description": "Auto load symbols when loading an .so file",
                "text": "set auto-solib-add",
                "ignoreFailures": false
            }
        ]
    }
}
  1. Run Launch configuration
  2. Debug

Example

Start a notebook with (attach)

import boxm2_adaptor as b
b.ocl_info()

or launch with args

"args": [
  "/opt/vip/wrap", "python", "-c", "import boxm2_adaptor as b; b.ocl_info()"
]

And add a breakpoint in bocl_info_process.cxx

Method 2 - Text gdb only (simplest)

  1. Run just gdb. This will start gdb inside the docker in tui mode
  2. Select the process to attach to
  3. Debug!

Note: When debugging celery, knowing which process to attach to can be a problem. The easiest solution to this is to add the following to your local_vip.env file

export VIP_NUMBER_CORES=1

Method 2

Unlike Method 1, gdbserver is run against a specific pid. Method 1 should be superior, but just in case

  1. Add export VIP_GDBSERVER=1 to your local_vip.env file
  2. Determine the pid of the target process. (See Method 1)
  3. Start gdbserver in normal attach mode. Run ./just gdbserver2 After selecting the container you want to run gdbserver in, a list of pids running in that container will be displayed. Enter the number for the pid you want to attach to
  4. A file gdbinit is auto generated in the base repo directory. This is a list of commands that need to be run when starting gdb. You can use gdb -e {voxel_globe_repo_dir}/gdbint if using pure gdb
  5. Connect your favorite gdb debugger to remote gdb, using localhost:{port#} printed by the ./just gdbserver2 command.

VS Code Example

  1. Start VS Code (with cpp tools installed)
  2. Load up

Eclipse Example

  1. Start Eclipse (with CDT installed)
  2. Run-> Debug Configuration...
  3. Select C/C++ Remote Application
  4. Create New Launch Configuration and name it
  5. Under the Main tab, a C/C++ Application must be specified. Anything will do, try /bin/ls. This is just how the Eclipse CDT is hard coded to behave.
  6. Under Debugger Tab, In the Main sub-tab under GDB command file enter {voxel_globe_repo_dir}/gdbinit. The gdbinit gets updated by the ./just gdbserver2 command every time.
  7. Under the Connection Sub tab, you will need to manually update the hostname and port number every time you restart a container.
  8. You no longer have to (but can if you want) Go to the source tab and add {voxel_globe_repo_dir}/external to get the source files to break correctly. This is also handled by gdbinit now
  9. Click Debug. From now on you can just run Launch. You will only have to reconfigure it every time the port number changes (or you switch which container you are debugging, since it will be a different port number)

Method 3

Less Preferred

  1. Modify C. Add

     #include <signals.h> //At top of file

 printf("%d\n", getpid());
 raise(SIGSTOP); // Where you want the debugger to attach

    Or c++

     #include <csignal> //At top of file

 cout << getpid() << endl; 
 raise(SIGSTOP); // Where you want the debugger to attach

  2. Run ./just vxl restart celery

  3. User either Method 1 or Method 2 to connect to gdb server

Visual Studios 2015 Method

Based off of this

While Method 4 will work in Windows, you will be limited to using just gdb in tui mode, not the ideal debugger. Visual Studios 2015 has a plugin that allows debugging over ssh.

Install Android Plugin

  1. Install Visual Studio 2015 (I only tested Professional edition)
    • If it is already installed, go to Control Panel, Select "Programs and Features", scroll down to "Microsoft Visual Studio 2015", right click and choose Change. Adter the window pops up and scans, choose Modify
  2. Under "Cross Platform Mobile Development" turn on "Visual C++ Android Development"
  3. Continue install, restart, whatever is required.
  4. Install putty, plink, puttygen, and pscp

Create shared key

  1. ./just start to start celery/notebook

  2. ./just ssh-keygen, and then choose the celery container to generate ssh keys

$ just ssh-keygen

  1. vip-celery-0 (32770)
  2. vip-notebook (32771)
  1. Now a gdb_rsa key will appear in the root repo directory
  2. Convert the rsa key to putty format using puttygen, for example puttygen gdb_rsa and click "Save Private Key" and save ppk file, say gdb_rsa.ppk
  3. Start putty, enter "localhost" for the "Host Name", and the number in parentheses when you ran just ssh-keygen, for example for vip-celery-0, the port number would be 32770. (It can be different every time you run VoxelGlobe)
    • Click the [+] next to SSH on the categories on the left, select "Auth" and in the "Private key file for authorization" choose get gen_rsa key generated above
    • Alternatively from command prompt, putty localhost -P {{port number}} -l root -i gdb_rsa.ppk
  4. Say yes to the fingerprint

Create a new project for ssh debugging

Based off of this

  1. New Project
  2. Visual C++, Makefile project
  3. Next
  4. Enter the following Build command - "C:\Program Files\Git\usr\bin\bash.exe" d:\vip\just vxl
  • Rebuild command - "C:\Program Files\Git\usr\bin\bash.exe" d:\vip\just vxl-cmake vxl
  1. Next
  2. Check "Same as debug configuration" on, Next

Native Debugger using VS Code

Todo...

Printf debugging

Ideally, printing in a piece of python/C++ code should output to the logs. This is sometimes not direct, trivial, or instant, making debugging difficult. This section goes over each program to explain the caveats

  1. Nginx

    There is no custom code running on nginx. There is a 3rd party mod needed to echo in an nginx config, not included in voxel_globe

  2. Asgi (runworker)

    stdout and stderr should both just work in python and c. stderr always just works. stdout was buffered, but with the -u flag added, it should work too now.

  3. Uwsgi

    stdout and stderr should both just work in python and c

  4. Daphne

    stdout was buffered, but with -u flag added, it is now unbuffered. Currently no voxel_globe code runs in Daphne, so there is no way to print in it, but there is no need to either. If there ever is, stdout and stderr should work

  5. rabbitmq N/A

    RabbitMQ is a black box, soon to be replaced by redis

  6. Notebook

    Notebook is tricky. Python python stdout prints in notebook, and python stderr prints in the notebook, in red. C stdout, and C stderr print out in the logs.

  7. Redis N/A

    Redis is a black box. There is a "redis desktop manager" useful for exploring redis with. just build_rdm rdm

  8. Flower N/A

  9. Postgresql N/A. There are ways to print out ever SQL query in Django

  10. celery

    Python stdout and stderr are redirected to the logging via a LoggingProxy. There is no good way (and many bad ways) to capture C stdout and stderr. One such way is using vsi.tools.redirect.StdRedirect

    with StdRedirect(open(os.path.join(voxel_globe.tools.log_dir(), 
                             self.request.id)+'_out.log', 'w'),
           open(os.path.join(voxel_globe.tools.log_dir(), 
                             self.request.id)+'_err.log', 'w')):

Clone this wiki locally