Getting segmentation fault when used under certain load

[guangyudong]

Dear Authors of this plugin,

I'm using this plugin with mongodb and mapnik to render tiles. Basically my c++ program starts an http server (using libmicrohttpd) and response to requests for map tiles. It fetches data in mongodb in runtime, then render and return the image back to the client side.

The versions of the packages I'm using is as following:
Mapnik 2.3.x
Mongodb 2.6

It was working fine when I was developing, when every time I send only one or two requests to see if it works. However, when I tried to deploy my program as the backend map tile server for our map application, which generates a lot of requests, I began to have segmentation fault. This happens everytime I send requests too often, say, more than two per second.

Here is a copy of error messages I got: http://pastebin.com/NdeCK8rF

I wonder if this plugin is thread-safe, and whether there is anything I need to pay special attention when using. And what do you think that might cause this error?

Thanks very much

[springmeyer]

It looks like a failure to connect is either the cause or the result - hard to tell upon first glance.

Overall: keep in mind that by design Mapnik Map's are not designed to be shared between threads (and therefore each rendering thread should have its own map). Can you are more details about how your rendering is working?

[guangyudong]

I just found I even get segmentation fault for the example programs came with the mongodb c++ driver. Let me solve this first. Maybe it is the cause.

By design, my tile program should fetch data from mongodb and render to tiles of size 256x256. Each request will be handled by a separate thread, and they do have separate Mapnik Map objects. What they are sharing is Mapnik parameters and layers, which should be unchanged through all threads.

I'll try to fix the driver problem first. Thanks very much

[guangyudong]

Hi Dane,

I solved segmentation fault of the examples. It was because I had an old version of boost_thread-mt which is 1.41.0 that was installed long time ago. The boost_thread is newer though, 1.51.0.

I have deleted the old boost_thread-mt link and created a new link boost_thread-mt to boost_thread.

I also recompiled the mongodb.input and my program. I'm getting less error but still finally got segmentation fault. The good thing is that the error message is much less, following is a copy of the output:

/tile/14/4538/6983
/tile/14/4535/6983
### exception: dbclient error communicating with server: localhost:27017
/tile/14/4537/6983
### Unknown exception.
/tile/14/4537/6984
/tile/14/4538/6984
/tile/14/4535/6984
### exception: dbclient error communicating with server: localhost:27017
/tile/14/4535/6983
### Unknown exception.
/tile/14/4538/6982
### exception: assertion src/mongo/util/net/message_port.cpp:284
/tile/14/4536/6984
Segmentation fault (core dumped)

Of which the /tile/14/xxxx/xxxx are correctly handled requests.

Any ideas?

Thanks

[guangyudong]

Hi,

Today I tried not to use multithreading to handle requests in the http server and it is working well, but apparently slower. So I guess the problem is related to the multithreading.

Then it came to my mind that I have modified the source code of mapnik-mongo a little bit in order to compile with mongodb 2.6. I did two modifications:

1. remove all <mongo/client/dbclientxxxx.h> in header, because dbclient.h has already included all the headers in the version I'm using
2. in connection.hpp, line 49. The static function mongo::ScopedDbConnection::getScopedDbConnection was removed in a newer version.
   Please see the difference in the documentation:
   http://api.mongodb.org/cplusplus/2.6.1/classmongo_1_1_scoped_db_connection.html
   http://api.mongodb.org/cplusplus/2.3.0/classmongo_1_1_scoped_db_connection.html

My modification is change line 49 in connection.hpp to

conn_.reset(new mongo::ScopedDbConnection(connection_str));

I'm not sure if this modification is correct and thread-safe. Any suggestions?

Thanks

[springmeyer]

Not sure, but seems like you are on the right track. I've not looked at this code for a while, but it seems important to get a handle on the reentrancy of the client connections. Also, sharing mapnik layers across threads is usually not safe so try giving each thread its own mapnik map and layer.

[lintonye]

@springmeyer with this issue not solved, I wonder if the plugin is considered stable?

[springmeyer]

@lintonye - correct, this plugin is not production ready. It was written only as an experiment and has not seen much testing. Also I'm not actively maintaining it - so if someone wants to take over ownership of it, please feel free.