externalize timeouts?

[netmg]

Hi Christophe -

I'm just scraping the surface on your code, but after experiencing some timeouts with our MISP server I experimented with the same line that was changed for #52 .

88 self.misp = PyMISP(url=misp_url, key=misp_key, ssl=misp_verify, debug=misp_debug, tool='misp_maltego', timeout=(2, 60))

I found I could modify the 2nd parameter and avoid the timeouts.

Would it be possible to externalize both of these timeout values, making them accessible in MISP_maltego.conf ?

[cvandeplas]

Sure, that can be done, however are you sure that's really what you want?

Right now the timeout is on 2 seconds to connect (tcp), and 60 seconds for receiving the data.
The 2 seconds allow a user to know quite quickly when a server is not reachable (2 seconds).
The 60 seconds are how long MISP-maltego will wait for the response from a server. Waiting for a whole minute to get results is already kinda long, no?

Which timeout are you hitting? (the connect or response?)

[netmg]

Yes - it does seem a bit extreme. But we have a massive instance of MISP behind the curtain, and regularly see multi-minute waits. We've gotten used to it - and this enhancement would help us work around it.

It's the response timeout that we are getting hit with, the connect seems ok (at least in our circumstance).

[github-germ]

Hi @cvandeplas -- I work with @netmg. I'm new to Maltego -- your plugin is very cool! And thanks for being so responsive here!

Being that these recent postings peeked my curiosity re' the timeout, I wrote a simple script that calls pymisp.search the same as your AttributeToEvent line 176 to measure time and data size simulating a To MISP Events Maltego click on an attribute that resides on an event with 1M attributes (yes, large events are common here). Here's the interesting results based on various scenarios we used when performance becomes an issue:

1. Your search call as is with controller='events' retrieved:

• 3GB in 266 secs

2. Adding metadata=True with controller='events' retrieved:

• 300KB in 10 secs

3. Changing the search to controller='attributes' retrieved:

• 300KB in 10 secs

Being new with your code, what's not clear, is whether options 2 and 3 above will yield the data content you need further downstream in the Maltego client. I'm eager to hear your thoughts. Thanks again.

[packet-rat]

I've also found that setting a "limit" value, even if very high (i.e., 1,000,000) can significantly improve pymisp response times (at least in JupyterLab):

r = misp.search(
    controller='attributes',
    type_attribute=['ip-src', 'ip-dst', 'url', 'domain'],
    #org='IBM-ATP',
    tags=['nist-cyber-framework:function="protect"','nist-cyber-framework:function="detect"'],
    metadata=True, 
    pythonify=True, 
    publish_timestamp='3d', 
    limit=1,000,000)

[cvandeplas]

Being new with your code, what's not clear, is whether options 2 and 3 above will yield the data content you need further downstream in the Maltego client. I'm eager to hear your thoughts. Thanks again.

There is indeed optimisation possible, especially with events that are so large.
The currently logic is:

• search for the event that contains that attribute, return the full event including attributes & objects
• search in the result if the attribute is part of the event , or if it is part of the object
• if it's a direct attribute: return the event
• if it's within an object: return a MISPObject instead

The reason I did so was that without the check there would be a direct link between the attribute and the event, breaking a bit the visual look. However I did not anticipate you would be using such large datasets.

Changing the code with metadata=True will break things for sure. The event_to_entity for example expects a full event, as it uses that to return the attribute & object count.

So technically the code can be changed to accommodate this situation, but honestly, I think we need to look at it with a big step back to not break the other functionality & usability. Maybe some things can be improved by requesting small changes on MISP side too. (like returning attribute & object counts)

Please feel free to play around with the code, I'm happy to review pull-requests and merge them in the project code.

[github-germ]

@cvandeplas Thanks for the feedback. Don't have cycles to attack this now, but it's in the queue.