General ideas for improving the tool

[MariasStory]

Hi team,
I love the tool and hope that it will be so cool and free in future.

I want to suggest to increase the tools capability, and make it a full forensic investigation tool.
I am sure that you all know it, but I still want to say it.
For this, we have to think about the Five Ws. I prefer to start with the question "When" and follow with "What".

So, in our case, I suggest to:

1. Parse as many time stamps as possible, with events related to it.
2. Use full text indexing for additional indicators, as it is done now.
   Good example for parsers approach is "plaso". Still, IPED is a complete tool on its own.

Most of the features are already present, just need adjustment:

1. Add parsed/extracted artifacts in the main table. It is a specially useful in the timeline view
   • Some of the table details will be missing, for the extracted records. It is OK. Most important that when you search for an indicator you'll get a timestamp or related indicator that can be farther correlated.
   • The extracted records should be "marked" in order to recognize/hide or filter them
2. Bookmarks approach is OK for the events tagging. May need to be extended.
   • Another tool used in such cases is Timesketch
3. Make a possibility to have the search results in a separate window and show the selected item in the timeline view with the possibility to scroll and mark related events
   • Maybe, something similar to the Excel search box

Please consider my suggestions and fill free to improve or criticize.

Thank you,
Anatoliy

[lfcnassif]

Hi @MariasStory. Most of your suggestions are already the goal of other tickets or pull requests, for example see:

#1193 (if you could help, tests are welcome here)
#467 (some of them only needs regex rules to be configured, even non developers can help)
#668 (already suggested by you, being implemented on #1285, tests are welcome too)
#452
#281
etc

You can find others searching the issue tracker. This is an open source project with very few human resources 100% dedicated (currently just 2, and this is temporary, most past time it was just 1 full time), so any contribution is welcome, we aren't able to implement all desired features, so we need your help.

I'm updating the title since IPED already is a forensic investigation tool used by law enforcement and independent examiners of many countries. Since this is about very general suggestions, I'm also moving to discussions.

[lfcnassif]

Just to complement, IPED is a general purpose forensic tool, used for very different use cases. If you want to use just one specific feature, e.g. timeline, you would be better using a specific tool for timeline, since specific solutions will be better than general purpose ones for a specific task. edited: Of course, if you need many different features, an all-in-one tool would be easier to use than many separate tools...

And if you can help implementing one of your suggestions, please submit a pull request, I'll be happy to review, test and improve if needed.

Best,

[MariasStory]

I just want to make sure that the tool has the features that cover the most important parts of forensic investigation.
From my point of view, forensics is all about the timeline.
We need to explain when and what happened with all possible details and evidence.

[lfcnassif]

3. Make a possibility to have the search results in a separate window and show the selected item in the timeline view with the possibility to scroll and mark related events

About above: at first I decided to create a switch button to change the main table to timeline view (it is a small clock at the table top-right) because of memory consumption concerns: another item table will use more heap memory, that may be an issue with huge cases with dozens of millions of items. But I understand that when a filter is applied (term search, category, combo filter, etc), if the tool user finds an useful artifact and wants to see adjacent events, those events probably will be filtered out. It is possible to check the item, clear all filters, sort by checkbox, select the artifact of interest, then sort by timestamp to see surrounding events, but a more straightforward way to see surrounding events could easy user's life. Displaying them in another table is an option. But when selecting an item on main table, scrolling to the event on the other table couldn't be done automatically, because the artifact could have many different associated events, the user would need to click on a context option and select the timestamp of interest. The opposite is easy: from the separate timeline table view, there is just one source item to be automatically selected on the main table and rendered in the viewer.

So, my question for contributors that may help to implement, test or help in other ways: should we create a separate timeline table view and pay the memory usage cost associated with it, or should we keep current less memory usage approach, where it is possible to see surrounding events with some UI interactions (that some users wouldn't know)?

[lfcnassif]

I took a look at TimeItemId class and I think it uses about 20 bytes for attributes + 12 bytes header = 32 bytes (considering a JVM with less than 32GB of heap). So in a 10 million items case (large, but not huge), a separate timeline table view would use at least 10M * (32 bytes obj + 4 bytes ref) = 360MB to keep the event array, not considering the heap space already used by Lucene structures and to track the new table column sorting state (more + 240MB if current approach is used). That doesn't seem that much, but this should be profiled in practice, possibly we may need different data structures...

[lfcnassif]

track the new table column sorting state (more + 240MB if current approach is used).

If it would be always sorted by timestamp (and never by event type or event source), this may not be needed.

[patrickdalla]

I think a different table could be a good option for the user. I the app manage well to load the timeline table only when the user choose so, it seems a good option. Em qui., 22 de set. de 2022 12:03, Luis Filipe Nassif < ***@***.***> escreveu:

…

track the new table column sorting state (more + 240MB if current approach is used). If it would be always sorted by timestamp (and never by event type or event source), this may not be needed. — Reply to this email directly, view it on GitHub <#1304 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AG247SZEWLPJKGCUEH5TXHTV7R7MFANCNFSM6AAAAAAQCULWZU> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[MariasStory]

My idea was not to have a separate view for all events, but only the search results.
That view can event be limited to ~1000 results.
https://www.tldraw.com/r/1663955549616