Regarding adding a database of known file hashes

[reacan]

Dear digital forensics enthusiasts!

I found out about IPED from an Interpol newsletter. I work for the State Police of Latvia and I demonstrated the capabilities of IPED to my colleauges and everyone was very impressed. We all would like to send a big thank you to everyone involved in this project, thank you for making something like this possible.

I know you guys are not techsupport but I do not know whom else to ask, so I am really hoping that you could help us out a little bit.

I am examining couple of servers with IPED. I finished the 1st server, gave the portable case to investigators and they were super happy I still have 5 servers to go. Since there is Hyper-V Replication enabled on these servers I want to process the next server using the hashes from the previous one to ensure that more or less unique information is included in the new portable case.

So far I have tried the following:

1. Created a hash database with 1 hash per line in plaintext, edited LocalConfig.txt to point to that plaintext file, tried to process using -importkff parameter. Did not succeed.

2. Used a NIST NSRL database to test out this function in general, since I read that this format is supported. However I got the same error as before:

java -jar ~/Apps/IPED/target/release/iped-3.17.1/iped.jar -d /media/RAID/1.Apskates/XXXX_serveri/XXXX/vhdx_to_vhd/XXXX-data.vhd -l /media/Skyhawk10TB/1.Apskates/XXXX_serveri/XXXX_srvs_keywords_v2.txt -o /media/Skyhawk10TB/1.Apskates/XXXX_serveri/XXXX/iped_XXXX_vol1 -tz GMT+2 -profile forensic -importkff /root/Documents/hash_base_XXXX/NIST_NSRL/NSRLFile.txt

ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
Exception in thread "main" java.io.IOError: java.io.IOException: storage has invalid header
at org.mapdb.StoreDirect.checkHeaders(StoreDirect.java:254)
at org.mapdb.StoreDirect.(StoreDirect.java:207)
at org.mapdb.DBMaker.extendStoreDirect(DBMaker.java:971)
at org.mapdb.DBMaker.makeEngine(DBMaker.java:758)
at org.mapdb.DBMaker.make(DBMaker.java:701)
at dpf.sp.gpinf.indexer.process.task.KFFTask.init(KFFTask.java:132)
at dpf.sp.gpinf.indexer.process.task.KFFTask.init(KFFTask.java:74)
at dpf.sp.gpinf.indexer.IndexFiles.importKFF(IndexFiles.java:188)
at dpf.sp.gpinf.indexer.CmdLineArgsImpl.handleSpecificArgs(CmdLineArgsImpl.java:323)
at dpf.sp.gpinf.indexer.CmdLineArgsImpl.takeArgs(CmdLineArgsImpl.java:296)
at dpf.sp.gpinf.indexer.IndexFiles.(IndexFiles.java:124)
at dpf.sp.gpinf.indexer.IndexFiles.main(IndexFiles.java:300)
Caused by: java.io.IOException: storage has invalid header
... 12 more

I am using a Linux system and I am comfortable working with the commandline, however I am fairly new to forensics. I will greatly appreciate any help on how I can get IPED to recognize my hashes and ignore them

Kind regards and wishing you all a wonderful day ahead,

Dāvis

[lfcnassif]

Hi @reacan, thank you for your interest in IPED tool! No problem asking here for now, currently we don't have a user channel, possibly we will create an user mailing list so the community could help users.

Did you read the wiki about importing NSRL hashset format? It must be done alone before processing: https://github.com/lfcnassif/IPED/wiki/User-Manual#NIST-NSRL
Let me know if the docs are not clear.

You should give your hashset a "ProductName" in NSRLProd.txt file (just this and NSRLFile.txt file are used in importing step). Finally, go to IPEDConfig.txt and turn excludeKffIgnorable on.

Actually conf/KFFTaskConfig.txt lists alert product names, not ignorable, you don't need to edit it.

Hope this helps.

[reacan]

Thank you, Luis, for getting back to me. I did not read that manual page, but now that I have, it starts to make sense :) I will try my luck with getting it to work!

I think soon more and more people will start using IPED. The tool is very powerful. Recently I processed an image containing an APFS FileVault encrypted partition providing the password in the processing options and IPED handled it very well to say the least :)

[gisms]

I was trying to do the same informed here: "I processed an image containing an APFS FileVault encrypted partition providing the password in the processing options and IPED handled it very well to say the least :)" but I didn't find this processing option. Can anyone help me with that?

[lfcnassif]

If the image uses the standard APFS encryption (that is not FileVault if I remember), you can pass the password through the command line using the -password option.

[gisms]

It was exactly the case. It worked :))))))))))))))

I had forgotten about --help. I used the -p option

[lfcnassif]

We should finish the manual in the wiki, I hope I'll have time in the future...

[gisms]

So i'll start helping with that.
Every time I experience something that isn't on the wik, I'll write it down

[lfcnassif]

You are welcome!

[reacan]

I managed to get it to work, the instructions were perfectly clear, thank you very much. I also managed to add my own hash database, however the solution is not elegant at all. I wrote this oneliner:

echo "\"SHA-1"\","\"MD5"\","\"CRC32"\","\"FileInode"\","\"FileSize"\","\"ProductCode"\","\"OpSystemCode"\","\"SpecialCode"\" && find -type f -exec bash -c 'a=(sha1sum "{}"|cut -d " " -f 1) b=(md5sum "{}"|cut -d " " -f 1) c=(crc32 "{}") d=(stat -c %i "{}") e=(stat -c %s "{}") \\; echo "\"$a\",\"$b\",\"$c\",\"$d\",$e,200828,\"365\",\"\""' \; | sed -e 's/[./]//g' -e 's/[\]//g' > NSRLFile.txt

then I created a corresponding NSRLProd.txt and it worked, got the hashbase imported.

I am wondering which columns in the NSRLFile.txt are not critical for IPED to be able to perform known hash look up? I only have MD5 enabled in the processing options, if I create my NSRLFile.txt with static information in all the other fields except MD5, FileName, FileSize, will IPED be able to correctly ignore known hashes?

The columns in the NSRLFile.txt are:
SHA-1, MD5, CRC32, FileName, FileSize, ProductCode, OpSystemCode, SpecialCode

[lfcnassif]

Hi @reacan, reading the (old) source at https://github.com/lfcnassif/IPED/blob/master/iped-engine/src/main/java/dpf/sp/gpinf/indexer/process/task/KFFTask.java
seems just sha-1, md5 and ProductCode are used. If you just uses md5, I "think" you can put an empty value in sha-1 column.

Tip: you can search for or group by (in metadata tab) kffgroup property, your known files will have the value specified in NSRLProd.txt file

[reacan]

Hi @lfcnassif, thank you for the insight! I'll do some testing today and I'll report back on what I come up with, maybe it will be useful for someone.

[lfcnassif]

Closing for now. If you still have problems with NSRL importing, please reopen.