Implemented a better handling of (partially) unsalted hashlists #1657

[Eric-Wasson]

I've implemented the things @s3inlc mentioned in his comment on issue #1657.

Regarding the "report back on the hashlist creation with numbers (e.g. valid hashes: xy, invalid hashes: ab).", I went with the approach of modifying the db and adding new attributes to each hashlist (totalLines, emptyLines, validHashesWithoutExpectedSalt, validHashes and invalidHashes). Another approach would be to not store these values and only report them back to the user on the hashlist creation. But then it wouldn't be possible to display these attributes later on, like on the hashlist detail page where the number of hashes without an expected salt may be interesting.

This is how the hashlist data would now look like with the new attributes:

The counting also works if the list gets added using the old UI, but it may make sense to only add the warning about the empty/invalid/... lines in the new frontend, because the old UI will get retired eventually.

What do you think about this proposed solution?

Open questions:

• //TODO: check hash length here in HashlistUtils.class.php on line 904: does it refer to checking if the hash is longer than 0 characters (which is already being checked, so it could be removed) or should additional checks be implemented to check if the hash is as long as expected (e.g. md5 being 32 hex digits)?

Open todos:

• Add handling of these values in the frontend to show a warning after hashlist creation
• Docs (/doc) don't have to be changed but the database has to modified (see image below) and that db change would need to be integrated into an updatescript.

DB change:

[s3inlc]

I'm not sure if it is worth to introduce four new columns to the Hashlist table just for storing the values which are relevant on the hashlist creation as later if you don't have the original file you used for upload, this information is of little use. So we carry four columns for every entry forever just for this initial information. Is there a benefit you would have when seeing that there were some hashes not accepted on the import?

So, as you stated with "Another approach would be to not store these values and only report them back to the user on the hashlist creation.", this describes what I had in mind how it would be done. It's done similarly on the old UI as well e.g. for the pre-cracked hashes import.

The main problem we have here, is that the plan is to have no changes to the database structure to avoid having to do a lot of changes which become irrelevant already again very soon, when we are removing all the stuff from the old UI. It may be that it would work with just not displaying them on the old UI, but all the additional stuff we have to do when we have database changes still require some work on backend and frontend (there will be more attributes on the API, update script, etc.)

TL;DR: IMHO this information is only relevant on the import itself, so there is no need to save it and instead it should just be shown on the import reply.

[Eric-Wasson]

Thanks for the feedback, I moved the creation Information to the meta section on the response:

To get this information, the frontend has to send the getCreationInformation on the POST request to hashlists: