Delete sessions & pcap

[Anadema]

Hello Malcolm community,

The tool is amazing, I love it and it could be a perfect pcap library. The only issue I have is to delete a session, if I follow the arkime documentation, there is a arkime-delete binary we can use to delete and clean a session using tags and that is perfect, but when I'm on Malcolm in the Arkime container I don't have this command.

So my question is how to delete session and pcap based on tags please?

[Anadema]

I found that in the internal documentation but where is this privilege settings, I'm login as admin so I must have. And where is the button?

[Anadema]

In fact I have just that so must be a privilege to activate, but I'm admin.

[Anadema]

I found that in user_settings.json, let's test...

[Anadema]

I found also some roles files

arkime_hunt_access.json
arkime_pcap_access.json
arkime_read_access.json
arkime_read_write_access.json

But don't know how manage that. Maybe the best solution is to install only Arkime wihout Malcolm...

[Anadema]

Maybe it's not the good way to do, if the user tab is not available, all must be manage by another functionnality, but the user malcolm page is very limited and it's not possible to add or remover privilege.

So how to delete sessions / pcap based on tags?

[Anadema]

In fact the best solution could be to apply the role superAdmin on Arkime to the user admin of Malcolm. Is it possible?

[mmguero]

Unless you're using role-based access control with Keycloak in Malcolm, your user does have admin access: it's just that Malcolm manages user accounts at a higher level than Arkime, which is why some of the user-management stuff in Arkime is not available in Malcolm (in other words, you manage user accounts at the Malcolm level, not the arkime level).

As far as the deleting of data in Malcolm goes, here are some options.

Automatic storage management

Malcolm can be configured to make sure you don't run out of disk space, see Managing Disk Usage.

To delete everything (wipe Malcolm sessions and PCAP back to a clean state)

./scripts/wipe

To delete indexes (i.e., sessions)

1. Go into Dashboards
2. Click the tool menu on the upper-left and select Management > Index Management:

3. select Indexes
4. filter by arkime_sessions:

5. Select the dates of the indexes you want to delete
6. click Actions > Delete

To delete PCAP

Malcolm doesn't surface an option in the UI to delete PCAP, but you can delete PCAP manually by just removing the PCAP file from within your ./pcap directory under the Malcolm installation directory, or wherever Malcolm is storing PCAP files:

$ grep -B 1 "target: /pcap" docker-compose.yml
      source: ./pcap
      target: /pcap
...
$ ls -l ./pcap/processed/
total 2,936,516,608
-rw-r--r-- 1 user user   1,190,578 Jan 29 14:18 NBSITEID0,09ffabf7774a43a38c9768f2046fd385.pcap
-rw-r--r-- 1 user user     363,986 Jan 29 14:18 NBSITEID0,0e328ab712b248438717a5b3ebef33a8.pcap
-rw-r--r-- 1 user user     223,799 Jan 29 14:08 NBSITEID0,110378724.pcap
-rw-r--r-- 1 user user  14,794,470 Jan 29 14:08 NBSITEID0,114019667.pcap
-rw-r--r-- 1 user user   6,708,630 Jan 29 14:09 NBSITEID0,116070525.pcap
-rw-r--r-- 1 user user   6,375,002 Jan 29 14:09 NBSITEID0,117922543.pcap
-rw-r--r-- 1 user user      15,528 Jan 29 14:09 NBSITEID0,120903667.pcap
...
$ rm ./pcap/processed/NBSITEID0,110378724.pcap

[Anadema]

Is there a way to delete the file in the Files tab in Arkime, if I delete it manually from the folder processed and upload with ssh but ii's not deleted in Arkime?

To delete the session it's ok.

[mmguero]

Yes, there is: run $ARKIME_DIR/db/db.pl --insecure "$(grep -oP '^elasticsearch=\K.*' /opt/arkime/etc/config.ini)" rm-missing "${PCAP_NODE_NAME}-upload" in a shell inside the arkime container, e.g.:

$ docker compose exec -u $(id -u) arkime bash
arkime@arkime:~$ $ARKIME_DIR/db/db.pl --insecure "$(grep -oP '^elasticsearch=\K.*' /opt/arkime/etc/config.ini)" rm-missing "${PCAP_NODE_NAME}-upload"
Need to remove references to these files from database:
/data/pcap/processed/NBSITEID0,Advantech.pcap 

Type "YES" to continue - Do you want to remove file references from database??
YES

If this file was captured live instead of uploaded, remove the -upload from the end of the command.

[Anadema]

Hello mmguero, thank you very much, I have deleted some index and it's working perfectly. I have three questions remaining if you can give me advice:

1- Is it possible to modify the tags also using opensearch?
2- If I install Keycloak can I have the option missing with my current admin role in Arkime, modify tag, delete session, directly in the UI?
3- How to delete or modify the content of the Files tab in Arkime, because it's not updated when we clean opensearch index?

[mmguero]

With Keycloak as your auth method (you can use Malcolm's embedded keycloak instance) I believe your user can have the permissions to remove things from within the Arkime interface.

Without Keycloak, I think I know how to allow your user to delete things via the Arkime interface (full disclosure, haven't tested but looking at the source code):

• In your Malcolm directory, edit ./arkime/etc/user_settings.json and change removeEnabled from false to true.
• In your docker-compose.yml file, in the volumes section for the arkime service, add this:

    - type: bind
      bind:
        create_host_path: false
      source: ./arkime/etc/user_settings.json
      target: /opt/arkime/etc/user_settings.json
      read_only: true

At this point you'd have to wipe malcolm to a blank slate (so Arkime will recreate the user from scratch) with ./scripts/wipe and restart it, and you should have those permissions.

If you only have one user account (the one that you create with auth_setup) then that should be all you need to do. If you are going to have more than one user account managed with Malcolm's local account management then you'd have to do a similar thing in [./etc/arkime/config.ini](https://github.com/idaholab/Malcolm/blob/f1e0199919e7ff0c11d96c4310e84e58f0831413/arkime/etc/config.ini#L73) and change removeEnabled to true on the userAutoCreateTmpl line, and bind-mount it in to your arkime container in the same way:

    - type: bind
      bind:
        create_host_path: false
      source: ./arkime/etc/config.ini
      target: /opt/arkime/etc/config.orig.ini
      read_only: true

[Anadema]

Nice, I'm going to check that thank a lot.