Recorded Future Integration

[craigsmooth]

Version

2.4.40

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Distributed

Location

airgap

Hardware Specs

Meets minimum requirements

CPU

8

RAM

32

Storage for /

3TB

Storage for /nsm

10TB

Network Traffic Collection

span port

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

I am trying to add the Recorded Future Integration but I see no events being generated in the data stream. Are there any troubleshooting tips or has anyone had any success with this integration that provide any guidance. Thank you very much.

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[cm-ops]

Do you see an index created?

[cm-ops]

Okay, so you see the integration in the policy in Fleet, right? Did you also allow the port in the firewall in SOC? What happens if you tcpdump the interface for the port?

[craigsmooth]

Correct. I see the integration under the so-grid-nodes_general policy. If I do a curl or wget to the Recorded Future API endpoint of https://api.recordedfuture.com/v2 then I get a response. Running a tcpdump looking for traffic to the IPs that the hostname resolves to shows me no traffic. Anything else you would suggest I check to troubleshoot this. Thank you @cm-ops

[craigsmooth]

Actually I do see some traffic but still nothing in the index.........

15:36:43.870438 IP so2mgr.57736 > 162.159.128.62.https: Flags [S], seq 4207859176, win 64240, options [mss 1460,sackOK,TS val 3437629617 ecr 0,nop,wscale 7], length 0
15:36:43.878612 IP 162.159.128.62.https > so2mgr.57736: Flags [S.], seq 3335352209, ack 4207859177, win 65160, options [mss 1400,sackOK,TS val 1059747002 ecr 3437629617,nop,wscale 13], length 0
15:36:43.878654 IP so2mgr.57736 > 162.159.128.62.https: Flags [.], ack 1, win 502, options [nop,nop,TS val 3437629625 ecr 1059747002], length 0
15:36:43.878854 IP so2mgr.57736 > 162.159.128.62.https: Flags [P.], seq 1:271, ack 1, win 502, options [nop,nop,TS val 3437629625 ecr 1059747002], length 270
15:36:43.886845 IP 162.159.128.62.https > so2mgr.57736: Flags [.], ack 271, win 7, options [nop,nop,TS val 1059747010 ecr 3437629625], length 0
15:36:43.890485 IP 162.159.128.62.https > so2mgr.57736: Flags [.], seq 1:1449, ack 271, win 8, options [nop,nop,TS val 1059747014 ecr 3437629625], length 1448
15:36:43.890513 IP so2mgr.57736 > 162.159.128.62.https: Flags [.], ack 1449, win 501, options [nop,nop,TS val 3437629637 ecr 1059747014], length 0
15:36:43.890528 IP 162.159.128.62.https > so2mgr.57736: Flags [P.], seq 1449:3941, ack 271, win 8, options [nop,nop,TS val 1059747014 ecr 3437629625], length 2492
15:36:43.890535 IP so2mgr.57736 > 162.159.128.62.https: Flags [.], ack 3941, win 489, options [nop,nop,TS val 3437629637 ecr 1059747014], length 0
15:36:43.892175 IP so2mgr.57736 > 162.159.128.62.https: Flags [P.], seq 271:335, ack 3941, win 501, options [nop,nop,TS val 3437629638 ecr 1059747014], length 64
15:36:43.892536 IP so2mgr.57736 > 162.159.128.62.https: Flags [P.], seq 335:702, ack 3941, win 501, options [nop,nop,TS val 3437629639 ecr 1059747014], length 367
15:36:43.901361 IP 162.159.128.62.https > so2mgr.57736: Flags [.], ack 702, win 8, options [nop,nop,TS val 1059747024 ecr 3437629638], length 0
15:36:43.966490 IP 162.159.128.62.https > so2mgr.57736: Flags [P.], seq 3941:4887, ack 702, win 8, options [nop,nop,TS val 1059747089 ecr 3437629638], length 946
15:36:43.966490 IP 162.159.128.62.https > so2mgr.57736: Flags [P.], seq 4887:4914, ack 702, win 8, options [nop,nop,TS val 1059747089 ecr 3437629638], length 27
15:36:43.966490 IP 162.159.128.62.https > so2mgr.57736: Flags [P.], seq 4914:4938, ack 702, win 8, options [nop,nop,TS val 1059747089 ecr 3437629638], length 24
15:36:43.966490 IP 162.159.128.62.https > so2mgr.57736: Flags [F.], seq 4938, ack 702, win 8, options [nop,nop,TS val 1059747089 ecr 3437629638], length 0
15:36:43.966578 IP so2mgr.57736 > 162.159.128.62.https: Flags [.], ack 4939, win 501, options [nop,nop,TS val 3437629713 ecr 1059747089], length 0
15:36:43.966856 IP so2mgr.57736 > 162.159.128.62.https: Flags [P.], seq 702:726, ack 4939, win 501, options [nop,nop,TS val 3437629713 ecr 1059747089], length 24
15:36:43.966882 IP so2mgr.57736 > 162.159.128.62.https: Flags [F.], seq 726, ack 4939, win 501, options [nop,nop,TS val 3437629713 ecr 1059747089], length 0
15:36:43.976342 IP 162.159.128.62.https > so2mgr.57736: Flags [.], ack 727, win 7, options [nop,nop,TS val 1059747099 ecr 3437629713], length 0
16:02:29.726706 IP so2mgr.38308 > 162.159.128.62.https: Flags [S], seq 2683733969, win 64240, options [mss 1460,sackOK,TS val 3439175473 ecr 0,nop,wscale 7], length 0
16:02:29.735812 IP 162.159.128.62.https > so2mgr.38308: Flags [S.], seq 3831185314, ack 2683733970, win 65160, options [mss 1400,sackOK,TS val 3076047752 ecr 3439175473,nop,wscale 13], length 0
16:02:29.735869 IP so2mgr.38308 > 162.159.128.62.https: Flags [.], ack 1, win 502, options [nop,nop,TS val 3439175482 ecr 3076047752], length 0
16:02:29.736112 IP so2mgr.38308 > 162.159.128.62.https: Flags [P.], seq 1:271, ack 1, win 502, options [nop,nop,TS val 3439175482 ecr 3076047752], length 270
16:02:29.744920 IP 162.159.128.62.https > so2mgr.38308: Flags [.], ack 271, win 7, options [nop,nop,TS val 3076047761 ecr 3439175482], length 0
16:02:29.750870 IP 162.159.128.62.https > so2mgr.38308: Flags [P.], seq 1:3941, ack 271, win 8, options [nop,nop,TS val 3076047767 ecr 3439175482], length 3940
16:02:29.750897 IP so2mgr.38308 > 162.159.128.62.https: Flags [.], ack 3941, win 489, options [nop,nop,TS val 3439175497 ecr 3076047767], length 0
16:02:29.752205 IP so2mgr.38308 > 162.159.128.62.https: Flags [P.], seq 271:335, ack 3941, win 501, options [nop,nop,TS val 3439175498 ecr 3076047767], length 64
16:02:29.752373 IP so2mgr.38308 > 162.159.128.62.https: Flags [P.], seq 335:698, ack 3941, win 501, options [nop,nop,TS val 3439175499 ecr 3076047767], length 363
16:02:29.760354 IP 162.159.128.62.https > so2mgr.38308: Flags [.], ack 698, win 8, options [nop,nop,TS val 3076047777 ecr 3439175498], length 0
16:02:29.792448 IP 162.159.128.62.https > so2mgr.38308: Flags [P.], seq 3941:4888, ack 698, win 8, options [nop,nop,TS val 3076047809 ecr 3439175498], length 947
16:02:29.792448 IP 162.159.128.62.https > so2mgr.38308: Flags [P.], seq 4888:4915, ack 698, win 8, options [nop,nop,TS val 3076047809 ecr 3439175498], length 27
16:02:29.792448 IP 162.159.128.62.https > so2mgr.38308: Flags [P.], seq 4915:4939, ack 698, win 8, options [nop,nop,TS val 3076047809 ecr 3439175498], length 24
16:02:29.792449 IP 162.159.128.62.https > so2mgr.38308: Flags [F.], seq 4939, ack 698, win 8, options [nop,nop,TS val 3076047809 ecr 3439175498], length 0
16:02:29.792535 IP so2mgr.38308 > 162.159.128.62.https: Flags [.], ack 4940, win 501, options [nop,nop,TS val 3439175539 ecr 3076047809], length 0
16:02:29.792677 IP so2mgr.38308 > 162.159.128.62.https: Flags [P.], seq 698:722, ack 4940, win 501, options [nop,nop,TS val 3439175539 ecr 3076047809], length 24
16:02:29.792703 IP so2mgr.38308 > 162.159.128.62.https: Flags [F.], seq 722, ack 4940, win 501, options [nop,nop,TS val 3439175539 ecr 3076047809], length 0
16:02:29.801321 IP 162.159.128.62.https > so2mgr.38308: Flags [.], ack 723, win 7, options [nop,nop,TS val 3076047818 ecr 3439175539], length 0

[cm-ops]

Check your Logstash log for any errors, /opt/so/log/logstash/logstash.log

[craigsmooth]

Nothing related to the integration. Just a periodic WARN log that appears to be unrelated.....

[2024-01-25T16:20:38,346][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["create", {:_id=>nil, :_index=>"logs-soc-so", :routing=>nil, :pipeline=>"common"}, {"event"=>{"dataset"=>"soc", "dataset_temp"=>"server", "action"=>"Host not found in process status metrics", "category"=>"host", "module"=>"soc"}, "log"=>{"level"=>"warn", "file"=>{"path"=>"/opt/so/log/soc/sensoroni-server.log"}, "offset"=>15982710}, "message"=>"{\"fields\":{\"host\":\"so2idh\",\"processStatus\":{\"so2mgr\":\"green\"}},\"level\":\"warn\",\"timestamp\":\"2024-01-25T16:20:27.833104616Z\",\"message\":\"Host not found in process status metrics\"}", "ecs"=>{"version"=>"8.0.0"}, "type"=>"redis-input", "data_stream"=>{"dataset"=>"soc", "namespace"=>"so", "type"=>"logs"}, "tags"=>["so-soc", "elastic-agent", "input-so2mgr", "beats_input_codec_plain_applied"], "input"=>{"type"=>"log"}, "container"=>{"id"=>"sensoroni-server.log"}, "metadata"=>{"pipeline"=>"common", "stream_id"=>"logfile-log.logs-1c4c9d95-e0e8-40a3-9bec-bd1fb5269329", "version"=>"8.10.4", "input_id"=>"logfile-logs-1c4c9d95-e0e8-40a3-9bec-bd1fb5269329", "type"=>"_doc", "raw_index"=>"logs-soc-so", "input"=>{"beats"=>{"host"=>{"ip"=>"172.30.1.102"}}}, "beat"=>"filebeat"}, "@timestamp"=>2024-01-25T16:20:35.347Z, "soc"=>{"fields"=>{"processStatus"=>{"so2mgr"=>"green"}, "host"=>"so2idh"}, "timestamp"=>"2024-01-25T16:20:27.833104616Z"}, "agent"=>{"version"=>"8.10.4", "id"=>"3cf0c3ee-0721-45f5-abe5-591875442cec", "type"=>"filebeat", "ephemeral_id"=>"0dd94517-125b-4adb-a088-78a53d555e3b", "name"=>"so2mgr"}, "@version"=>"1", "host"=>{"containerized"=>false, "hostname"=>"so2mgr", "id"=>"98c027f63dba407490598c999017eae7", "ip"=>["172.30.1.102", "fe80::20c:29ff:fe5a:5149", "172.17.0.1", "172.17.1.1"], "mac"=>["00-0C-29-5A-51-49", "02-42-7F-36-1E-44", "02-42-D3-61-FF-69", "12-9C-86-D6-4C-29", "16-DE-7B-3A-4C-31", "26-7A-36-40-34-D0", "46-5E-0D-B5-68-86", "6A-8C-AA-41-32-7B", "76-DF-85-A2-55-DD", "86-3B-2A-1E-AA-AF", "86-B9-23-F6-94-B7", "8A-6B-1F-01-61-6A", "8E-DE-17-B0-66-3A", "AA-D0-C5-A4-BE-CB", "B2-C2-62-91-01-10", "CE-37-97-6E-B3-BF", "F6-A1-F3-73-BA-73", "FE-68-20-DB-CF-4F", "FE-DB-F3-23-2A-3A"], "name"=>"so2mgr", "architecture"=>"x86_64", "os"=>{"kernel"=>"5.15.0-202.135.2.el9uek.x86_64", "version"=>"9.3", "type"=>"linux", "name"=>"Oracle Linux Server", "family"=>"redhat", "platform"=>"ol"}}, "elastic_agent"=>{"snapshot"=>false, "version"=>"8.10.4", "id"=>"3cf0c3ee-0721-45f5-abe5-591875442cec"}}], :response=>{"create"=>{"_index"=>".ds-logs-soc-so-2024.01.19-000001", "_id"=>"mOJsQY0BkwXL4Cw8YbhC", "status"=>400, "error"=>{"type"=>"document_parsing_exception", "reason"=>"[1:634] failed to parse field [soc.fields.processStatus.so2mgr] of type [long] in document with id 'mOJsQY0BkwXL4Cw8YbhC'. Preview of field's value: 'green'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"For input string: \"green\""}}}}}

[craigsmooth]

Just to add some closure here. It turns out the API key that I had was for version 1 and the Elastic integration uses version 2 of Recorded Future's API. Once I had them cut me a new key, all was well. Thank you @cm-ops for helping me troubleshoot this one. FYI, the elastic agent logs were extremely helpful:
/opt/Elastic/Agent