OTX Rate Limiting

[WavedirectTel]

Graylog version 6.06

Example error included - [OTXDataAdapter] OTX IPv4 request for key <5.135.238.149> failed: Response{protocol=http/1.1, code=429, message=Too Many Requests, url=https://otx.alienvault.com//api/v1/indicators/IPv4/5.135.238.149/general}

After about 30 minutes of just doing a check whether true/false on the OTX I get rate limited on their end. I increased the cache entries to 10k and many minutes, waited for another day to try again and same issue.

I also have an API key I tried to use as the individual key should allow more requests per hour, after inserting my key into the data adapter it would break java and it wouldn't function.

Maybe I'm not doing this right? How would I go about getting an increase in allowance to use the API? Especially since I will be increase the load by about 8x once this works.

It would be useful if you could document these sorts of caveats as to what our limits are with the free plugin.

Also on a side note it applies a "False" to private ip's. So I had to add

has_field("src_ip") &&
!cidr_match("10.0.0.0/8", to_ip($message.src_ip)) &&
!cidr_match("172.16.0.0/12", to_ip($message.src_ip)) &&
!cidr_match("192.168.0.0/16", to_ip($message.src_ip)) &&
!cidr_match("127.0.0.0/8", to_ip($message.src_ip)) &&
!cidr_match("169.254.0.0/16", to_ip($message.src_ip)) &&
!contains(to_string($message.src_ip), ":") && // Exclude IPv6
to_string($message.src_ip) != "0.0.0.0" &&
to_string($message.src_ip) != "255.255.255.255" // Exclude specific IPs

[logopk]

Consequence of this is that I get thousands of processing errors.

Is there a way to catch this error and skip the lookup?