VT quota management

[MarcOverIP]

Im getting Virus Total quota issues although it seems that I still have space left in my quota. Output from daemon.log reads:

2022-12-05 12:26:26,399 - DEBUG - alarm_filehash - module.py - check_hashes -- Results from VirusTotal: {'02bc35ae711eee0977c0cc099954ecad': {'result': 'skipped, quota reached'},
<here follows a very long list of hashes>

My hypothesis is that although the quota says 240 left for now, the actual list of items to check exceeds 240. I checked this manually for one call and indeed it wanted to check 386 hashes in one go. VT will see the too big result come in and therefor nothing will be checked. This is an endless loop as the amount of to be checked items will only grow.

It would be ideal if alarm_filehash module would actually parse the output of VT's report on available quota, and tune the amount of to be checked hashed accordingly.

[fastlorenzo]

This has normally the same behavior has HA module, it first checks the remaining quota:

RedELK/elkserver/docker/redelk-base/redelkinstalldata/scripts/modules/alarm_filehash/ioc_vt.py

Lines 27 to 63 in 5f50fc6

	def get_remaining_quota(self):
	"""Returns the number of hashes that could be queried within this run"""
	url = f"https://www.virustotal.com/api/v3/users/{self.api_key}/overall_quotas"
	headers = {"Accept": "application/json", "x-apikey": self.api_key}
	# Get the quotas, if response code != 200, return 0 so we don't query further
	response = requests.get(url, headers=headers)
	if response.status_code == 200:
	json_response = response.json()
	else:
	self.logger.warning(
	"Error retrieving VT Quota (HTTP Status code: %d)", response.status_code
	)
	return 0
	# Extract the hourly, daily and monthly remaining quotas
	remaining_hourly = get_value(
	"data.api_requests_hourly.user.allowed", json_response, 0
	) - get_value("data.api_requests_hourly.user.used", json_response, 0)
	remaining_daily = get_value(
	"data.api_requests_daily.user.allowed", json_response, 0
	) - get_value("data.api_requests_daily.user.used", json_response, 0)
	remaining_monthly = get_value(
	"data.api_requests_monthly.user.allowed", json_response, 0
	) - get_value("data.api_requests_monthly.user.used", json_response, 0)
	self.logger.debug(
	"Remaining quotas: hourly(%d) / daily(%d) / monthly(%d)",
	remaining_hourly,
	remaining_daily,
	remaining_monthly,
	)
	# Get the smallest one and return it
	remaining_min = min(remaining_hourly, remaining_daily, remaining_monthly)
	return remaining_min

Then checks each hash one by one until the quota is reached, the remainder should be done on next run when the quota is available again:

RedELK/elkserver/docker/redelk-base/redelkinstalldata/scripts/modules/alarm_filehash/ioc_vt.py

Lines 91 to 100 in 5f50fc6

	# Get the remaining quota for this run
	remaining_quota = self.get_remaining_quota()
	vt_results = {}
	# Query VT API for file hashes
	count = 0
	for md5 in hash_list:
	if count < remaining_quota:
	# Within quota, let's check the file hash with VT
	vt_result = self.get_vt_file_results(md5)

It's normally not a "all or nothing", it should be a smart queue 😄