Merging two sources and perform an action

[nathanle]

I have two sources:

One is a number of S3 buckets allowed in a datacenter.
The second source is the number of buckets used.

I have reformatted the log lines so that they both have a common key which is "datacenter".

I want to merge these two sources and subtract the number of buckets used from the number of buckets entitled.

{"account":"123456","bucket_limit":"1000","byte_limit":"5","datacenter":"us-east-1","objects_limit":"50"}
{"count":2,"datacenter":"us-east-1"}

So all I want is to subtract 2 from 1000 then I will transform that into a metric and send it to a sink.

Is this possible?

[nathanle]

I was able to do what I needed to do with Lua, but based on:

"The lua transform is ~60% slower than the remap transform; we recommend that you use the remap transform whenever possible. The lua transform is designed solely for edge cases not covered by the remap transform and not as a go-to option. If the remap transform doesn’t cover your use case, please open an issue and let us know."

Should I open an issue or should I have been able to do this in VRL the same way?

reductions_by_dc:
   type: lua
   inputs:
     - transform_status_monitor
     - obj_transform_id_test
   version: "2"
   source: |-
       state = state or {}
       function process(event, emit)
         local datacenter = event.log.datacenter
         if event.log.bucket_limit then
           state[datacenter] = tonumber(event.log.bucket_limit)
         elseif event.log.count and state[datacenter] then
           local available = state[datacenter] - tonumber(event.log.count)
           event.log.available_buckets = available
           emit(event)
         end
       end
   hooks:
     process: process

  Results:

      {"available_buckets":998,"count":2,"datacenter":"us-east-1"}
      {"available_buckets":991,"count":9,"datacenter":"us-ord-1"}