We still need the Lua transform

[hunsbea]

This is not an issue per se, but https://vector.dev/docs/reference/configuration/transforms/lua/ has a warning banner encouraging me to tell you if any Lua use cases are not covered by the VRL remap transform.

We are doing something like this, to run arbitrary system commands only once per log file to enrich log metadata with the owner of the log file, and the process writing to the log file. We plan to extend this further to get more specific metadata related to our environment with additional commands.

transforms:
  enrich_logs:
    type: lua
    inputs:
      - app_logs
    version: "2"
    # This will only run the expensive system commands the first time we see a log file, then we'll use the cached results
    source: |
      function get_file_owner(filepath)
        local handle = io.popen("stat -c '%U' '" .. filepath .. "' 2>/dev/null")
        local user = handle and handle:read("*a"):gsub("%s+", "") or "unknown"
        if handle then handle:close() end
        return user
      end

      function get_process_name(filepath)
        local handle = io.popen("fuser '" .. filepath .. "' 2>/dev/null | xargs -r ps -o comm= -p | grep -v vector | head -1")
        local result = handle and handle:read("*a"):gsub("%s+", "") or ""
        if handle then handle:close() end
        return result
      end

      function init(emit)
        file_metadata_cache = {}
      end

      function process(event, emit)
        local filepath = event.log.file
        if filepath and not file_metadata_cache[filepath] then
          local user = get_file_owner(filepath)
          local process_name = get_process_name(filepath)
          file_metadata_cache[filepath] = {
            process_name = process_name,
            owner_user = user
          }
        end

        if filepath and file_metadata_cache[filepath] then
          local metadata = file_metadata_cache[filepath]
          event.log.process.name = metadata.process_name
          event.log.file.owner_user = metadata.owner_user
        end

        emit(event)
      end
    hooks:
      init: "init"
      process: "process"

If I could run arbitrary system commands and store a global cache state using VRL, I would happily switch to that. I don't need any immediate assistance here, but I would just say please don't deprecate the Lua transform without allowing this kind of thing in VRL.

[pront]

Hi @hunsbea,

If I could run arbitrary system commands and store a global cache state using VRL, I would happily switch to that.

Thank you for sharing this with us. Vector now offers a memory enrichment table which can be used a cache. But I don't think you run arbitrary commands in VRL for various reasons: https://github.com/vectordotdev/vrl/blob/main/DESIGN.md#performance-warning

I don't need any immediate assistance here, but I would just say please don't deprecate the Lua transform without allowing this kind of thing in VRL.

Rest assured, we will not deprecate the Lua transform. The warning is there to encourage folks to consider using Remap if possible.