Part of the data in the filter plugin is invalid

[XTUxiongda]

Describe the bug

I used the record_transformer plug-in to intercept the super long field, but found that only part of the data took effect

To Reproduce

must appear

Expected behavior

I want fliter to act on all tag matching data

Your Environment

- Fluentd version: 1.1.3

Your Configuration

<source>
@type tail
path "/logs/#{ENV['APPNAME']}/*.log"
pos_file "/var/log/#{ENV['APPNAME']}.log.pos"
format json
tag jgt-dev2
time_key @timestamp
</source>

<filter jgt-dev2*>
  @type record_transformer
  enable_ruby true
  <record>
    message ${record["message"][0, 30]}
  </record>
</filter>

<match jgt-dev2*>
@type elasticsearch
logstash_format true
logstash_prefix "jgt-dev2-#{ENV['NAME']}"
logstash_dateformat %Y.%m.%d
host elasticsearch
port 9200
flush_interval 1s
utc_index false
</match>

Your Error Log

none

Additional context

The tag of all my data is jgt-dev2, but only some of the data whose length exceeds 30 have been intercepted, and some have not been intercepted. I want to know whether my usage is wrong or fluentd is just like this

[daipom]

Please follow the contributing guidelines.

With this information, we are unable to reproduce this issue and cannot judge if it is a bug or not.
If you're unsure whether it's a bug or not, please start by discussing it here.

[daipom]

If the tag is jgt-dev2, the value of message field should be sliced.

Could you please share some sample logs to reproduce this?

[XTUxiongda]

{"appName":"basic-account","serverIP":"10.11.182.560000","traceId":"6256d6300bd04f5d9dae6b6a0e858ff4.1479.16902490400237953","createTime":"2023-07-25 09:37:20 027","level":"DEBUG","thread":"http-nio-8080-exec-10","message":"==> Preparing: SELECT COUNT( * ) FROM operator_sync_subscriber_record WHERE (sync_flag IN (?,?) AND delete_flag = ? AND create_time > now() - interval '24 hour')"} {"appName":"basic-account","serverIP":"10.11.182.560000","traceId":"6256d6300bd04f5d9dae6b6a0e858ff4.1479.16902490400237953","createTime":"2023-07-25 09:37:20 027","level":"DEBUG","thread":"http-nio-8080-exec-10","message":"==> Parameters: 0(String), 3(String), 0(String)"} {"appName":"basic-account","serverIP":"10.11.182.560000","traceId":"6256d6300bd04f5d9dae6b6a0e858ff4.1479.16902490400237953","createTime":"2023-07-25 09:37:20 031","level":"DEBUG","thread":"http-nio-8080-exec-10","message":"<== Total: 1"} {"appName":"basic-account","serverIP":"10.11.182.560000","traceId":"6256d6300bd04f5d9dae6b6a0e858ff4.1479.16902490420277959","createTime":"2023-07-25 09:37:22 031","level":"DEBUG","thread":"http-nio-8080-exec-9","message":"==> Preparing: SELECT COUNT( * ) FROM operator_sync_subscriber_record WHERE (sync_flag IN (?,?) AND delete_flag = ? AND create_time > now() - interval '24 hour')"} {"appName":"basic-account","serverIP":"10.11.182.560000","traceId":"6256d6300bd04f5d9dae6b6a0e858ff4.1479.16902490420277959","createTime":"2023-07-25 09:37:22 031","level":"DEBUG","thread":"http-nio-8080-exec-9","message":"==> Parameters: 0(String), 3(String), 0(String)"} {"appName":"basic-account","serverIP":"10.11.182.560000","traceId":"6256d6300bd04f5d9dae6b6a0e858ff4.1479.16902490420277959","createTime":"2023-07-25 09:37:22 035","level":"DEBUG","thread":"http-nio-8080-exec-9","message":"<== Total: 1"} {"appName":"basic-account","serverIP":"10.11.182.560000","traceId":"6256d6300bd04f5d9dae6b6a0e858ff4.1479.16902490440237965","createTime":"2023-07-25 09:37:24 027","level":"DEBUG","thread":"http-nio-8080-exec-2","message":"==> Preparing: SELECT COUNT( * ) FROM operator_sync_subscriber_record WHERE (sync_flag IN (?,?) AND delete_flag = ? AND create_time > now() - interval '24 hour')"} {"appName":"basic-account","serverIP":"10.11.182.560000","traceId":"6256d6300bd04f5d9dae6b6a0e858ff4.1479.16902490440237965","createTime":"2023-07-25 09:37:24 027","level":"DEBUG","thread":"http-nio-8080-exec-2","message":"==> Parameters: 0(String), 3(String), 0(String)"} {"appName":"basic-account","serverIP":"10.11.182.560000","traceId":"6256d6300bd04f5d9dae6b6a0e858ff4.1479.16902490440237965","createTime":"2023-07-25 09:37:24 031","level":"DEBUG","thread":"http-nio-8080-exec-2","message":"<== Total: 1"} {"appName":"basic-account","serverIP":"10.11.182.560000","traceId":"bb79388354524669b78e174c57c8417b.128.16902490447046891","createTime":"2023-07-25 09:37:24 955","level":"INFO","thread":"http-nio-8080-exec-6","message":"request: url=/basic/account/query/establish, header=[userCode='null', userName='null', operatorId='null', operatorCode='null', orgId='null', orgCode='null', fofundNo='null', custNo='null', establishAccoNo='null', tradeAcco='null', menuName='null', frontFlag='null', ip='null', httpFlag='false', loginType='null', trialId='null', token='null'], clientId=10.11.122.47, params=[FofundRequest(fofundDetail=[FofundDetailRequest(orgCode=00006, fofundNo=1000000199, accountNo=null)])], tid=3ae486236ce54e4e8c8a3e2709b40df9, classMethod=com.fofund.basic.account.controller.AccoInfoController.getEstablishInfo"} {"appName":"basic-account","serverIP":"10.11.182.560000","traceId":"bb79388354524669b78e174c57c8417b.128.16902490447046891","createTime":"2023-07-25 09:37:24 956","level":"DEBUG","thread":"http-nio-8080-exec-6","message":"==> Preparing: select cust.fofund_name as fofundName, cust.fofund_short_name as fofundShortName, cea.id, cea.fofund_no, cea.establish_acco_no, cea.third_acco, cea.source, cea.creator, cea.create_time, cea.modifier, cea.last_modify_time, cea.delete_flag, cea.creator_name, cea.modifier_name, cea.channel, oc.org_code, cea.invest_portfolio_name from customer_establish_account cea join organization_customer oc on cea.fofund_no = oc.fofund_no LEFT JOIN customer cust on cust.fofund_no=cea.fofund_no AND cust.delete_flag='0' where cea.delete_flag = '0' and oc.delete_flag = '0' and oc.org_code in ( ? ) and cea.fofund_no in ( ? )"} {"appName":"basic-account","serverIP":"10.11.182.560000","traceId":"bb79388354524669b78e174c57c8417b.128.16902490447046891","createTime":"2023-07-25 09:37:24 956","level":"DEBUG","thread":"http-nio-8080-exec-6","message":"==> Parameters: 00006(String), 1000000199(String)"} {"appName":"basic-account","serverIP":"10.11.182.560000","traceId":"bb79388354524669b78e174c57c8417b.128.16902490447046891","createTime":"2023-07-25 09:37:24 959","level":"DEBUG","thread":"http-nio-8080-exec-6","message":"<== Total: 1"} {"appName":"basic-account","serverIP":"10.11.182.560000","traceId":"bb79388354524669b78e174c57c8417b.128.16902490447046891","createTime":"2023-07-25 09:37:24 959","level":"INFO","thread":"http-nio-8080-exec-6","message":"response: code=00000000, msg=success, data=null, tid=3ae486236ce54e4e8c8a3e2709b40df9, time=4ms"} {"appName":"basic-account","serverIP":"10.11.182.560000","traceId":"6256d6300bd04f5d9dae6b6a0e858ff4.1479.16902490460217971","createTime":"2023-07-25 09:37:26 026","level":"DEBUG","thread":"http-nio-8080-exec-1","message":"==> Preparing: SELECT COUNT( * ) FROM operator_sync_subscriber_record WHERE (sync_flag IN (?,?) AND delete_flag = ? AND create_time > now() - interval '24 hour')"} {"appName":"basic-account","serverIP":"10.11.182.560000","traceId":"6256d6300bd04f5d9dae6b6a0e858ff4.1479.16902490460217971","createTime":"2023-07-25 09:37:26 026","level":"DEBUG","thread":"http-nio-8080-exec-1","message":"==> Parameters: 0(String), 3(String), 0(String)"} {"appName":"basic-account","serverIP":"10.11.182.560000","traceId":"6256d6300bd04f5d9dae6b6a0e858ff4.1479.16902490460217971","createTime":"2023-07-25 09:37:26 029","level":"DEBUG","thread":"http-nio-8080-exec-1","message":"<== Total: 1"}

but the result some is fail
{ "_index": "jgt-dev2-basic-account-2023.07.25", "_type": "_doc", "_id": "_oboiokBq_CZg9mO1TM7", "_version": 1, "_seq_no": 252578, "_primary_term": 1, "found": true, "_source": { "appName": "basic-account", "serverIP": "10.11.65.930000", "traceId": "Ignored_Trace", "createTime": "2023-07-25 10:37:40 067", "level": "INFO", "thread": "http-nio-8080-exec-3", "message": "request: url=/basic/account/open-register-info/status-id-search, header=[userCode='null', userName='null', operatorId='null', operatorCode='null', orgId='null', orgCode='null', fofundNo='null', custNo='null', establishAccoNo='null', tradeAcco='null', menuName='null', frontFlag='null', ip='null', httpFlag='false', loginType='null', trialId='null', token='null'], clientId=10.11.44.188, params=[StatusIdSearchRequest(lastModifyTime=20230725000000, status=2, auditStatus=null, registerIdList=null)], tid=1a50c02b793249cba67325f1dd3275b3, classMethod=com.fofund.basic.account.controller.OpenRegisterController.queryRegisterByStatusId", "@timestamp": "2023-07-25T10:37:40.067495559+08:00" } }

and some is success
{ "_index": "jgt-dev2-basic-account-2023.07.25", "_type": "_doc", "_id": "2gboiokB70AokzY71KSI", "_version": 1, "_seq_no": 252573, "_primary_term": 1, "found": true, "_source": { "appName": "basic-account", "serverIP": "10.11.182.560000", "traceId": "6256d6300bd04f5d9dae6b6a0e858ff4.1479.16902526600278681", "createTime": "2023-07-25 10:37:40 032", "level": "DEBUG", "thread": "http-nio-8080-exec-9", "message": "==> Preparing: SELECT COUNT( ", "@timestamp": "2023-07-25T10:37:40.032488189+08:00" } }

[XTUxiongda]

Is this related to the amount of logs

[daipom]

Thanks!
In my environment, I confirmed the latest Fluentd v1.16.2 could slice all message.

There are 2 possible causes:

• There are too many logs, and Fluentd is not working well.
• There is a bug in the old version.

Fluentd v1.1.3 is very old.
Could you try the latest version v1.16.2?