About logstash

[nw-engineer]

There was a description that Logstash can be used with the latest version of manticoresearch.

Experimental: you can now execute Elasticsearch-compatible insert and replace JSON queries which enables using Manticore with tools like Logstash (version < 7.13), Filebeat and other tools from the Beats family. Enabled by default. You can disable it using SET GLOBAL ES_COMPAT=off.

What should I use for the logstash output plugin?
※If possible, it would be helpful if you could provide a sample definition that has been confirmed to work from input to output.

Also, will logstash version 7.13 or higher work without problems?
Thank you.

[Nick-S-2018]

You should use the 'elasticsearch' plugin with the following configuration:

output {
    elasticsearch {
 	index => "MANTICORE_TABLE_NAME"
        hosts => ["http://${MANTICORE_HOST}:${MANTICORE_PORT}"]
        ilm_enabled => false
        manage_template => false
    }
}

Note that now you must create your Manticore table before using Logstash.

Manticore doesn't support Logstash versions >= 7.13 yet, but we're working on this.
It'll also require the 'manticore-extra' package to work with the higher versions and to enable automatic table creation.

[nw-engineer]

@sanikolaev
Thanks for the answer.

That error with "evenement" should have been already fixed in the latest dev version.

Hmm. So why did it fail to boot?
The version of buddy currently installed is as follows.
manticore-buddy-0.3.5_23021410.ed09f8f-1.noarch

[sanikolaev]

Please upgrade your dev packages and try again. That was just a bug. Hopefully fixed now.

[nw-engineer]

I tried upgrading the target package.

[root@manticoresearch ~]# rpm -qa | grep manticore-buddy
manticore-buddy-0.3.5_23021808.ed09f8f-1.noarch
[root@manticoresearch ~]#
[root@manticoresearch ~]#
[root@manticoresearch ~]# tail -f /var/log/manticore/searchd.log
[Sat Feb 18 18:31:43.210 2023] [1384] listening on 10.2.0.17:9312 for sphinx and http(s)
[Sat Feb 18 18:31:43.210 2023] [1384] listening on 10.2.0.17:9306 for mysql
[Sat Feb 18 18:31:43.210 2023] [1384] listening on 10.2.0.17:9308 for sphinx and http(s)
[Sat Feb 18 18:31:43.559 2023] [1384] expression stack for creation is 208. Consider to add env MANTICORE_KNOWN_CREATE_SIZE=208 to store this value persistent for this binary
[Sat Feb 18 18:31:43.560 2023] [1384] expression stack for eval/deletion is 32. Consider to add env MANTICORE_KNOWN_EXPR_SIZE=32 to store this value persistent for this binary
[Sat Feb 18 18:31:43.567 2023] [1384] filter stack delta is 224. Consider to add env MANTICORE_KNOWN_FILTER_SIZE=224 to store this value persistent for this binary
[Sat Feb 18 18:31:43.611 2023] [1388] prereading 1 tables
[Sat Feb 18 18:31:43.611 2023] [1388] preread 1 tables in 0.000 sec
[Sat Feb 18 18:31:45.257 2023] [1384] [BUDDY] started '/usr/share/manticore/modules/manticore-buddy --listen=http://10.2.0.17:9312  --threads=8' at http://127.0.0.1:44909
[Sat Feb 18 18:31:45.257 2023] [1384] accepting connections

No more errors on the manticore side.
However, logstash now gives the following error:

[root@logstash001 logstash]# /usr/share/logstash/bin/logstash -f conf.d/file.conf
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2023-02-18 18:35:44.541 [main] runner - Starting Logstash {"logstash.version"=>"7.12.0", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.10+9 on 11.0.10+9 +indy +jit [linux-x86_64]"}
[WARN ] 2023-02-18 18:35:44.683 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2023-02-18 18:35:45.003 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2023-02-18 18:35:45.451 [Converge PipelineAction::Create<main>] Reflections - Reflections took 17 ms to scan 1 urls, producing 23 keys and 47 values
[WARN ] 2023-02-18 18:35:45.639 [Converge PipelineAction::Create<main>] elasticsearch - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2023-02-18 18:35:45.826 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://10.2.0.17:9308/]}}
[WARN ] 2023-02-18 18:35:45.897 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://10.2.0.17:9308/"}
[INFO ] 2023-02-18 18:35:45.980 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>7}
[WARN ] 2023-02-18 18:35:45.981 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[ERROR] 2023-02-18 18:35:45.991 [[main]-pipeline-manager] javapipeline - Pipeline error {:pipeline_id=>"main", :exception=>#<TypeError: no implicit conversion of String into Integer>, :backtrace=>["org/jruby/RubyArray.java:952:in `fetch'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/license_checker.rb:39:in `valid_es_license?'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/license_checker.rb:17:in `appropriate_license?'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:285:in `block in healthcheck!'", "org/jruby/ext/thread/Mutex.java:164:in `synchronize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:274:in `block in healthcheck!'", "org/jruby/RubyHash.java:1415:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:265:in `healthcheck!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:367:in `update_urls'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:83:in `update_initial_urls'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:77:in `start'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/http_client.rb:303:in `build_pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/http_client.rb:64:in `initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:106:in `create_http_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:102:in `build'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:34:in `build_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch.rb:270:in `register'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:131:in `register'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:68:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:228:in `block in register_plugins'", "org/jruby/RubyArray.java:1809:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:227:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:585:in `maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:240:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:185:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:137:in `block in start'"], "pipeline.sources"=>["/etc/logstash/conf.d/file.conf"], :thread=>"#<Thread:0x3fd0eae4 run>"}
[INFO ] 2023-02-18 18:35:45.993 [[main]-pipeline-manager] javapipeline - Pipeline terminated {"pipeline.id"=>"main"}
[ERROR] 2023-02-18 18:35:46.007 [Converge PipelineAction::Create<main>] agent - Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
[INFO ] 2023-02-18 18:35:46.043 [LogStash::Runner] runner - Logstash shut down.

I also tried ubuntu (22.04 LTM), but the situation was the same. . .
Thank you for helping me so many times.

I kindly thank you.

[nw-engineer]

When I changed Logstash as follows, the error no longer appeared.

old) logstash-7.12.0-1.x86_64
new) logstash-oss-7.12.1-1.x86_64

[root@logstash001 logstash]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/file.conf
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2023-02-18 23:08:27.525 [main] runner - Starting Logstash {"logstash.version"=>"7.12.1", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.10+9 on 11.0.10+9 +indy +jit [linux-x86_64]"}
[WARN ] 2023-02-18 23:08:27.700 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2023-02-18 23:08:28.008 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2023-02-18 23:08:28.457 [Converge PipelineAction::Create<main>] Reflections - Reflections took 19 ms to scan 1 urls, producing 23 keys and 47 values
[WARN ] 2023-02-18 23:08:28.679 [Converge PipelineAction::Create<main>] elasticsearch - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2023-02-18 23:08:28.868 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://10.2.0.17:9308/]}}
[WARN ] 2023-02-18 23:08:28.937 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://10.2.0.17:9308/"}
[INFO ] 2023-02-18 23:08:29.018 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>7}
[WARN ] 2023-02-18 23:08:29.019 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[INFO ] 2023-02-18 23:08:29.029 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://10.2.0.17:9308"]}
[INFO ] 2023-02-18 23:08:29.069 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/etc/logstash/conf.d/file.conf"], :thread=>"#<Thread:0x2d13ce95 run>"}
[INFO ] 2023-02-18 23:08:29.444 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>0.37}
[INFO ] 2023-02-18 23:08:29.553 [[main]-pipeline-manager] file - No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/usr/share/logstash/data/plugins/inputs/file/.sincedb_15940cad53dd1d99808eeaecd6f6ad3f", :path=>["/var/log/httpd/access_log"]}
[INFO ] 2023-02-18 23:08:29.560 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2023-02-18 23:08:29.588 [[main]<file] observingtail - START, creating Discoverer, Watch with file and sincedb collections
[INFO ] 2023-02-18 23:08:29.595 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}

However, an error occurs when data transfer is started.

[ERROR] 2023-02-18 23:09:35.969 [[main]>worker0] elasticsearch - Encountered a retryable error. Will Retry with exponential backoff  {:code=>500, :url=>"http://10.2.0.17:9308/_bulk", :content_length=>407}
[ERROR] 2023-02-18 23:09:35.973 [[main]>worker2] elasticsearch - Encountered a retryable error. Will Retry with exponential backoff  {:code=>500, :url=>"http://10.2.0.17:9308/_bulk", :content_length=>814}
[ERROR] 2023-02-18 23:09:35.976 [[main]>worker3] elasticsearch - Encountered a retryable error. Will Retry with exponential backoff  {:code=>500, :url=>"http://10.2.0.17:9308/_bulk", :content_length=>326}
[ERROR] 2023-02-18 23:09:35.977 [[main]>worker1] elasticsearch - Encountered a retryable error. Will Retry with exponential backoff  {:code=>500, :url=>"http://10.2.0.17:9308/_bulk", :content_length=>409}

Attached is the packet information when the internal error is issued.（And the log when "--logdebugvv" is enabled）
result_of_dump.txt
logdegvv.txt

Also, as far as I can confirm with "show variables", there is no item for "auto_schema", is this correct?

MySQL [(none)]> show variables;
+--------------------------+----------+
| Variable_name            | Value    |
+--------------------------+----------+
| autocommit               | 1        |
| auto_optimize            | 1        |
| optimize_cutoff          | 16       |
| collation_connection     | libc_ci  |
| query_log_format         | sphinxql |
| session_read_only        | 0        |
| log_level                | info     |
| max_allowed_packet       | 8388608  |
| character_set_client     | utf8     |
| character_set_connection | utf8     |
| grouping_in_utc          | 0        |
| last_insert_id           |          |
| pseudo_sharding          | 1        |
| secondary_indexes        | 1        |
| accurate_aggregation     | 0        |
| threads_ex_effective     |          |
| threads_ex               |          |
+--------------------------+----------+
17 rows in set (0.000 sec)

MySQL [(none)]> exit

[nw-engineer]

I checked if it is possible to insert without creating a table with the mysql command instead of Logstash.

[root@manticoresearch testindex001]# mysql -P 9306 -u root -h 10.2.0.17
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 68
Server version: 6.0.3 4f5acaf5d@230217 dev (columnar 2.0.1 37e6c01@230209) (secondary 2.0.1 37e6c01@230209) git branch HEAD (no branch)

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MySQL [(none)]> drop table testindex003;
Query OK, 0 rows affected (0.001 sec)

MySQL [(none)]>
MySQL [(none)]>
MySQL [(none)]>
MySQL [(none)]> show tables;
+--------------+------+
| Index        | Type |
+--------------+------+
| testindex001 | rt   |
+--------------+------+
1 row in set (0.000 sec)

MySQL [(none)]> insert into testindex003 (message, path, host) values('10.2.0.191 - - [19/Feb/2023:01:19:57 +0900] "GET /noindex/css/fonts/Light/OpenSans-Light.woff HTTP/1.1" 404 241 "http://10.2.0.15/noindex/css/open-sans.css" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36"', '/var/log/httpd/access_log', 'logstash001');
Query OK, 1 row affected (0.005 sec)

MySQL [(none)]>
MySQL [(none)]> show tables;
+--------------+------+
| Index        | Type |
+--------------+------+
| testindex001 | rt   |
| testindex003 | rt   |
+--------------+------+
2 rows in set (0.000 sec)

MySQL [(none)]> select * from testindex003;
+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------+-------------+
| id                  | message                                                                                                                                                                                                                                                                        | path                      | host        |
+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------+-------------+
| 4973988392770469897 | 10.2.0.191 - - [19/Feb/2023:01:19:57  0900] "GET /noindex/css/fonts/Light/OpenSans-Light.woff HTTP/1.1" 404 241 "http://10.2.0.15/noindex/css/open-sans.css" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36" | /var/log/httpd/access_log | logstash001 |
+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------+-------------+
1 row in set (0.000 sec)

MySQL [(none)]>
MySQL [(none)]> show create table testindex003;
+--------------+-----------------------------------------------------------------------------+
| Table        | Create Table                                                                |
+--------------+-----------------------------------------------------------------------------+
| testindex003 | CREATE TABLE testindex003 (
id bigint,
message text,
path text,
host text
) |
+--------------+-----------------------------------------------------------------------------+
1 row in set (0.000 sec)

MySQL [(none)]>

Inserted without problems. . . . (increasingly mysterious)

[tomatolog]

for me

curl -sX POST "localhost:9312/_bulk" -H "content-type: application/x-ndjson" --data-binary @lstash-bulk2.log

with the following content lstash-bulk2.log failed with the error message

{"items":[{"index":{"_index":"testindex001","_type":"doc","_id":"0","status":400,"error":{"type":"mapper_parsing_exception","reason":"table 'testindex001' absent, or does not support INSERT"}}},{"index":{"_index":"testindex001","_type":"doc","_id":"0","status":400,"error":{"type":"mapper_parsing_exception","reason":"table 'testindex001' absent, or does not support INSERT"}}}],"errors":true,"took":1}

[nw-engineer]

@tomatolog

I got the same result.

{"items":[{"index":{"_index":"testindex005","_type":"doc","_id":"0","status":400,"error":{"type":"mapper_parsing_exception","reason":"table 'testindex005' absent, or does not support INSERT"}}},{"index":{"_index":"testindex005","_type":"doc","_id":"0","status":400,"error":{"type":"mapper_parsing_exception","reason":"table 'testindex005' absent, or does not support INSERT"}}}],"errors":true,"took":1}

[Nick-S-2018]

Probably, there was some misunderstanding. In my first comment I meant we are now working on both the support of higher Logstash versions and auto table creation. Currently, installing manticore-extra won't help if you work with Logstash. This feature is to appear in the next update. It'll also be available in next dev packages.

[nw-engineer]

It'll also be available in next dev packages.

Thank you for contacting us.
We are looking forward to the next update.

[nw-engineer]

Hello.

I used the latest development package.
I was able to confirm that data can be inserted from Logstash without having to define a schema in advance.

However, when I tried to send Paloalto Logs using the Logstash Filter file, I got an error.

[INFO ] 2023-07-02 18:55:07.383 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
Oct 09 10:19:15 SumPunFw07.sumotest.com 1,2019/10/09 10:19:15,001234567890002,TRAFFIC,drop,2304,2019/10/09 10:19:15,209.118.103.150,160.177.222.249,0.0.0.0,0.0.0.0,InternalServer,,,not-applicable,vsys1,inside,z1-FW-Transit,ethernet1/2,,All traffic,2019/10/09 10:19:15,0,1,63712,443,0,0,0x0,udp,deny,60,60,0,1,2019/10/09 10:19:15,0,any,0,0123456789,0x0,Netherlands,10.0.0.0-10.255.255.255,0,1,0,policy-deny,0,0,0,0,,SumPunFw07,from-policy,,,0,,0,,N/A,0,0,0,0,1202585d-b4d5-5b4c-aaa2-d80d77ba456e,0
{
                     "LogAction" => "All traffic",
             "SCTPAssociationID" => "0",
               "DestinationZone" => "z1-FW-Transit",
                     "BytesSent" => "60",
                       "Packets" => "1",
                    "SourceZone" => "inside",
                         "Bytes" => "60",
               "ParentStartTime" => nil,
                    "SCTPChunks" => "0",
               "ParentSessionID" => "0",
                      "SourceIP" => "209.118.103.150",
                    "TunnelType" => "N/A",
                "SequenceNumber" => "0123456789",
                   "RepeatCount" => "1",
    "DeviceGroupHierarchyLevel1" => "0",
                   "ActionFlags" => "0x0",
           "DestinationLocation" => "10.0.0.0-10.255.255.255",
               "PacketsReceived" => "0",
                    "@timestamp" => 2023-07-02T09:55:12.044Z,
                    "SourcePort" => "63712",
                         "Flags" => "0x0",
             "DestinationVMUUID" => nil,
              "NATDestinationIP" => "0.0.0.0",
                      "Protocol" => "udp",
                   "NATSourceIP" => "0.0.0.0",
                   "PacketsSent" => "1",
                  "SourceVMUUID" => nil,
                 "TunnelID_IMSI" => "0",
    "DeviceGroupHierarchyLevel2" => "0",
              "SessionEndReason" => "policy-deny",
                          "Type" => "TRAFFIC",
              "InboundInterface" => "ethernet1/2",
                    "FUTURE_USE" => "0",
                      "column67" => "0",
                   "ElapsedTime" => "0",
                 "DestinationIP" => "160.177.222.249",
             "OutboundInterface" => nil,
                          "host" => "manticore-logstash",
                      "@version" => "1",
                    "DeviceName" => "SumPunFw07",
                   "Application" => "not-applicable",
                    "SourceUser" => nil,
            "NATDestinationPort" => "0",
               "DestinationUser" => nil,
                        "Action" => "deny",
                 "BytesReceived" => "0",
                 "NATSourcePort" => "0",
                     "StartTime" => "2019/10/09 10:19:15",
    "DeviceGroupHierarchyLevel3" => "0",
                  "SerialNumber" => "001234567890002",
                      "RuleName" => "InternalServer",
                 "VirtualSystem" => "vsys1",
                     "SessionID" => "0",
               "DestinationPort" => "443",
    "DeviceGroupHierarchyLevel4" => "0",
               "MonitorTag_IMEI" => nil,
                      "column66" => "1202585d-b4d5-5b4c-aaa2-d80d77ba456e",
                      "Category" => "any",
            "SCTPChunksReceived" => "0",
                 "GeneratedTime" => "2019/10/09 10:19:15",
                  "ActionSource" => "from-policy",
                "SCTPChunksSent" => "0",
            "Threat_ContentType" => "drop",
                   "ReceiveTime" => "2019/10/09 10:19:15",
                "SourceLocation" => "Netherlands",
             "VirtualSystemName" => nil
}
[ERROR] 2023-07-02 18:55:12.289 [[main]>worker0] elasticsearch - An unknown error occurred sending a bulk request to Elasticsearch. We will retry indefinitely {:error_message=>"no implicit conversion of String into Integer", :error_class=>"TypeError", :backtrace=>["org/jruby/RubyArray.java:1487:in `[]'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.6-java/lib/logstash/outputs/elasticsearch/http_client.rb:158:in `block in join_bulk_responses'", "org/jruby/RubyArray.java:4493:in `any?'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.6-java/lib/logstash/outputs/elasticsearch/http_client.rb:158:in `join_bulk_responses'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.6-java/lib/logstash/outputs/elasticsearch/http_client.rb:146:in `bulk'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.6-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:271:in `safe_bulk'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.6-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:192:in `submit'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.6-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:160:in `retrying_submit'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.6-java/lib/logstash/outputs/elasticsearch.rb:303:in `multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:143:in `multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:295:in `block in start_workers'"]}

I changed the log to recognize everything as a string and it inserted fine.

[INFO ] 2023-07-02 18:58:01.451 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pi lines=>[]}
"Oct 09 10:19:15 SumPunFw07.sumotest.com 1","2019/10/09 10:19:15","001234567890002","TRAFFIC","drop","2304","2019/10/09 10:19:15" 209.118.103.150","160.177.222.249","0.0.0.0","0.0.0.0","InternalServer","","","not-applicable","vsys1","inside","z1-FW-Transit"," hernet1/2","","All traffic","2019/10/09 10:19:15","0","1","63712","443","0","0","0x0","udp","deny","60","60","0","1","2019/10/09  :19:15","0","any","0","0123456789","0x0","Netherlands","10.0.0.0-10.255.255.255","0","1","0","policy-deny","0","0","0","0","","Su unFw07","from-policy","","","0","","0","","N/A","0","0","0","0","1202585d-b4d5-5b4c-aaa2-d80d77ba456e","0"
{
                 "NATSourcePort" => "0",
                "SCTPChunksSent" => "0",
                    "@timestamp" => 2023-07-02T09:58:15.582Z,
                     "StartTime" => "2019/10/09 10:19:15",
                    "SCTPChunks" => "0",
                 "BytesReceived" => "0",
                      "column66" => "1202585d-b4d5-5b4c-aaa2-d80d77ba456e",
                 "GeneratedTime" => "2019/10/09 10:19:15",
            "Threat_ContentType" => "drop",
               "PacketsReceived" => "0",
              "SessionEndReason" => "policy-deny",
                    "SourceZone" => "inside",
                    "TunnelType" => "N/A",
                "SourceLocation" => "Netherlands",
                        "Action" => "deny",
                   "PacketsSent" => "1",
                   "NATSourceIP" => "0.0.0.0",
              "InboundInterface" => "ethernet1/2",
                 "DestinationIP" => "160.177.222.249",
                  "SerialNumber" => "001234567890002",
    "DeviceGroupHierarchyLevel4" => "0",
                "SequenceNumber" => "0123456789",
                      "Category" => "any",
    "DeviceGroupHierarchyLevel3" => "0",
             "SCTPAssociationID" => "0",
             "VirtualSystemName" => "",
            "SCTPChunksReceived" => "0",
                    "SourceUser" => "",
               "DestinationZone" => "z1-FW-Transit",
           "DestinationLocation" => "10.0.0.0-10.255.255.255",
              "NATDestinationIP" => "0.0.0.0",
               "ParentSessionID" => "0",
               "DestinationUser" => "",
                      "Protocol" => "udp",
                  "ActionSource" => "from-policy",
                    "SourcePort" => "63712",
             "DestinationVMUUID" => "",
               "ParentStartTime" => "",
                      "SourceIP" => "209.118.103.150",
                   "Application" => "not-applicable",
                      "column67" => "0",
                       "Packets" => "1",
                 "VirtualSystem" => "vsys1",
                    "DeviceName" => "SumPunFw07",
                    "FUTURE_USE" => "0",
                  "SourceVMUUID" => "",
                 "TunnelID_IMSI" => "0",
                     "LogAction" => "All traffic",
                   "RepeatCount" => "1",
               "DestinationPort" => "443",
                     "SessionID" => "0",
                   "ActionFlags" => "0x0",
                     "BytesSent" => "60",
                         "Bytes" => "60",
                          "Type" => "TRAFFIC",
                         "Flags" => "0x0",
    "DeviceGroupHierarchyLevel2" => "0",
                      "RuleName" => "InternalServer",
                      "@version" => "1",
                   "ReceiveTime" => "2019/10/09 10:19:15",
             "OutboundInterface" => "",
                          "host" => "manticore-logstash",
                   "ElapsedTime" => "0",
    "DeviceGroupHierarchyLevel1" => "0",
            "NATDestinationPort" => "0",
               "MonitorTag_IMEI" => ""
}

If you check on the server side, it is inserted correctly.

mysql: Deprecated program name. It will be removed in a future release, use '/usr/bin/mariadb' instead
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 2566
Server version: 6.0.5 1399e6f6b@230629 dev (columnar 2.0.5 106d244@230620) (secondary 2.0.5 106d244@230620) git branch master...origin/master

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MySQL [(none)]> select * from paloaltoindex001;
+---------------------+------------------+-------------------+------------+---------------------+-------------+------------------+---------------------+-----------------+----------+----------------+----------------------------+------------+--------------------+-----------+---------------+-----------------+------------+-------+------------+-----------------+-----------------+-----------------+----------------------------+----------+----------+--------------+-----------------+-------+-------------+----------------+--------------------+----------------------------+------------------+---------+----------------+-------------------+---------------+--------------+------------+---------------+-----------+-------------------+---------+-------------+--------+---------------------+--------------------------------------+--------------------+-------------+------------+----------------+---------------+----------------------------+-----------------+------------+-----------------+-----------------+-------------+----------------+-------------+--------------------------+----------+-------------------------+-----------------+--------------------+-------------------+
| id                  | sessionendreason | virtualsystemname | sourceuser | receivetime         | natsourceip | natdestinationip | starttime           | parentsessionid | category | rulename       | devicegrouphierarchylevel3 | sctpchunks | host               | bytessent | tunnelid_imsi | destinationip   | tunneltype | bytes | future_use | parentstarttime | destinationuser | packetsreceived | devicegrouphierarchylevel4 | column67 | @version | actionsource | destinationport | flags | elapsedtime | sourcelocation | natdestinationport | devicegrouphierarchylevel2 | inboundinterface | packets | sctpchunkssent | sctpassociationid | bytesreceived | sourcevmuuid | sourcezone | virtualsystem | sessionid | destinationvmuuid | type    | repeatcount | action | generatedtime       | column66                             | threat_contenttype | packetssent | sourceport | application    | natsourceport | devicegrouphierarchylevel1 | sourceip        | devicename | monitortag_imei | serialnumber    | logaction   | sequencenumber | actionflags | @timestamp               | protocol | destinationlocation     | destinationzone | sctpchunksreceived | outboundinterface |
+---------------------+------------------+-------------------+------------+---------------------+-------------+------------------+---------------------+-----------------+----------+----------------+----------------------------+------------+--------------------+-----------+---------------+-----------------+------------+-------+------------+-----------------+-----------------+-----------------+----------------------------+----------+----------+--------------+-----------------+-------+-------------+----------------+--------------------+----------------------------+------------------+---------+----------------+-------------------+---------------+--------------+------------+---------------+-----------+-------------------+---------+-------------+--------+---------------------+--------------------------------------+--------------------+-------------+------------+----------------+---------------+----------------------------+-----------------+------------+-----------------+-----------------+-------------+----------------+-------------+--------------------------+----------+-------------------------+-----------------+--------------------+-------------------+
| 4974179136428834820 | policy-deny      |                   |            | 2019/10/09 10:19:15 | 0.0.0.0     | 0.0.0.0          | 2019/10/09 10:19:15 | 0               | any      | InternalServer | 0                          | 0          | manticore-logstash | 60        | 0             | 160.177.222.249 | N/A        | 60    | 0          |                 |                 | 0               | 0                          | 0        | 1        | from-policy  | 443             | 0x0   | 0           | Netherlands    | 0                  | 0                          | ethernet1/2      | 1       | 0              | 0                 | 0             |              | inside     | vsys1         | 0         |                   | TRAFFIC | 1           | deny   | 2019/10/09 10:19:15 | 1202585d-b4d5-5b4c-aaa2-d80d77ba456e | drop               | 1           | 63712      | not-applicable | 0             | 0                          | 209.118.103.150 | SumPunFw07 |                 | 001234567890002 | All traffic | 0123456789     | 0x0         | 2023-07-02T08:36:46.080Z | udp      | 10.0.0.0-10.255.255.255 | z1-FW-Transit   | 0                  |                   |
| 4974179136428834821 | policy-deny      |                   |            | 2019/10/09 10:19:15 | 0.0.0.0     | 0.0.0.0          | 2019/10/09 10:19:15 | 0               | any      | InternalServer | 0                          | 0          | manticore-logstash | 60        | 0             | 160.177.222.249 | N/A        | 60    | 0          |                 |                 | 0               | 0                          | 0        | 1        | from-policy  | 443             | 0x0   | 0           | Netherlands    | 0                  | 0                          | ethernet1/2      | 1       | 0              | 0                 | 0             |              | inside     | vsys1         | 0         |                   | TRAFFIC | 1           | deny   | 2019/10/09 10:19:15 | 1202585d-b4d5-5b4c-aaa2-d80d77ba456e | drop               | 1           | 63712      | not-applicable | 0             | 0                          | 209.118.103.150 | SumPunFw07 |                 | 001234567890002 | All traffic | 0123456789     | 0x0         | 2023-07-02T09:10:46.427Z | udp      | 10.0.0.0-10.255.255.255 | z1-FW-Transit   | 0                  |                   |
| 4974179136428834822 | policy-deny      |                   |            | 2019/10/09 10:19:15 | 0.0.0.0     | 0.0.0.0          | 2019/10/09 10:19:15 | 0               | any      | InternalServer | 0                          | 0          | manticore-logstash | 60        | 0             | 160.177.222.249 | N/A        | 60    | 0          |                 |                 | 0               | 0                          | 0        | 1        | from-policy  | 443             | 0x0   | 0           | Netherlands    | 0                  | 0                          | ethernet1/2      | 1       | 0              | 0                 | 0             |              | inside     | vsys1         | 0         |                   | TRAFFIC | 1           | deny   | 2019/10/09 10:19:15 | 1202585d-b4d5-5b4c-aaa2-d80d77ba456e | drop               | 1           | 63712      | not-applicable | 0             | 0                          | 209.118.103.150 | SumPunFw07 |                 | 001234567890002 | All traffic | 0123456789     | 0x0         | 2023-07-02T09:58:15.582Z | udp      | 10.0.0.0-10.255.255.255 | z1-FW-Transit   | 0                  |                   |
| 4974179136428834819 | policy-deny      |                   |            | 2019/10/09 10:19:15 | 0.0.0.0     | 0.0.0.0          | 2019/10/09 10:19:15 | 0               | any      | InternalServer | 0                          | 0          | manticore-logstash | 60        | 0             | 160.177.222.249 | N/A        | 60    | 0          |                 |                 | 0               | 0                          | 0        | 1        | from-policy  | 443             | 0x0   | 0           | Netherlands    | 0                  | 0                          | ethernet1/2      | 1       | 0              | 0                 | 0             |              | inside     | vsys1         | 0         |                   | TRAFFIC | 1           | deny   | 2019/10/09 10:19:15 | 1202585d-b4d5-5b4c-aaa2-d80d77ba456e | drop               | 1           | 63712      | not-applicable | 0             | 0                          | 209.118.103.150 | SumPunFw07 |                 | 001234567890002 | All traffic | 0123456789     | 0x0         | 2023-07-02T07:53:58.163Z | udp      | 10.0.0.0-10.255.255.255 | z1-FW-Transit   | 0                  |                   |
+---------------------+------------------+-------------------+------------+---------------------+-------------+------------------+---------------------+-----------------+----------+----------------+----------------------------+------------+--------------------+-----------+---------------+-----------------+------------+-------+------------+-----------------+-----------------+-----------------+----------------------------+----------+----------+--------------+-----------------+-------+-------------+----------------+--------------------+----------------------------+------------------+---------+----------------+-------------------+---------------+--------------+------------+---------------+-----------+-------------------+---------+-------------+--------+---------------------+--------------------------------------+--------------------+-------------+------------+----------------+---------------+----------------------------+-----------------+------------+-----------------+-----------------+-------------+----------------+-------------+--------------------------+----------+-------------------------+-----------------+--------------------+-------------------+
4 rows in set (0.004 sec)

MySQL [(none)]>

Is this "problem" already known and being fixed?
Thank you.

[tomatolog]

we have a ticket to allow string to integer conversion on data insert
I will inform you on push the fix into master

[nw-engineer]

thank you!
We are looking forward to hearing from you!

[nw-engineer]

@tomatolog
we have a ticket to allow string to integer conversion on data insert
I will inform you on push the fix into master

Do you have an estimated time for when this will be pushed to master?

[sanikolaev]

@nw-engineer the likely related ticket is #2312

but it would be great if you prepared a minimal reproducible example to reproduce the issue you are experiencing to make sure it will be solved by #2312.

[nw-engineer]

tomatolog , sanikolaev

The issue turned out not to be Manticore itself but my Logstash pipeline: some fields were being sent as nil (JSON null). Those nil values caused indexing/query issues on the Manticore side. After normalizing all nil fields to empty strings ("") in Logstash, inserts and searches started working as expected.

For reference, I added this Ruby filter to the pipeline to sanitize the event before output:

filter {
  ruby {
    code => '
      event.to_hash.each do |k, v|
        next if k.start_with?("@")
        if v.nil?
          event.set(k, "")
        elsif v.is_a?(Array)
          event.set(k, v.map { |e| e.nil? ? "" : e })
        elsif v.is_a?(Hash)
          v.each { |k2, v2| event.set("[#{k}][#{k2}]", v2.nil? ? "" : v2) }
        end
      end
    '
  }
}

After this change, the data indexed cleanly and filters (including time-range queries) behaved correctly.
Environment: Manticore 13.6.7 (HTTP 9308), Logstash on Linux.

Many thanks for the help and pointers here. This is resolved on my side—feel free to mark the thread accordingly. If useful, I can share the minimal Logstash config before/after the fix as well.