update

Signed-off-by: daniel parriott <[email protected]>

Author: ddpbsd

_sources/elk/agent/integrations/custom-logs.rst.txt

@@ -0,0 +1,34 @@
+===========
+custom logs
+===========
+
+http
+^^^^
+
+Using the custom api logs integration is tricky, and not recommended.
+Elastic recommends using the Common Expression Language (`cel`) version now, but I don't know that yet.
+
+Get agent status
+################
+
+Get the agent status using the `get agents <https://elastic.co/docs/api/doc/kibana/operation/operation-get-fleet-agents>`_ api.
+
+Use something like the following to get the `unhealthy` or `offline` agents.
+
+.. code-block:: console
+
+   https://kibana.k8s.wafflelab.online:5601/api/fleet/agents?perPage=100&showInactive=true&getStatusSummary=true&kuery=status:unhealthy%20or%20status:offline
+
+Do api authorization by setting the `request transforms` to the following:
+
+.. code-block:: console
+
+   - set:
+       target: headers.Authorization
+       value: "API KEY"
+   - set:
+       target: headers.kbn-xsrf
+       value: true
+
+The `kbn-xsrf` header is necessary to get around `xsrf` protections in kibana.
+

_sources/elk/agent/integrations/index.rst.txt

@@ -10,4 +10,4 @@ Contents:
 
    alienvault-otx
    cel
-
+   custom-logs

_sources/elk/beats/filebeat/notes.rst.txt

@@ -3,8 +3,8 @@
 Filebeat
 ========
 
-Added fields:
-^^^^^^^^^^^^^
+added fields
+^^^^^^^^^^^^
 
 Filebeat appears to add some fields:
 
@@ -14,3 +14,26 @@ Filebeat appears to add some fields:
 
 There may be more, but these are what I have so far.
 
+filestream input
+^^^^^^^^^^^^^^^^
+
+The filestream input requires the files to be greater than 1024 bytes in size to be read.
+This can be a problem for testing.
+
+Adjusting the `prospector` settings can make this easier, but `64` is the absolute minimum.
+I believe the `offset` starts the read at that offset, but I could be mistaken.
+I'm leaving it in because it's what I used and it worked for my uses.
+
+.. code-block:: yaml
+
+   - type: filestream
+     id: logs
+     paths:
+       - /var/log/test.log
+     enabled: true
+     prospector:
+       scanner:
+         fingerprint:
+           length: 64
+           offset: 0
+

_sources/elk/eck.rst.txt

@@ -45,7 +45,7 @@ get the password
 
 .. code-block:: console
 
-   $ kubectl get secret elasticsearch-wln-es-elastic-user  -o go-template='{{.data.elastic | base64decode }}'
+   $ kubectl get secret elasticsearch-es-elastic-user  -o go-template='{{.data.elastic | base64decode }}'
    PASSWORD
 
 links

_sources/elk/elasticsearch/dev.rst.txt

@@ -0,0 +1,49 @@
+
+Change the breaker limit
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+This is a temporary change, and probably a dumb one.
+
+.. code-block:: console
+
+   curl -XPUT localhost:9200/_cluster/settings -d '{
+     "transient" : {
+       "indices.breaker.request.limit" : "41%"
+     }
+   }'
+
+
+get info about nodes in cluster
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+sort by role, then disk usage
+
+.. code-block:: console
+
+   GET _cat/nodes?v&h=name,role,master,shardStatsTotalCount,diskTotal,diskUsedPercent,cpu,load_1m,load_5m,load_15m,heapMax,heapCurrent,heapPercent&s=role,diskUsedPercent
+
+get pipeline statistics for troubleshooting ingest processing times
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: console
+
+   GET _nodes/stats/ingest?filter_path=nodes.*.ingest
+
+get number of indexing failures for all indices
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: console
+
+   GET _all/_stats/indexing?filter_path=indices.*.total.indexing.index_failed
+
+show the current write thread pool statistics
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+queue is particularly useful for diagnosing the overall write load on the cluster
+see: `cat-thread-pool <https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-thread-pool.html>`_
+
+.. code-block:: console
+
+   GET _cat/thread_pool/write?v&h=node_name,name,active,size_queue,queue_size,rejected
+
+

_sources/elk/elasticsearch/enrich_policy.rst.txt

@@ -0,0 +1,16 @@
+=============
+enrich policy
+=============
+
+
+enrichment policies can only have 1 match
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+I guess we need to make sure there is only 1 field you want to match against.
+I could not find information about this in the documentation, but that is not a surprise.
+
+race conditions
+^^^^^^^^^^^^^^^
+
+If a new enrichment policy is put in place and executed, wait until it is finished before installing an ingest pipeline that uses it.
+

_sources/elk/elasticsearch/index.rst.txt

@@ -0,0 +1,22 @@
+
+=============
+elasticsearch
+=============
+
+Contents:
+
+.. toctree::
+   :maxdepth: 2
+
+   settings
+   indices/index
+   dev
+   enrich_policy
+   status
+   templates
+   shards
+   tasks
+   security/index
+   ingest_pipelines
+   interesting
+

_sources/elk/elasticsearch/indices/create_index.rst.txt

@@ -0,0 +1,10 @@
+
+Create an index on ES
+^^^^^^^^^^^^^^^^^^^^^
+
+There are defaults for shards and replicas, so if those are okay, drop the `-d` and everything after it.
+
+.. code-block:: console
+
+   curl -H'Content-Type: application/json' -XPUT 'localhost:9200/ossec?pretty'
+

_sources/elk/elasticsearch/indices/datastream.rst.txt

@@ -0,0 +1,35 @@
+============
+data streams
+============
+
+
+
+decommission a data stream
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. warning::
+   This doesn't actually do what we want.
+
+If a data stream won't be used any longer but the old data needs
+to be kept, it can be modified using the `modify_data_stream` api
+to not have a write index anymore.
+This prevents an empty data stream from sitting around taking up space
+and valuable shards.
+
+.. code-block:: console
+
+   POST /_data_stream/_modify
+   {
+     "actions": [
+       {
+         "remove_backing_index": {
+           "data_stream": "logs-xxx-default",
+           "index": ".ds-logs-xxx-default-000002"
+         }
+       }
+     ]
+   }
+
+Leave the ILM policy intact, so the old indices can continue through
+their stages of life.
+

_sources/elk/elasticsearch/indices/errors.rst.txt

@@ -0,0 +1,27 @@
+======
+errors
+======
+
+rollover target does not point to a write index
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Set one of the indices behind an alias to be the write index.
+
+.. code-block:: console
+
+   POST _aliases
+   {
+	"actions": {
+		"index": "index_name-000001",
+		"alias": "index_alias",
+		"is_write_index": true
+	}
+   }
+
+
+
+restore from snapshot
+^^^^^^^^^^^^^^^^^^^^^
+
+https://elastic.co/guide/en/elasticsearch/reference/current/restore-from-snapshot.html
+