Elasticsearch Basics.txt

Client -> HTTP Request -> Elasticsearch (REST API)
Client <- Response/Results <- Elasticsearch (REST API)

PUT /pages -- creates an index called pages
DELETE /pages -- deletes an index

GET /_cluster/health
GET /_cat/nodes?v 
GET /_cat/indices?v 
GET /_cat/shards?v 


Indexing
-----------
- a way to associate documents/data that are similar for easier and faster searching
- limit of 2 billion documents in one index (w/ sharding)
- index contains a single shared by defualt

Person Index 
	{name: Lowell Wooden, Age: 29}
	{name: Anthony Tortorello, Age: 27}
	
Products Index


Re-Indexing
----------------
{
POST /_reindex
  "source": {
  	"index": "person"
  },
  "dest": {
  	"index": "person_new"
  },
  // ran for every document being re-indexed
  "script": {       
  	"source": """
  	if (ctx._source.product_id != null) {
  		ctx._source.product_id = ctx._source.product_id.toString();
  	}
  	"""
  }
 }


Sharding
-----------
- sub-dvides indexes into smaller pieces
- each peice is referred to as a "shard"
- sharding is done at the index level and not cluseter
- allows us to store more documents
- fit large indices on nodes
- parallelization in queries (shards on different nodes)
- Split API - increases shards
- Shrink API - decreases shards

PUT /pages -- creates an index called pages


Index = 600 GB 
Node A = 500 GB -> Shared A (300 GB)
Node B = 500 GB -> Shared B (300 GB)
*400 GB free 


Routing + Reading
-----------------

Shard_number = hash(_routing) % num_primary_shards


Writing
---------

Request -> Primary Shard (for validation) -> Replica Shards


Replication
---------------
- enabled by default
- configured at the index level
- works by creating copies of shareds (replica shards)
- a shard that has been replicated is called a "primary shard"
- a primary shard and its replica shards are called a "replication group"
- replica shards can serve search requests like a primary shard
- repliaca shards are NEVER stored on the same node as their primary shard**
- replica shards in a single-node cluster remain in an "unassinged" state until another node is added
- replicate shards "once" if data loss doesn't result in a disaster
- replicate shards "twice" if it does

(primary shards -> replica shard) - Replication Group

Replication Group A
	Primary Shard A
	Replica A1
	Replica A2
	
Replication Group B
	Primary Shard B
	Replica B1
	Replica B2
	
Node A
	Primary Shard A
	Replica B1*
	Replica B2*

Node B
	Primary Shard B
	Replica A1*
	Replica A2
	
Node A
	Primary Shard
	
Node B
	Replica A1 -- can serve request
	Replica A2 -- can serve request
	Replica A3 -- can serve request

*Elasticsearch routes requests to the best shard
	
[Snapshots]
- restore to a given point in time
- taken at the index or entire cluster level
- used for backups, and replication for high availability
- good to use when you need to restructure the way documents are stored on a index
- snapshot the index first and then try to run queries against the changes index
- if it fails, just revert back to the snapshot you took
- snapshots are used for daily backups and before major changes
- replication is needed to keep serving request in case of a failure


Node Roles 
----------------

Master Role
	responsible for performing cluster wide operations (creating, deleting indices, etc); 
	should have a dedicated master node; 
	node.master = true || false

Data Role
	allows a node to store data
	performs search queries
	node.data: true || false

Ingest Role
	enables a node to run ingest pipleines (adding a document to an index)
	processors perform transformation operations on a document before inserting
	simplied version of Logstash (within Elasticsearch)
	useful for simple data transformations
	node.ingest: true || false

Machine Learning Role
	node.ml: true || false
	xpack.ml.enabled: true || false

Coordination Role
	handles the distribution of queries and aggregation results
	useful for large clusters as a loadbalancing mechanism

Voting Role
	participate in voting for a new master node
	cannot be elected itself
	only useful for large clusters
	
ip         heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
172.25.0.2           32          73   1    0.00    0.05     0.07 dilmrt    -      es02

d - data
i - ingest
l - 
m - master
t - 


Importing Data Using Curl
----------------------------------

- use curl to perform a post opertation on the elasticsearch bulk api
- (2) dashes must proceed the data-binary swtich to let the system know that we are specifying a file and not a path
- by not specifying a index name in the file, we determine what index the data will be added to based on the request we send (recipe in this case)

{"index":{"_id":1}}
{"title":"Fast and Easy Pasta With Blistered Cherry Tomato Sauce"}

curl -H "Content-Type application/x-ndjson" -XPOST http://localhost:9200/recipe/_bulk --data-binary "@original.json"
	
Create Index
-------------

PUT /products
{
"Settings": {
  "number_of_shards": 2,
  "number_of_replicas": 2
  }
}	

OR

PUT /products



Indexing Documents
------------------

POST /products/_doc 
{
  "Name": "Coffee Maker",
  "Price": 64
  "in_stock": 10
  "Id": 100
}

Mapping A Index
--------------------

[Explicit Mapping]

PUT /person
{
  "mappings": {
  	"properties": {
  		"first_name": {  "type": "text" },
  		"last_name": {  "type": "text" },
  		"age": {  "type": "integer" },
  		"location": {
  			"properties": {
  				"city": {  "type": "text" },
  				"state": {  "type": "text" }
  			}
  		}
  	}
  }
}


[Add Mapping to Existing Index]

PUT /person/_mapping
{
  	"properties": {
  		"created_at": {  "type": "date" }
  	}
}

[Retrieving Mapping]

GET /person/_mapping 
GET /person/_mapping/field/first_name




Retrieving Documents
-------------------

GET /products/_doc/{id}

GET /products/_search
{
    "query": {
        "match_all": {}
    }
}


Updating Documents
------------------

POST /products/_update/{id}
{
"doc": {
  "in_stock": 3
 }
}

*Documents are immutable
*Behind the scenes, elastic search grabs the current document, changes the fields,
and replaces the old document with the modified document



Scripted Updates
---------------

ctx = context

POST /products/_update/{id}
{
  "Script": {
     "Source": "ctx._source.in_stock--"
  }	
}

POST /products/_update/{id}
{
  "Script": {
     "Source": "ctx._source.in_stock = 10"
  }	
}

POST /products/_update/{id}
{
  "Script": {
     "Source": "ctx._source.in_stock -= params.quantity",
     "Params": {
	"quantity:" 4
    }
  }	
}

POST /products/_update/{id}
{
  "Script": {
     "source": """
     if (ctx._source.in_stock == 0) {
	ctx.op = 'noop;
  }	
   ctx._source.in_stock--;
   """
  
}


Upserts
-------
- to conditionally update/insert a document based on whether it exist or not
- if it already exists, then a script is run, but if doesn't, a new document is indexed with the upsert criteria

POST /products/_update/{id}
{
  "script": {
     "source": "ctx._source.in_stock++"
   },
  "upsert": {
   "name": "Blender"
   "price": 399,
   "in_stock": 5

  }
}

Replace Documents
------------------

PUT /products/_doc/{id}
{
  "Name": "Toaster",
  "Price": 49,
  "in_stock": 4
}

// get updated document to verify 
GET /products/_doc/(id)


Deleting Documents
------------------

DELETE /products/_doc/{id}


Update By Query
----------------------

POST /products/_update_by_query
{
	"script": {
		"source": "ctx._source.in_stock--"
	},
	"query" : {
		"match_all": {}
	}
}

1. Query request comes into coordinating node
2. Snapshot is taken of the index
3. Search query is performed on all index shards to get documents that fit the criteria
4. Bulk update is performed


Delete By Query
----------------------

DELETE /products/_delete_by_query
{
	"query" : {
		"match_all": {}
	}
}

Bulk API
------------

POST /_bulk
{
  "index": { "_index": "products", "_id": 200 }
  {  "name": "Espresso Machine", "price": 1.99, "in_stock": 5 }
  {  "create":  { "_index": "products", "_id": 201 }  }
  {  "name": "Machiato", "price": 2.99, "in_stock": 6 }
}

POST /products/_bulk
{
  {"update": { "_id": 201 } }
  {  "doc": {"price": 129 } }
  "delete": { "_id": 200 } }
}


Search Query
-----------------

term - looks for exact matches (BAD)

match (full text search) - uses the same analyzer as the inverted index (GOOD); best used for multi-word fields like blog post, articles, description

[DSL Syntax]

GET /products/_search
{
	"query" : {
		"match":  {
			"description": "red wine"
	}
  }
}

[URI Syntax]

GET /products/_search?q=*  
GET /products/_search?q=name:Lowell
GET /products/_search?q=name:Lowell AND age:29


Query DSL Syntax
-------------------------

GET /products/_search
{
	"query" : {
		"match_all":  {}
  }
}

**By default, matches are sorted by their "relevance score" (_score)
Elasticsearch is interested in how "well" documents match (not exactly)

TF/IDF = Term Frequency / Inverse Document Frequency 
  the more times the term appears in the relevent field the more relevant the document
  how many times does the word appear within the index (the word "is" appears everywhere so its not relevent)
  how long is the field (the word salad appearing in a 50 char field is more relevant than it is in a 5000 char field)
  
Okapi BM25
  better at handling stop words
  
GET /products/_search?explain
{
	"query" : {
		"match":  {
			"description": "red wine"
	}
  }
}

Search For Multiple Terms In A Field
----------------------------------------------

- use an array to search for multiple terms

GET /product/_search
{
  "query": {
    "terms": {
      "name": [ "Tomato", "Paste" ]
    }
  }
}


Search By Range
-----------------------

GET /product/_search
{
	"query" : {
		"range":  {
			"in_stock": {
				"gte": 1, // greater than
				"lte": 5 // less than
		}
	 }
   }
}
  


Working With Dates
--------------------------
 Retrive documents that are relative to an "anchor" date
 -1y-1d = anchor date "minus" one year and one day
 -1y/M = round up the "month"
 now/M-1y = perform search relative to the current time (now)
 
GET /product/_search
{
	"query" : {
		"range":  {
			"created": {
				"gte": "2010/01/01 || -1y-1d" // [Anchor Date || expression to apply to anchor date]
		}
	}
  }
}


Searching For Non-Null Values
----------------------------------------
- uses "exists" clause 

GET /product/_search
{
	"query" : {
		"exists":  {
			"field": "tags"
		}
	}
}

Matching Based On Prefixes
-------------------------------------

GET /product/_search
{
	"query" : {
		"prefix":  {
			"tags.keyword": "Vege"
		}
	}
}


Using Wildcards
----------------------

- wildcard performance is slow
- avoid putting wildcards at the beginning of a term

GET /product/_search
{
	"query" : {
		"wildcard":  {
			"tags.keyword": "Veg*ble"
		}
	}
}


Using Regular Expressions
------------------------------------

- uses Lucene regular expressions under the hood
- this means I don't have access to the full functionality

GET /product/_search
{
	"query" : {
		"regexp":  {
			"tags.keyword": "Veget[a-zA-Z]+ble"
		}
	}
}


Full Text Searches

GET /recipe/_search
{
	"query" : {
		"match":  {
			  "title": "Recipes with pasta or spaghetti"
			}
	}
}

GET /recipe/_search
{
	"query" : {
		"match":  {
			  "title": {
			  	"query": "Recipes with pasta or spaghetti",
			  	"operator": "and" // default is or; we are setting it explicitly to and
			  }
		}
	}
}


Matching Phrases
------------------------
- the order of the phrases matter in the "match_phrase" 
- if the order is not correct then results can undesirable

GET /recipe/_search
{
	"query" : {
		"match_phrase":  {
			  "title": "spaghetti puttanesca"
			}
	}
}

Searching Multiple Fields
---------------------------------

GET /recipe/_search
{
	"query" : {
		"multi_match":  {
			  "query": "pasta",
			  "fields": [ "title", "description" ]
			}
	}
}