Skip to content

Commit d73f5ee

Browse files
Merge pull request #104562 from yossi-karp/user/v-yokarp/ongoing-changes
Updates and new kql quick ref topic
2 parents 093f83c + 9ccb435 commit d73f5ee

File tree

4 files changed

+107
-58
lines changed

4 files changed

+107
-58
lines changed

articles/data-explorer/ingest-data-streaming.md

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -11,14 +11,14 @@ ms.date: 08/30/2019
1111

1212
# Streaming ingestion (Preview)
1313

14-
Streaming ingestion is targeted for scenarios that require low latency with an ingestion time of less than 10 seconds for varied volume data. It's used for optimization of operational processing of many tables, in one or more databases where the stream of data into each table is relatively small (few records per second) but overall data ingestion volume is high (thousands of records per second).
14+
Streaming ingestion is targeted for scenarios that require low latency with an ingestion time of less than 10 seconds for varied volume data. It's used to optimize operational processing of many tables, in one or more databases, where the stream of data into each table is relatively small (few records per second) but overall data ingestion volume is high (thousands of records per second).
1515

1616
Use the classic (bulk) ingestion instead of streaming ingestion when the amount of data grows to more than 1 MB per second per table. Read [Data ingestion overview](/azure/data-explorer/ingest-data-overview) to learn more about the various methods of ingestion.
1717

1818
> [!NOTE]
1919
> Streaming ingestion doesn't support the following features:
2020
> * [Database cursors](/azure/kusto/management/databasecursor).
21-
> * [Data mapping](/azure/kusto/management/mappings). Only [pre-created](/azure/kusto/management/create-ingestion-mapping-command) data mapping is supported.
21+
> * [Data mapping](/azure/kusto/management/mappings). Only [pre-created](/azure/kusto/management/tables#create-ingestion-mapping) data mapping is supported.
2222
2323
## Prerequisites
2424

@@ -59,7 +59,7 @@ There are two supported streaming ingestion types:
5959
> [!WARNING]
6060
> Disabling streaming ingestion may take a few hours.
6161
62-
1. Drop [streaming ingestion policy](/azure/kusto/concepts/streamingingestionpolicy) from all relevant tables and databases. The streaming ingestion policy removal triggers streaming ingestion data movement from the initial storage to the permanent storage in the column store (extents or shards). The data movement can last between a few seconds to a few hours, depending on the amount of data in the initial storage and the CPU and memory utilization of the cluster.
62+
1. Drop [streaming ingestion policy](/azure/kusto/concepts/streamingingestionpolicy) from all relevant tables and databases. The streaming ingestion policy removal triggers streaming ingestion data movement from the initial storage to the permanent storage in the column store (extents or shards). The data movement can last between a few seconds to a few hours, depending on the amount of data in the initial storage and how the CPU and memory is used by the cluster.
6363
1. In the Azure portal, go to your Azure Data Explorer cluster. In **Settings**, select **Configurations**.
6464
1. In the **Configurations** pane, select **Off** to disable **Streaming ingestion**.
6565
1. Select **Save**.
@@ -68,10 +68,11 @@ There are two supported streaming ingestion types:
6868

6969
## Limitations
7070

71-
* Streaming ingestion performance and capacity scales with increased VM and cluster sizes. Concurrent ingestions are limited to 6 ingestions per core. For example, for 16 core SKUs, such as D14 and L16, the maximal supported load is 96 concurrent ingestions. For 2 core SKUs, such as D11, the maximal supported load is 12 concurrent ingestions.
71+
* Streaming ingestion performance and capacity scales with increased VM and cluster sizes. Concurrent ingestions are limited to six ingestions per core. For example, for 16 core SKUs, such as D14 and L16, the maximal supported load is 96 concurrent ingestions. For two core SKUs, such as D11, the maximal supported load is 12 concurrent ingestions.
7272
* The data size limitation per ingestion request is 4 MB.
7373
* Schema updates, such as creation and modification of tables and ingestion mappings, may take up to 5 minutes for the streaming ingestion service.
7474
* Enabling streaming ingestion on a cluster, even when data isn't ingested via streaming, uses part of the local SSD disk of the cluster machines for streaming ingestion data and reduces the storage available for hot cache.
75+
* [Extent tags](/azure/kusto/management/extents-overview.md#extent-tagging) can't be set on the streaming ingestion data.
7576

7677
## Next steps
7778

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
---
2+
title: KQL quick reference
3+
description: A list of useful KQL functions and their definitions with syntax examples.
4+
author: yossi-karp
5+
ms.author: v-yokarp
6+
ms.reviewer:
7+
ms.service: data-explorer
8+
ms.topic: conceptual
9+
ms.date: 01/19/2020
10+
---
11+
12+
# KQL quick reference
13+
14+
This article shows you a list of functions and their descriptions to help get you started using Kusto Query Language.
15+
16+
| Operator/Function | Description | Syntax |
17+
| :---------------------------------------------- | :------------------------------------ |:-------------------------------------------------|
18+
|**Filter/Search/Condition** |**_Find relevant data by filtering or searching_** | |
19+
| [where](/azure/kusto/query/whereoperator.md) | Filters on a specific predicate | `T | where Predicate` |
20+
| [where contains/has](/azure/kusto/query/whereoperator.md) | `Contains`: Looks for any substring match <br> `Has`: Looks for a specific word (better performance) | `T | where col1 contains/has "[search term]"`|
21+
| [search](/azure/kusto/query/searchoperator.md) | Searches all columns in the table for the value | `[TabularSource |] search [kind=CaseSensitivity] [in (TableSources)] SearchPredicate` |
22+
| [take](/azure/kusto/query/takeoperator.md) | Returns the specified number of records. Use to test a query<br>**_Note_**: `_take`_ and `_limit`_ are synonyms. | `T | take NumberOfRows` |
23+
| [case](/azure/kusto/query/casefunction.md) | Adds a condition statement, similar to if/then/elseif in other systems. | `case(predicate_1, then_1, predicate_2, then_2, predicate_3, then_3, else)` |
24+
| [distinct](/azure/kusto/query/distinctoperator.md) | Produces a table with the distinct combination of the provided columns of the input table | `distinct [ColumnName], [ColumnName]` |
25+
| **Date/Time** |**_Operations that use date and time functions_** | |
26+
|[ago](/azure/kusto/query/agofunction.md) | Returns the time offset relative to the time the query executes. For example, `ago(1h)` is one hour before the current clock’s reading. | `ago(a_timespan)` |
27+
| [format_datetime](/azure/kusto/query/format-datetimefunction.md) | Returns data in [various date formats](/azure/kusto/query/format-datetimefunction.md#supported-formats). | `format_datetime(datetime , format)` |
28+
| [bin](/azure/kusto/query/binfunction.md) | Rounds all values in a timeframe and groups them | `bin(value,roundTo)` |
29+
| **Create/Remove Columns** |**_Add or remove columns in a table_** | |
30+
| [print](/azure/kusto/query/printoperator.md) | Outputs a single row with one or more scalar expressions | `print [ColumnName =] ScalarExpression [',' ...]` |
31+
| [project](/azure/kusto/query/projectoperator.md) | Selects the columns to include in the order specified | `T | project ColumnName [= Expression] [, ...]` <br> Or <br> `T | project [ColumnName | (ColumnName[,]) =] Expression [, ...]` |
32+
| [project-away](/azure/kusto/query/projectawayoperator.md) | Selects the columns to exclude from the output | `T | project-away ColumnNameOrPattern [, ...]` |
33+
| [extend](/azure/kusto/query/extendoperator.md) | Creates a calculated column and adds it to the result set | `T | extend [ColumnName | (ColumnName[, ...]) =] Expression [, ...]` |
34+
| **Sort and Aggregate Dataset** |**_Restructure the data by sorting or grouping them in meaningful ways_**| |
35+
| [sort](/azure/kusto/query/sortoperator.md) | Sorts the rows of the input table by one or more columns in ascending or descending order | `T | sort by expression1 [asc|desc], expression2 [asc|desc], …` |
36+
| [top](/azure/kusto/query/topoperator.md) | Returns the first N rows of the dataset when the dataset is sorted using `by` | `T | top numberOfRows by expression [asc|desc] [nulls first|last]` |
37+
| [summarize](/azure/kusto/query/summarizeoperator.md) | Groups the rows according to the `by` group columns, and calculates aggregations over each group | `T | summarize [[Column =] Aggregation [, ...]] [by [Column =] GroupExpression [, ...]]` |
38+
| [count](/azure/kusto/query/countoperator.md) | Counts records in the input table (for example, T)<br>This operator is shorthand for `summarize count() `| `T | count` |
39+
| [join](/azure/kusto/query/joinoperator.md) | Merges the rows of two tables to form a new table by matching values of the specified column(s) from each table. Supports a full range of join types: `flouter`, `inner`, `innerunique`, `leftanti`, `leftantisemi`, `leftouter`, `leftsemi`, `rightanti`, `rightantisemi`, `rightouter`, `rightsemi` | `LeftTable | join [JoinParameters] ( RightTable ) on Attributes` |
40+
| [union](/azure/kusto/query/unionoperator.md) | Takes two or more tables and returns all their rows | `[T1] | union [T2], [T3], …` |
41+
| [range](/azure/kusto/query/rangeoperator.md) | Generates a table with an arithmetic series of values | `range columnName from start to stop step step` |
42+
| **Format Data** | **_Restructure the data to output in a useful way_** | |
43+
| [lookup](/azure/kusto/query/lookupoperator.md) | Extends the columns of a fact table with values looked-up in a dimension table | `T1 | lookup [kind = (leftouter|inner)] ( T2 ) on Attributes` |
44+
| [mv-expand](/azure/kusto/query/mvexpandoperator.md) | Turns dynamic arrays into rows (multi-value expansion) | `T | mv-expand Column` |
45+
| [parse](/azure/kusto/query/parseoperator.md) | Evaluates a string expression and parses its value into one or more calculated columns. Use for structuring unstructured data. | `T | parse [kind=regex [flags=regex_flags] |simple|relaxed] Expression with * (StringConstant ColumnName [: ColumnType]) *...` |
46+
| [make-series](/azure/kusto/query/make-seriesoperator.md) | Creates series of specified aggregated values along a specified axis | `T | make-series [MakeSeriesParamters] [Column =] Aggregation [default = DefaultValue] [, ...] on AxisColumn from start to end step step [by [Column =] GroupExpression [, ...]]` |
47+
| [let](/azure/kusto/query/letstatement.md) | Binds a name to expressions that can refer to its bound value. Values can be lambda expressions to create ad-hoc functions as part of the query. Use `let` to create expressions over tables whose results look like a new table. | `let Name = ScalarExpression | TabularExpression | FunctionDefinitionExpression` |
48+
| **General** | **_Miscellaneous operations and function_** | |
49+
| [invoke](/azure/kusto/query/invokeoperator.md) | Runs the function on the table that it receives as input. | `T | invoke function([param1, param2])` |
50+
| [evaluate pluginName](/azure/kusto/query/evaluateoperator.md) | Evaluates query language extensions (plugins) | `[T |] evaluate [ evaluateParameters ] PluginName ( [PluginArg1 [, PluginArg2]... )` |
51+
| **Visualization** | **_Operations that display the data in a graphical format_** | |
52+
| [render](/azure/kusto/query/renderoperator.md) | Renders results as a graphical output | `T | render Visualization [with (PropertyName = PropertyValue [, ...] )]` |

articles/data-explorer/toc.yml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,10 @@
3232
- name: Data ingestion overview
3333
displayName: pipelines, connectors, plugins, Python, .NET, Java, Node, REST
3434
href: ingest-data-overview.md
35+
- name: Kusto Query Language
36+
items:
37+
- name: Quick reference guide
38+
href: kql-quick-reference.md
3539
- name: Data visualization overview
3640
href: viz-overview.md
3741
- name: Security overview

0 commit comments

Comments
 (0)