Merge branch 'master' into autostart-engine

Author: stepansergeevitch

docs/pages/product/apis-integrations/ai-api.mdx

@@ -138,3 +138,83 @@ One way of handling this is to pass the error message back into the AI API; it m
 #### 3. Continue wait
 
 When using `"runQuery": true`, you might sometimes receive a query result containing `{ "error": "Continue wait" }`. If this happens, you should use `/load` ([described above](#2-load)) instead of `runQuery` to run the query, and handle retries as described in the [REST API documentation](/product/apis-integrations/rest-api#continue-wait).
+
+## Advanced Usage
+
+<InfoBox>
+    The advanced features discussed here are available on Cube version 1.1.7 and above.
+</InfoBox>
+
+### Custom prompts
+
+You can prompt the AI API with custom instructions. For example, you may want it to always
+respond in a particular language, or to refer to itself by a name matching your brand.
+Custom prompts also allow you to give the model more context on your company and data model,
+for example if it should usually prefer a particular view.
+
+To use a custom prompt, set the `CUBE_CLOUD_AI_API_PROMPT` environment variable in your deployment.
+
+<InfoBox>
+  Custom prompts add to, rather than overwrite, the AI API's existing prompting, so you
+  do not need to re-write instructions around how to generate the query itself.
+</InfoBox>
+
+### Meta tags
+
+The AI API can read [meta tags](/reference/data-model/view#meta) on your dimensions, measures, 
+segments, and views.
+
+Use the `ai` meta tag to give context that is specific to AI and goes beyond what is 
+included in the description. This can have any keys that you want. For example, you can use it
+to give the AI context on possible values in a categorical dimension:
+```yaml
+      - name: status
+        sql: status
+        type: string
+        meta:
+          ai:
+            values:
+              - shipped
+              - processing
+              - completed
+```
+
+### Other LLM providers
+
+<InfoBox>
+  These environment variables also apply to the [AI Assistant](/product/workspace/ai-assistant),
+  if it is enabled on your deployment.
+</InfoBox>
+
+If desired, you may "bring your own" LLM model by providing a model and API credentials
+for a supported model provider. Do this by setting environment variables in your Cube
+deployment. See below for required variables by provider (required unless noted):
+
+#### AWS Bedrock
+
+<WarningBox>
+  The AI API currently supports only Anthropic Claude models on AWS Bedrock. Other
+  models may work but are not fully supported.
+</WarningBox>
+
+- `CUBE_BEDROCK_MODEL_ID` - A supported [AWS Bedrock chat model](https://docs.aws.amazon.com/bedrock/latest/userguide/models-supported.html), for example `anthropic.claude-3-5-sonnet-20241022-v2:0`
+- `CUBE_BEDROCK_ACCESS_KEY` - An access key for an IAM user with `InvokeModelWithResponseStream` permissions on the desired region/model.
+- `CUBE_BEDROCK_ACCESS_SECRET` - The corresponding access secret
+- `CUBE_BEDROCK_REGION_ID` - A supported AWS Bedrock region, for example `us-west-2`
+
+#### GCP Vertex
+
+<WarningBox>
+  The AI API currently supports only Anthropic Claude models on GCP Vertex. Other
+  models may work but are not fully supported.
+</WarningBox>
+
+- `CUBE_VERTEX_MODEL_ID` - A supported GCP Vertex chat model, for example `claude-3-5-sonnet@20240620`
+- `CUBE_VERTEX_PROJECT_ID` - The GCP project the model is deployed in
+- `CUBE_VERTEX_REGION` - The GCP region the model is deployed in, for example `us-east5`
+- `CUBE_VERTEX_CREDENTIALS` - The private key for a service account with permissions to run the chosen model
+
+#### OpenAI
+
+- `OPENAI_MODEL` - An OpenAI chat model ID, for example `gpt-4o`
+- `OPENAI_API_KEY` - An OpenAI API key (we recommend creating a service account for the AI API)

docs/pages/product/caching/running-in-production.mdx

@@ -325,6 +325,28 @@ Refresh worker should be able to finish pre-aggregation refresh before
 garbage collection starts. It means that all pre-aggregation partitions
 should be built before any tables are removed.
 
+#### Supported file systems
+
+The garbage collection mechanism relies on the ability of the underlying file
+system to report the creation time of a file.
+
+If the file system does not support getting the creation time, you will see the
+following error message in Cube Store logs:
+
+```
+ERROR [cubestore::remotefs::cleanup] <pid:1>
+error while getting created time for file "<name>.chunk.parquet":
+creation time is not available for the filesystem
+```
+
+<ReferenceBox>
+
+XFS is known to not support getting the creation time of a file.
+Please see [this issue](https://github.com/cube-js/cube/issues/7905#issuecomment-2504212623)
+for possible workarounds.
+
+</ReferenceBox>
+
 ## Security
 
 ### Authentication

docs/pages/product/workspace/ai-assistant.mdx

@@ -58,6 +58,49 @@ The query will automatically run in the sidebar and can be opened in the [Playgr
 
 <Screenshot src="https://ucarecdn.com/4249ff1e-fae1-42c8-ad3a-b9e406ea2022/Screenshot20240624at34327PM.png" />
 
+## Advanced Usage
+
+<InfoBox>
+    The advanced features discussed here are available on Cube version 1.1.7 and above.
+</InfoBox>
+
+### Custom prompts
+
+You can prompt the AI Assistant with custom instructions. For example, you may want it to always
+respond in a particular language, or to refer to itself by a name matching your brand.
+Custom prompts also allow you to give the model more context on your company and data model,
+for example if it should usually prefer a particular view.
+
+To use a custom prompt, set the `CUBE_CLOUD_AI_ASSISTANT_PROMPT` environment variable in your deployment.
+
+<InfoBox>
+  Custom prompts add to, rather than overwrite, the AI Assistant's existing prompting.
+</InfoBox>
+
+### Meta tags
+
+The AI Assistant can read [meta tags](/reference/data-model/view#meta) on your dimensions, measures, 
+segments, and views.
+
+Use the `ai` meta tag to give context that is specific to AI and goes beyond what is 
+included in the description. This can have any keys that you want. For example, you can use it
+to give the AI context on possible values in a categorical dimension:
+```yaml
+      - name: status
+        sql: status
+        type: string
+        meta:
+          ai:
+            values:
+              - shipped
+              - processing
+              - completed
+```
+
+### Other LLM providers
+
+See the [AI API's documentation][ref-ai-api-providers] for information on how to "bring your own" LLM. 
+
 ## FAQ and limitations
 
 ### 1. What language model(s) does the AI Assistant use?
@@ -83,3 +126,4 @@ The query will automatically run in the sidebar and can be opened in the [Playgr
 [ref-catalog]: /product/workspace/semantic-catalog
 [ref-playground]: /product/workspace/playground
 [ref-catalog-downstream]: /product/workspace/semantic-catalog#connecting-downstream-tools
+[ref-ai-api-providers]: /product/apis-integrations/ai-api#other-llm-providers

docs/pages/reference/ai-api.mdx

@@ -11,11 +11,21 @@ Generate a Cube query that can be used to answer a user's question, and (optiona
 | `messages` | ✅ Yes   | An array of messages in the format: `{ "role": "user" \| "assistant", "content": "string" }`    |
 | `views`    |          | An array of view names (used to limit the views that the AI API can use to generate its answer) |
 | `runQuery` |          | Boolean (true or false) whether to run the query and return its results                         |
+| `options`  |          | An object in the format `{ "chart": true \| false }`
 
 Response
 
 - `message` - A message from the AI assistant describing the query, how it was chosen, why it could not generate the requested query, etc.
 - `cube_query` - A Cube [Query](/product/apis-integrations/rest-api/query-format) that could be used to answer the given question
+- `chart` - If the `chart` option is set to `true`, an object containing a chart spec for the generated query in the following format:
+    ```json
+    {
+        "type": "bar" | "line" | "pie" | "table" | "area" | "scatter",
+        "x": string,
+        "y": string[],
+        "pivot": string // optional; the field to pivot by, if any
+    }
+    ```
 
 ### Examples
 
@@ -28,7 +38,7 @@ curl \
  -X POST  \
  -H "Content-Type: application/json" \
  -H "Authorization: EXAMPLE-API-TOKEN" \
- --data '{ "messages": [{ "role": "user", "content": "What cities have the highest aov this year?" }]}' \
+ --data '{ "messages": [{ "role": "user", "content": "What cities have the highest aov this year?", "views": ["orders_view"] }]}' \
  https://YOUR_CUBE_API/cubejs-api/v1/ai/query/completions
 ```
 

docs/pages/reference/data-model/pre-aggregations.mdx

@@ -1635,6 +1635,13 @@ cubes:
 
 </CodeTabs>
 
+<ReferenceBox>
+
+In some cases, indexes would not work with `original_sql` pre-aggregations.
+Please [track this issue](https://github.com/cube-js/cube/issues/7420).
+
+</ReferenceBox>
+
 #### `type`
 
 This option is used to define [aggregating indexes][ref-aggregating-indexes]

packages/cubejs-databricks-jdbc-driver/src/DatabricksQuery.ts

@@ -186,6 +186,7 @@ export class DatabricksQuery extends BaseQuery {
 
   public sqlTemplates() {
     const templates = super.sqlTemplates();
+    templates.functions.CURRENTDATE = 'CURRENT_DATE';
     templates.functions.DATETRUNC = 'DATE_TRUNC({{ args_concat }})';
     templates.functions.DATEPART = 'DATE_PART({{ args_concat }})';
     templates.functions.BTRIM = 'TRIM({% if args[1] is defined %}{{ args[1] }} FROM {% endif %}{{ args[0] }})';
@@ -197,6 +198,7 @@ export class DatabricksQuery extends BaseQuery {
     templates.functions.TRUNC = 'CASE WHEN ({{ args[0] }}) >= 0 THEN FLOOR({{ args_concat }}) ELSE CEIL({{ args_concat }}) END';
     templates.expressions.timestamp_literal = 'from_utc_timestamp(\'{{ value }}\', \'UTC\')';
     templates.expressions.extract = 'EXTRACT({{ date_part }} FROM {{ expr }})';
+    templates.expressions.interval_single_date_part = 'INTERVAL \'{{ num }}\' {{ date_part }}';
     templates.quotes.identifiers = '`';
     templates.quotes.escape = '``';
     // TODO: Databricks has `TIMESTAMP_NTZ` with logic similar to Pg's `TIMESTAMP`

packages/cubejs-ksql-driver/src/KsqlDriver.ts

@@ -131,7 +131,9 @@ export class KsqlDriver extends BaseDriver implements DriverInterface {
     if (this.config.kafkaHost) {
       this.kafkaClient = new Kafka({
         clientId: 'Cube',
-        brokers: [this.config.kafkaHost],
+        brokers: this.config.kafkaHost
+          .split(',')
+          .map(h => h.trim()),
         // authenticationTimeout: 10000,
         // reauthenticationThreshold: 10000,
         ssl: this.config.kafkaUseSsl,

packages/cubejs-schema-compiler/package.json

@@ -57,7 +57,7 @@
     "uuid": "^8.3.2"
   },
   "devDependencies": {
-    "@cubejs-backend/apla-clickhouse": "^1.7.0",
+    "@clickhouse/client": "^1.7.0",
     "@cubejs-backend/linter": "^1.0.0",
     "@cubejs-backend/query-orchestrator": "1.1.7",
     "@types/babel__code-frame": "^7.0.6",

packages/cubejs-schema-compiler/src/adapter/BaseQuery.js

@@ -577,9 +577,9 @@ export class BaseQuery {
   }
 
   /**
-   * Returns an array of SQL query strings for the query.
+   * Returns a pair of SQL query string and parameter values for the query.
    * @param {boolean} [exportAnnotatedSql] - returns annotated sql with not rendered params if true
-   * @returns {Array<string>}
+   * @returns {[string, Array<unknown>]}
    */
   buildSqlAndParams(exportAnnotatedSql) {
     if (getEnv('nativeSqlPlanner')) {
@@ -1521,13 +1521,13 @@ export class BaseQuery {
         this.queryCache
       );
       if (m.expressionName && !collectedMeasures.length && !m.isMemberExpression) {
-        throw new UserError(`Subquery dimension ${m.expressionName} should reference at least one measure`);
+        throw new UserError(`Subquery measure ${m.expressionName} should reference at least one member`);
       }
       if (!collectedMeasures.length && m.isMemberExpression && m.query.allCubeNames.length > 1 && m.measureSql() === 'COUNT(*)') {
         const cubeName = m.expressionCubeName ? `\`${m.expressionCubeName}\` ` : '';
         throw new UserError(`The query contains \`COUNT(*)\` expression but cube/view ${cubeName}is missing \`count\` measure`);
       }
-      return [m.measure, collectedMeasures];
+      return [typeof m.measure === 'string' ? m.measure : `${m.measure.cubeName}.${m.measure.name}`, collectedMeasures];
     }));
   }
 
@@ -3214,24 +3214,35 @@ export class BaseQuery {
         DATE: 'DATE({{ args_concat }})',
       },
       statements: {
-        select: 'SELECT {% if distinct %}DISTINCT {% endif %}' +
+        select: '{% if ctes %} WITH \n' +
+          '{{ ctes | join(\',\n\') }}\n' +
+          '{% endif %}' +
+          'SELECT {% if distinct %}DISTINCT {% endif %}' +
           '{{ select_concat | map(attribute=\'aliased\') | join(\', \') }} {% if from %}\n' +
           'FROM (\n' +
           '{{ from | indent(2, true) }}\n' +
-          ') AS {{ from_alias }}{% endif %}' +
+          ') AS {{ from_alias }}{% elif from_prepared %}\n' +
+          'FROM {{ from_prepared }}' +
+          '{% endif %}' +
           '{% if filter %}\nWHERE {{ filter }}{% endif %}' +
           '{% if group_by %}\nGROUP BY {{ group_by }}{% endif %}' +
+          '{% if having %}\nHAVING {{ having }}{% endif %}' +
           '{% if order_by %}\nORDER BY {{ order_by | map(attribute=\'expr\') | join(\', \') }}{% endif %}' +
           '{% if limit is not none %}\nLIMIT {{ limit }}{% endif %}' +
           '{% if offset is not none %}\nOFFSET {{ offset }}{% endif %}',
         group_by_exprs: '{{ group_by | map(attribute=\'index\') | join(\', \') }}',
+        join: '{{ join_type }} JOIN {{ source }} ON {{ condition }}',
+        cte: '{{ alias }} AS ({{ query | indent(2, true) }})'
       },
       expressions: {
+        column_reference: '{% if table_name %}{{ table_name }}.{% endif %}{{ name }}',
         column_aliased: '{{expr}} {{quoted_alias}}',
+        query_aliased: '{{ query }} AS {{ quoted_alias }}',
         case: 'CASE{% if expr %} {{ expr }}{% endif %}{% for when, then in when_then %} WHEN {{ when }} THEN {{ then }}{% endfor %}{% if else_expr %} ELSE {{ else_expr }}{% endif %} END',
         is_null: '{{ expr }} IS {% if negate %}NOT {% endif %}NULL',
         binary: '({{ left }} {{ op }} {{ right }})',
         sort: '{{ expr }} {% if asc %}ASC{% else %}DESC{% endif %} NULLS {% if nulls_first %}FIRST{% else %}LAST{% endif %}',
+        order_by: '{% if index %} {{ index }} {% else %} {{ expr }} {% endif %} {% if asc %}ASC{% else %}DESC{% endif %}{% if nulls_first %} NULLS FIRST{% endif %}',
         cast: 'CAST({{ expr }} AS {{ data_type }})',
         window_function: '{{ fun_call }} OVER ({% if partition_by_concat %}PARTITION BY {{ partition_by_concat }}{% if order_by_concat or window_frame %} {% endif %}{% endif %}{% if order_by_concat %}ORDER BY {{ order_by_concat }}{% if window_frame %} {% endif %}{% endif %}{% if window_frame %}{{ window_frame }}{% endif %})',
         window_frame_bounds: '{{ frame_type }} BETWEEN {{ frame_start }} AND {{ frame_end }}',
@@ -3260,7 +3271,8 @@ export class BaseQuery {
         gt: '{{ column }} > {{ param }}',
         gte: '{{ column }} >= {{ param }}',
         lt: '{{ column }} < {{ param }}',
-        lte: '{{ column }} <= {{ param }}'
+        lte: '{{ column }} <= {{ param }}',
+        always_true: '1 == 1'
 
       },
       quotes: {
@@ -3270,6 +3282,10 @@ export class BaseQuery {
       params: {
         param: '?'
       },
+      join_types: {
+        inner: 'INNER',
+        left: 'LEFT'
+      },
       window_frame_types: {
         rows: 'ROWS',
         range: 'RANGE',