diff --git a/notebooks/01-llama-server-openai-api-endpoints.ipynb b/notebooks/01-llama-server-openai-api-endpoints.ipynb new file mode 100644 index 0000000..c38c7b5 --- /dev/null +++ b/notebooks/01-llama-server-openai-api-endpoints.ipynb @@ -0,0 +1,4819 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": 101, + "id": "422dc501-69bf-4105-b800-9f14c596c67d", + "metadata": {}, + "outputs": [], + "source": [ + "import json\n", + "import numpy as np\n", + "import requests\n", + "from IPython.display import JSON, Markdown\n" + ] + }, + { + "cell_type": "markdown", + "id": "6ff78001-5c11-499c-9bb9-e150fa928ac9", + "metadata": {}, + "source": [ + "# 1. LLaMA C++ HTTP Server API Endpoints" + ] + }, + { + "cell_type": "markdown", + "id": "75e14ac0-87fd-4331-834b-8d191f768d67", + "metadata": {}, + "source": [ + "## 1.1 Start `llama-server`" + ] + }, + { + "cell_type": "markdown", + "id": "bf78f2fd-7acc-454c-8d94-b43d12b14cbe", + "metadata": {}, + "source": [ + "```bash\n", + "MODEL=\"./models/gemma-1.1-7b-it.Q4_K_M.gguf\"\n", + "llama-server \\\n", + " --model $MODEL \\\n", + " --host localhost \\\n", + " --port 8080\n", + "```" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "acbdda17-c032-4380-80ff-e56924b17da7", + "metadata": {}, + "outputs": [], + "source": [] + }, + { + "cell_type": "markdown", + "id": "ce0c8f83-095a-4cf0-b336-691e5f355a4f", + "metadata": {}, + "source": [ + "## 1.2 GET `/health`\n", + "\n", + "Returns heath check result" + ] + }, + { + "cell_type": "code", + "execution_count": 108, + "id": "866b04bc-6d0f-4fa0-8e6c-9632de265438", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "b'{\"status\":\"ok\"}'\n" + ] + } + ], + "source": [ + "response = requests.get(\"http://localhost:8080/health\")\n", + "print(response.content)" + ] + }, + { + "cell_type": "markdown", + "id": "06121652-8d61-41f4-97f7-3b53abc02913", + "metadata": {}, + "source": [ + "### 1.2.1 Response format\n", + "\n", + "- HTTP status code 503\n", + " - Body: `{\"error\": {\"code\": 503, \"message\": \"Loading model\", \"type\": \"unavailable_error\"}}`\n", + " - Explanation: the model is still being loaded.\n", + "- HTTP status code 200\n", + " - Body: `{\"status\": \"ok\" }`\n", + " - Explanation: the model is successfully loaded and the server is ready." + ] + }, + { + "cell_type": "markdown", + "id": "5fb93c3f-3047-498b-905e-a3a53980e05e", + "metadata": {}, + "source": [ + "### 1.1 POST" + ] + }, + { + "cell_type": "markdown", + "id": "89551ed3-eb58-44a3-96e1-09649ad68271", + "metadata": {}, + "source": [ + "#### 1.1.1 Prompting\n", + "\n", + " *Options:*\n", + "\n", + " `prompt`: Provide the prompt for this completion as a string or as an array of strings or numbers representing tokens. Internally, if `cache_prompt` is `true`, the prompt is compared to the previous completion and only the \"unseen\" suffix is evaluated. A `BOS` token is inserted at the start, if all of the following conditions are true:\n", + "\n", + " - The prompt is a string or an array with the first element given as a string\n", + " - The model's `tokenizer.ggml.add_bos_token` metadata is `true`\n", + "\n", + " `cache_prompt`: Re-use KV cache from a previous request if possible. This way the common prefix does not have to be re-processed, only the suffix that differs between the requests. Because (depending on the backend) the logits are **not** guaranteed to be bit-for-bit identical for different batch sizes (prompt processing vs. token generation) enabling this option can cause nondeterministic results. Default: `false`\n", + "\n", + " `stream`: It allows receiving each predicted token in real-time instead of waiting for the completion to finish. To enable this, set to `true`.\n", + "\n", + " `stop`: Specify a JSON array of stopping strings.\n", + " These words will not be included in the completion, so make sure to add them to the prompt for the next iteration. Default: `[]`" + ] + }, + { + "cell_type": "code", + "execution_count": 110, + "id": "646df475-f0d6-4ff4-b3ea-82a000180472", + "metadata": {}, + "outputs": [], + "source": [ + "response = requests.post(\n", + " url=\"http://localhost:8080/completion\",\n", + " json={\n", + " \"prompt\": \"Why is the sky blue?\",\n", + " \"cache_prompt\": False,\n", + " \"stream\": False,\n", + " \"stop\": [],\n", + " }\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 113, + "id": "d97c3018-5a89-43b6-8d8f-1f979f4d4976", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "source": [ + "print(response)" + ] + }, + { + "cell_type": "code", + "execution_count": 116, + "id": "b184d94c-2150-4674-acab-350bce169332", + "metadata": {}, + "outputs": [ + { + "data": { + "application/json": { + "content": "\n\n**Answer:**\n\nThe sky is blue due to a phenomenon called **Rayleigh scattering**. \n\n* Sunlight is composed of all the colors of the rainbow, each with a specific wavelength.\n* When sunlight interacts with molecules in the atmosphere, such as nitrogen and oxygen, the molecules scatter the light. \n* Different wavelengths of light are scattered differently. \n\n**Rayleigh scattering:**\n\n- Shorter wavelengths of light (like blue light) are scattered more efficiently than longer wavelengths. \n- Since blue light is scattered more, it reaches our eyes from all directions in the sky, making the sky appear blue.\n\n**Other factors influencing the color of the sky:**\n\n- Time of day (sun angle)\n- Latitude and altitude\n- Clouds and pollution\n- Sun's position in the sky", + "generation_settings": { + "dynatemp_exponent": 1, + "dynatemp_range": 0, + "frequency_penalty": 0, + "grammar": "", + "ignore_eos": false, + "max_tokens": -1, + "min_keep": 0, + "min_p": 0.05000000074505806, + "mirostat": 0, + "mirostat_eta": 0.10000000149011612, + "mirostat_tau": 5, + "model": "./models/gemma-1.1-7b-it.Q4_K_M.gguf", + "n_ctx": 8192, + "n_discard": 0, + "n_keep": 0, + "n_predict": -1, + "n_probs": 0, + "penalize_nl": false, + "presence_penalty": 0, + "repeat_last_n": 64, + "repeat_penalty": 1, + "samplers": [ + "top_k", + "tfs_z", + "typ_p", + "top_p", + "min_p", + "temperature" + ], + "seed": 4294967295, + "seed_cur": 3670435482, + "stop": [], + "stream": false, + "temperature": 0.800000011920929, + "tfs_z": 1, + "top_k": 40, + "top_p": 0.949999988079071, + "typical_p": 1 + }, + "id_slot": 0, + "index": 0, + "model": "./models/gemma-1.1-7b-it.Q4_K_M.gguf", + "prompt": "Why is the sky blue?", + "stop": true, + "stopped_eos": true, + "stopped_limit": false, + "stopped_word": false, + "stopping_word": "", + "timings": { + "predicted_ms": 10549.645, + "predicted_n": 166, + "predicted_per_second": 15.735126632223169, + "predicted_per_token_ms": 63.55207831325301, + "prompt_ms": 420.141, + "prompt_n": 7, + "prompt_per_second": 16.661073306342395, + "prompt_per_token_ms": 60.02014285714286 + }, + "tokens_cached": 172, + "tokens_evaluated": 7, + "tokens_predicted": 166, + "truncated": false + }, + "text/plain": [ + "" + ] + }, + "execution_count": 116, + "metadata": { + "application/json": { + "expanded": false, + "root": "root" + } + }, + "output_type": "execute_result" + } + ], + "source": [ + "json_data = json.loads(response.content)\n", + "JSON(json_data)" + ] + }, + { + "cell_type": "markdown", + "id": "514dc61a-7148-47f0-b863-b0b38f500017", + "metadata": {}, + "source": [ + "#### 1.1.2 Response format\n", + "\n", + "- Note: When using streaming mode (`stream`), only `content` and `stop` will be returned until end of completion.\n", + "\n", + "- `completion_probabilities`: An array of token probabilities for each completion. The array's length is `n_predict`. Each item in the array has the following structure:\n", + "\n", + "```json\n", + "{\n", + " \"content\": \"\",\n", + " \"probs\": [\n", + " {\n", + " \"prob\": float,\n", + " \"tok_str\": \"\"\n", + " },\n", + " {\n", + " \"prob\": float,\n", + " \"tok_str\": \"\"\n", + " },\n", + " ...\n", + " ]\n", + "},\n", + "```\n", + "\n", + "Notice that each `probs` is an array of length `n_probs`.\n", + "\n", + "- `content`: Completion result as a string (excluding `stopping_word` if any). In case of streaming mode, will contain the next token as a string.\n", + "- `stop`: Boolean for use with `stream` to check whether the generation has stopped (Note: This is not related to stopping words array `stop` from input options)\n", + "- `generation_settings`: The provided options above excluding `prompt` but including `n_ctx`, `model`. These options may differ from the original ones in some way (e.g. bad values filtered out, strings converted to tokens, etc.).\n", + "- `model`: The path to the model loaded with `-m`\n", + "- `prompt`: The provided `prompt`\n", + "- `stopped_eos`: Indicating whether the completion has stopped because it encountered the EOS token\n", + "- `stopped_limit`: Indicating whether the completion stopped because `n_predict` tokens were generated before stop words or EOS was encountered\n", + "- `stopped_word`: Indicating whether the completion stopped due to encountering a stopping word from `stop` JSON array provided\n", + "- `stopping_word`: The stopping word encountered which stopped the generation (or \"\" if not stopped due to a stopping word)\n", + "- `timings`: Hash of timing information about the completion such as the number of tokens `predicted_per_second`\n", + "- `tokens_cached`: Number of tokens from the prompt which could be re-used from previous completion (`n_past`)\n", + "- `tokens_evaluated`: Number of tokens evaluated in total from the prompt\n", + "- `truncated`: Boolean indicating if the context size was exceeded during generation, i.e. the number of tokens provided in the prompt (`tokens_evaluated`) plus tokens generated (`tokens predicted`) exceeded the context size (`n_ctx`)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1b648612-3dbf-4de1-9ba2-adccd55c229f", + "metadata": {}, + "outputs": [], + "source": [ + "_generated_text = json_data[\"content\"]\n", + "Markdown(_generated_text)" + ] + }, + { + "cell_type": "markdown", + "id": "47d601c7-051e-4ad7-9640-3c299affc358", + "metadata": {}, + "source": [ + "#### 1.1.3 Random Number Generator (RNG) Seed\n", + "\n", + "The RNG seed is used to initialize the random number generator that influences the text generation process. By setting a specific value for `seed` you can obtain consistent and reproducible results across multiple runs with the same input and settings. This can be helpful for testing, debugging, or comparing the effects of different options on the generated text to see when they diverge. If the seed is set to a value less than 0, a random seed will be used, which will result in different outputs on each run. The default value is -1 which will choose a random value for `seed`." + ] + }, + { + "cell_type": "code", + "execution_count": 128, + "id": "2bd3b1f7-fe18-4c13-a985-8c2c60128c4b", + "metadata": {}, + "outputs": [], + "source": [ + "response = requests.post(\n", + " url=\"http://localhost:8080/completion\",\n", + " json={\n", + " \"prompt\": \"Why is the sky blue?\",\n", + " \"cache_prompt\": False,\n", + " \"stream\": False,\n", + " \"stop\": [],\n", + " \"seed\": 42\n", + " }\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 129, + "id": "95555aea-c700-41ab-aadf-dd272c7a2fea", + "metadata": {}, + "outputs": [ + { + "data": { + "text/markdown": [ + "\n", + "\n", + "**Answer:**\n", + "\n", + "The sky is blue due to a phenomenon called **Rayleigh scattering**. \n", + "\n", + "* Sunlight is composed of all the colors of the rainbow, each with a specific wavelength. \n", + "* When sunlight interacts with molecules in the atmosphere, such as nitrogen and oxygen, the different wavelengths are scattered in different directions. \n", + "* Shorter wavelengths of light, like blue light, are scattered more efficiently than longer wavelengths. \n", + "* Since our eyes are most sensitive to blue light, we perceive the sky as blue." + ], + "text/plain": [ + "" + ] + }, + "execution_count": 129, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "_json_data = json.loads(response.content)\n", + "_generated_text = _json_data[\"content\"]\n", + "Markdown(_generated_text)" + ] + }, + { + "cell_type": "markdown", + "id": "a722becf-8446-40f4-826b-da19189b45d1", + "metadata": {}, + "source": [ + "#### 1.1.4 Number of Tokens to Predict\n", + "\n", + "The `n_predict` (default: -1) controls the number of tokens the model generates in response to the input prompt. By adjusting this value, you can influence the length of the generated text. A higher value will result in longer text, while a lower value will produce shorter text.\n", + "\n", + "Even though all models have a finite context window, a value of -1 will enable *infinite* text generation. How? When the context window is full half of the tokens after `n_keep` will be discarded. The context must then be re-evaluated before generation can resume. On large models and/or large context windows, this can result in a significant pause in output. If the output delay is undesirable, a value of -2 will stop generation immediately when the context is filled.\n", + "\n", + "It is important to note that the generated text may be shorter than the specified number of tokens if an End-of-Sequence (EOS) token or a reverse prompt is encountered. In interactive mode, text generation will pause and control will be returned to the user. In non-interactive mode, the program will end. In both cases, the text generation may stop before reaching the specified `n_predict` value. If you want the model to keep going without ever producing End-of-Sequence on its own, you can use the `ignore_eos` parameter.\n", + "\n", + "\n", + " `n_keep`: Specify the number of tokens from the prompt to retain when the context size is exceeded and tokens need to be discarded. The number excludes the BOS token.\n", + " By default, this value is set to `0`, meaning no tokens are kept. Use `-1` to retain all tokens from the prompt.\n", + "\n", + " `min_keep`: If greater than 0, force samplers to return N possible tokens at minimum. Default: `0`\n", + "\n", + " `t_max_predict_ms`: Set a time limit in milliseconds for the prediction (a.k.a. text-generation) phase. The timeout will trigger if the generation takes more than the specified time (measured since the first token was generated) and if a new-line character has already been generated. Useful for FIM applications. Default: `0`, which is disabled." + ] + }, + { + "cell_type": "code", + "execution_count": 130, + "id": "d487725b-45f9-41ec-a483-22d85f3b9d8d", + "metadata": {}, + "outputs": [], + "source": [ + "response = requests.post(\n", + " url=\"http://localhost:8080/completion\",\n", + " json={\n", + " \"prompt\": \"Why is the sky blue?\",\n", + " \"cache_prompt\": False,\n", + " \"stream\": False,\n", + " \"stop\": [],\n", + " \"seed\": 42,\n", + " \"n_predict\": -1,\n", + " \"n_keep\": 0,\n", + " \"min_keep\": 0,\n", + " \"ignore_eos\": False\n", + " }\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 131, + "id": "4dcd35c8-d39d-4bca-8705-1bd10a302a39", + "metadata": {}, + "outputs": [ + { + "data": { + "text/markdown": [ + "\n", + "\n", + "**Answer:**\n", + "\n", + "The sky is blue due to a phenomenon called **Rayleigh scattering**. \n", + "\n", + "* Sunlight is composed of all the colors of the rainbow, each with a specific wavelength. \n", + "* When sunlight interacts with molecules in the atmosphere, such as nitrogen and oxygen, the different wavelengths are scattered in different directions. \n", + "* Shorter wavelengths of light, like blue light, are scattered more efficiently than longer wavelengths. \n", + "* Since our eyes are most sensitive to blue light, we perceive the sky as blue." + ], + "text/plain": [ + "" + ] + }, + "execution_count": 131, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "_json_data = json.loads(response.content)\n", + "_generated_text = _json_data[\"content\"]\n", + "Markdown(_generated_text)" + ] + }, + { + "cell_type": "markdown", + "id": "973d8a35-d587-4c99-b3fb-5307c2bcb50a", + "metadata": {}, + "source": [ + "#### 1.1.5 Temperature\n", + "\n", + "The `temperature` hyperparameter that controls the randomness of the generated text. It affects the probability distribution of the model's output tokens. A higher temperature makes the output more random and creative, while a lower temperature makes the output more focused, deterministic, and conservative.\n", + "\n", + " `dynatemp_range`: Dynamic temperature range. The final temperature will be in the range of `[temperature - dynatemp_range; temperature + dynatemp_range]` Default: `0.0`, which is disabled.\n", + "\n", + " `dynatemp_exponent`: Dynamic temperature exponent. Default: `1.0`" + ] + }, + { + "cell_type": "code", + "execution_count": 132, + "id": "90298924-b5ef-417a-8a83-7c62cdd79fef", + "metadata": {}, + "outputs": [], + "source": [ + "response = requests.post(\n", + " url=\"http://localhost:8080/completion\",\n", + " json={\n", + " \"prompt\": \"Why is the sky blue?\",\n", + " \"cache_prompt\": False,\n", + " \"stream\": False,\n", + " \"stop\": [],\n", + " \"seed\": 42,\n", + " \"n_predict\": -1,\n", + " \"n_keep\": 0,\n", + " \"min_keep\": 0,\n", + " \"ignore_eos\": False,\n", + " \"temperature\": 0.8,\n", + " \"dynatemp_range\": 0.1,\n", + " \"dynatemp_exponent\": 1.0\n", + " }\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 133, + "id": "56204f43-6125-495d-a0ca-0b771a6c4130", + "metadata": {}, + "outputs": [ + { + "data": { + "text/markdown": [ + "\n", + "\n", + "**Answer:**\n", + "\n", + "The sky is blue due to a phenomenon called **Rayleigh scattering**. \n", + "\n", + "* Sunlight is composed of all the colors of the rainbow, each with a specific wavelength. \n", + "* When sunlight interacts with molecules in the atmosphere, such as nitrogen and oxygen, the different wavelengths are scattered in different directions. \n", + "* Shorter wavelengths of light, such as blue light, are scattered more efficiently than longer wavelengths. \n", + "* As a result, more blue light is scattered in all directions, reaching our eyes and making the sky appear blue." + ], + "text/plain": [ + "" + ] + }, + "execution_count": 133, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "_json_data = json.loads(response.content)\n", + "_generated_text = _json_data[\"content\"]\n", + "\n", + "Markdown(_generated_text)" + ] + }, + { + "cell_type": "markdown", + "id": "8c773e60-24a4-4a36-8855-9cda246739b3", + "metadata": {}, + "source": [ + "#### 1.1.6 Repeat Penalty\n", + "\n", + "The `repeat_penalty` option helps prevent the model from generating repetitive or monotonous text. A higher value (e.g., 1.5) will penalize repetitions more strongly, while a lower value (e.g., 0.9) will be more lenient. The default value is 1.1.\n", + "\n", + "The `repeat_last_n` option controls the number of tokens in the history to consider for penalizing repetition. A larger value will look further back in the generated text to prevent repetitions, while a smaller value will only consider recent tokens. A value of 0 disables the penalty, and a value of -1 sets the number of tokens considered equal to the context size, `ctx_size`. The default value is 64. \n", + "\n", + " `penalize_nl`: Penalize newline tokens when applying the repeat penalty. Default: `true`\n" + ] + }, + { + "cell_type": "code", + "execution_count": 134, + "id": "941a7482-bc32-4cff-ac08-8489c45dea0d", + "metadata": {}, + "outputs": [], + "source": [ + "response = requests.post(\n", + " url=\"http://localhost:8080/completion\",\n", + " json={\n", + " \"prompt\": \"Why is the sky blue?\",\n", + " \"cache_prompt\": False,\n", + " \"stream\": False,\n", + " \"stop\": [],\n", + " \"seed\": 42,\n", + " \"n_predict\": -1,\n", + " \"n_keep\": 0,\n", + " \"min_keep\": 0,\n", + " \"ignore_eos\": False,\n", + " \"temperature\": 0.8,\n", + " \"dynatemp_range\": 0.1,\n", + " \"dynatemp_exponent\": 1.0,\n", + " \"repeat_penalty\": 1.1,\n", + " \"repeat_last_n\": 64,\n", + " \"penalize_nl\": True\n", + " }\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 135, + "id": "795b009c-3f02-4183-a0eb-34b8424bfeaf", + "metadata": {}, + "outputs": [ + { + "data": { + "text/markdown": [ + "\n", + "\n", + "**Answer:** \n", + "The sky is blue due to the process of **Rayleigh scattering**. Sunlight consists mainly of all wavelengths in a range from violet through red. When sunlight interacts with molecules like nitrogen and oxygen, shorter wavelength light (violet &blue) gets scattered more efficiently than longer wave length lights(orange&red).\n", + "\n", + "* Shorter waves have higher frequency/energy per unit area which results into greater scattering power against the particles of air .\n", + "**The blue wavelengths are dispersed in all directions by these gases.** \n", + "\n", + "\n", + "- Most of this diffused light reaches our eyes from a relatively small portion (about one degree) directly overhead. That's why we see mainly skyblue during clear weather when there isn’t much dust or cloud to absorb the scattered sunlight .\n", + "\n", + "**Factors influencing colour of Sky:**\n", + "* Time & Latitude - different time zones experience differently coloured skies due variations in sun elevation and composition of atmosphere \n", + "\n", + "\n", + "- Cloud Coverage/ Dust particles – clouds block direct contact between Sunlight& air molecules thereby limiting scattering. Similarly, large dust particle can scatter all wavelengths equally leading to a less blue sky ." + ], + "text/plain": [ + "" + ] + }, + "execution_count": 135, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "_json_data = json.loads(response.content)\n", + "_generated_text = _json_data[\"content\"]\n", + "\n", + "Markdown(_generated_text)" + ] + }, + { + "cell_type": "markdown", + "id": "1e2f173c-2dcc-4446-a3cb-c3c3c875bb30", + "metadata": {}, + "source": [ + "#### 1.1.7 Top-K Sampling\n", + "\n", + "Top-k sampling is a text generation method that selects the next token only from the `--top-k` most likely tokens predicted by the model. It helps reduce the risk of generating low-probability or nonsensical tokens, but it may also limit the diversity of the output. A higher value for top-k (e.g., 100) will consider more tokens and lead to more diverse text, while a lower value (e.g., 10) will focus on the most probable tokens and generate more conservative text. The default value is 40.\n", + "\n", + " `top_k`: Limit the next token selection to the K most probable tokens. Default: `40`\n" + ] + }, + { + "cell_type": "code", + "execution_count": 136, + "id": "608fd603-9339-452a-8dae-a4c14c7d7da2", + "metadata": {}, + "outputs": [], + "source": [ + "response = requests.post(\n", + " url=\"http://localhost:8080/completion\",\n", + " json={\n", + " \"prompt\": \"Why is the sky blue?\",\n", + " \"cache_prompt\": False,\n", + " \"stream\": False,\n", + " \"stop\": [],\n", + " \"seed\": 42,\n", + " \"n_predict\": -1,\n", + " \"n_keep\": 0,\n", + " \"min_keep\": 0,\n", + " \"ignore_eos\": False,\n", + " \"temperature\": 0.8,\n", + " \"dynatemp_range\": 0.1,\n", + " \"dynatemp_exponent\": 1.0,\n", + " \"repeat_penalty\": 1.1,\n", + " \"repeat_last_n\": 64,\n", + " \"penalize_nl\": True,\n", + " \"top_k\": 40,\n", + " }\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 137, + "id": "9de0ee1b-9085-42eb-9fab-ed365e810129", + "metadata": {}, + "outputs": [ + { + "data": { + "text/markdown": [ + "\n", + "\n", + "**Answer:** \n", + "The sky is blue due to the process of **Rayleigh scattering**. Sunlight consists mainly of all wavelengths in a range from violet through red. When sunlight interacts with molecules like nitrogen and oxygen, shorter wavelength light (violet &blue) gets scattered more efficiently than longer wave length lights(orange&red).\n", + "\n", + "* Shorter waves have higher frequency/energy per unit area which results into greater scattering power against the particles of air .\n", + "**The blue wavelengths are dispersed in all directions by these gases.** \n", + "\n", + "\n", + "- Most of this diffused light reaches our eyes from a relatively small portion (about one degree) directly overhead. That's why we see mainly skyblue during clear weather when there isn’t much dust or cloud to absorb the scattered sunlight .\n", + "\n", + "**Factors influencing colour of Sky:**\n", + "* Time & Latitude - different time zones experience differently coloured skies due variations in sun elevation and composition of atmosphere \n", + "\n", + "\n", + "- Cloud Coverage/ Dust particles – clouds block direct contact between Sunlight& air molecules thereby limiting scattering. Similarly, large dust particle can scatter all wavelengths equally leading to a less blue sky ." + ], + "text/plain": [ + "" + ] + }, + "execution_count": 137, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "json_data = json.loads(response.content)\n", + "_generated_text = json_data[\"content\"]\n", + "\n", + "Markdown(_generated_text)" + ] + }, + { + "cell_type": "markdown", + "id": "3795881f-dcf1-4fdd-8295-4a53d12078d4", + "metadata": {}, + "source": [ + "#### 1.1.8 Top-P Sampling\n", + "\n", + "Top-p sampling, `top-p`, also known as nucleus sampling, is another text generation method that selects the next token from a subset of tokens that together have a cumulative probability of at least p. This method provides a balance between diversity and quality by considering both the probabilities of tokens and the number of tokens to sample from. A higher value for top-p (e.g., 0.95) will lead to more diverse text, while a lower value (e.g., 0.5) will generate more focused and conservative text. The default value is 0.9.\n", + "\n", + " `top_p`: Limit the next token selection to a subset of tokens with a cumulative probability above a threshold P. Default: `0.95`" + ] + }, + { + "cell_type": "code", + "execution_count": 138, + "id": "8b882d6e-ed33-45c8-8ec8-ca3a2a767395", + "metadata": {}, + "outputs": [], + "source": [ + "response = requests.post(\n", + " url=\"http://localhost:8080/completion\",\n", + " json={\n", + " \"prompt\": \"Why is the sky blue?\",\n", + " \"cache_prompt\": False,\n", + " \"stream\": False,\n", + " \"stop\": [],\n", + " \"seed\": 42,\n", + " \"n_predict\": -1,\n", + " \"n_keep\": 0,\n", + " \"min_keep\": 0,\n", + " \"ignore_eos\": False,\n", + " \"temperature\": 0.8,\n", + " \"dynatemp_range\": 0.1,\n", + " \"dynatemp_exponent\": 1.0,\n", + " \"repeat_penalty\": 1.1,\n", + " \"repeat_last_n\": 64,\n", + " \"penalize_nl\": True,\n", + " \"top_k\": 40,\n", + " \"top_p\": 0.95,\n", + " }\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 139, + "id": "0dc9538a-9d32-407c-9bca-6f568643dc33", + "metadata": {}, + "outputs": [ + { + "data": { + "text/markdown": [ + "\n", + "\n", + "**Answer:** \n", + "The sky is blue due to the process of **Rayleigh scattering**. Sunlight consists mainly of all wavelengths in a range from violet through red. When sunlight interacts with molecules like nitrogen and oxygen, shorter wavelength light (violet &blue) gets scattered more efficiently than longer wave length lights(orange&red).\n", + "\n", + "* Shorter waves have higher frequency/energy per unit area which results into greater scattering power against the particles of air .\n", + "**The blue wavelengths are dispersed in all directions by these gases.** \n", + "\n", + "\n", + "- Most of this diffused light reaches our eyes from a relatively small portion (about one degree) directly overhead. That's why we see mainly skyblue during clear weather when there isn’t much dust or cloud to absorb the scattered sunlight .\n", + "\n", + "**Factors influencing colour of Sky:**\n", + "* Time & Latitude - different time zones experience differently coloured skies due variations in sun elevation and composition of atmosphere \n", + "\n", + "\n", + "- Cloud Coverage/ Dust particles – clouds block direct contact between Sunlight& air molecules thereby limiting scattering. Similarly, large dust particle can scatter all wavelengths equally leading to a less blue sky ." + ], + "text/plain": [ + "" + ] + }, + "execution_count": 139, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "json_data = json.loads(response.content)\n", + "_generated_text = json_data[\"content\"]\n", + "\n", + "Markdown(_generated_text)" + ] + }, + { + "cell_type": "markdown", + "id": "df20935d-0211-44a7-91f0-8df8953d8779", + "metadata": {}, + "source": [ + "#### 1.1.9 Min-P Sampling\n", + "\n", + "The `--min-p` sampling method sets a minimum base probability threshold for token selection and aims to ensure a balance of quality and variety in the generated text. The `--min-p` method was designed as an alternative to `--top-p`. The parameter $p$ represents the minimum probability for a token to be considered, relative to the probability of the most likely token. For example, with $p=0.05$ and the most likely token having a probability of 0.9, logits with a value less than 0.045 are filtered out. The default value is 0.1.\n", + "\n", + " `min_p`: The minimum probability for a token to be considered, relative to the probability of the most likely token. Default: `0.05`\n", + " " + ] + }, + { + "cell_type": "code", + "execution_count": 140, + "id": "3480cb23-7a09-4b60-9020-165231f72d9e", + "metadata": {}, + "outputs": [], + "source": [ + "response = requests.post(\n", + " url=\"http://localhost:8080/completion\",\n", + " json={\n", + " \"prompt\": \"Why is the sky blue?\",\n", + " \"cache_prompt\": False,\n", + " \"stream\": False,\n", + " \"stop\": [],\n", + " \"seed\": 42,\n", + " \"n_predict\": -1,\n", + " \"n_keep\": 0,\n", + " \"min_keep\": 0,\n", + " \"ignore_eos\": False,\n", + " \"temperature\": 0.8,\n", + " \"dynatemp_range\": 0.1,\n", + " \"dynatemp_exponent\": 1.0,\n", + " \"repeat_penalty\": 1.1,\n", + " \"repeat_last_n\": 64,\n", + " \"penalize_nl\": True,\n", + " \"top_k\": 40,\n", + " \"top_p\": 0.95,\n", + " \"min_p\": 0.05,\n", + " }\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 141, + "id": "acace774-f746-4142-90c3-6a8c39196577", + "metadata": {}, + "outputs": [ + { + "data": { + "text/markdown": [ + "\n", + "\n", + "**Answer:** \n", + "The sky is blue due to the process of **Rayleigh scattering**. Sunlight consists mainly of all wavelengths in a range from violet through red. When sunlight interacts with molecules like nitrogen and oxygen, shorter wavelength light (violet &blue) gets scattered more efficiently than longer wave length lights(orange&red).\n", + "\n", + "* Shorter waves have higher frequency/energy per unit area which results into greater scattering power against the particles of air .\n", + "**The blue wavelengths are dispersed in all directions by these gases.** \n", + "\n", + "\n", + "- Most of this diffused light reaches our eyes from a relatively small portion (about one degree) directly overhead. That's why we see mainly skyblue during clear weather when there isn’t much dust or cloud to absorb the scattered sunlight .\n", + "\n", + "**Factors influencing colour of Sky:**\n", + "* Time & Latitude - different time zones experience differently coloured skies due variations in sun elevation and composition of atmosphere \n", + "\n", + "\n", + "- Cloud Coverage/ Dust particles – clouds block direct contact between Sunlight& air molecules thereby limiting scattering. Similarly, large dust particle can scatter all wavelengths equally leading to a less blue sky ." + ], + "text/plain": [ + "" + ] + }, + "execution_count": 141, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "json_data = json.loads(response.content)\n", + "_generated_text = json_data[\"content\"]\n", + "\n", + "Markdown(_generated_text)" + ] + }, + { + "cell_type": "markdown", + "id": "2cda8568-978c-4f49-8473-420182261a00", + "metadata": {}, + "source": [ + "#### 1.1.10 Locally Typical Sampling\n", + "\n", + "Locally typical sampling, `typical_p`, promotes the generation of contextually coherent and diverse text by sampling tokens that are typical or expected based on the surrounding context. By setting the parameter $p$ between 0 and 1, you can control the balance between producing text that is locally coherent and diverse. The default setting is $p=1.0$, which disables locally typical sampling.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 142, + "id": "5b8153e3-e2ca-4cb2-8fa5-cf897e0f4029", + "metadata": {}, + "outputs": [], + "source": [ + "response = requests.post(\n", + " url=\"http://localhost:8080/completion\",\n", + " json={\n", + " \"prompt\": \"Why is the sky blue?\",\n", + " \"cache_prompt\": False,\n", + " \"stream\": False,\n", + " \"stop\": [],\n", + " \"seed\": 42,\n", + " \"n_predict\": -1,\n", + " \"n_keep\": 0,\n", + " \"min_keep\": 0,\n", + " \"ignore_eos\": False,\n", + " \"temperature\": 0.8,\n", + " \"dynatemp_range\": 0.0,\n", + " \"dynatemp_exponent\": 1.0,\n", + " \"repeat_penalty\": 1.1,\n", + " \"repeat_last_n\": 64,\n", + " \"penalize_nl\": True,\n", + " \"top_k\": 40,\n", + " \"top_p\": 0.95,\n", + " \"min_p\": 0.05,\n", + " \"typical_p\": 1.0,\n", + " }\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 143, + "id": "fe297741-2447-44f1-883e-d683d87c68c3", + "metadata": {}, + "outputs": [ + { + "data": { + "text/markdown": [ + "\n", + "\n", + "**Answer:** \n", + "The sky is blue due to the process of **Rayleigh scattering**. Sunlight consists mainly of all wavelengths or colors. When sunlight interacts with molecules in Earth's atmosphere, like nitrogen and oxygen atoms these particles scatter light rays randomly without altering their direction significantly . But different color lights are scattered differently based on wavelength:\n", + "\n", + "- Shorter Wavelength (blue/violet) - gets dispersed more efficiently\n", + "_LongerWavelength_(red / infra red)_ is less effectively diffused. \n", + "\n", + "\n", + "**The blue wavelengths of sunlight:** travel in straight lines until they reach our eyes, giving us the impression that sky appears **BLUE**." + ], + "text/plain": [ + "" + ] + }, + "execution_count": 143, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "json_data = json.loads(response.content)\n", + "_generated_text = json_data[\"content\"]\n", + "\n", + "Markdown(_generated_text)" + ] + }, + { + "cell_type": "markdown", + "id": "74a53574-f7ca-4243-ace6-8b3a11928337", + "metadata": {}, + "source": [ + "#### 1.1.11 Mirostat Sampling\n", + "\n", + "Mirostat is an algorithm that actively maintains the quality of generated text within a desired range during text generation. It aims to strike a balance between coherence and diversity, avoiding low-quality output caused by excessive repetition (boredom traps) or incoherence (confusion traps). To enable Mirostat sampling set `mirostat` to 1 = Mirostat 1.0 or 2 = Mirostat 2.0. By default Mirostat sampling is disabled `mirostat` to 0.\n", + "\n", + "The `mirostat_lr` option sets the Mirostat learning rate (eta). The learning rate influences how quickly the algorithm responds to feedback from the generated text. A lower learning rate will result in slower adjustments, while a higher learning rate will make the algorithm more responsive. The default value is `0.1`.\n", + "\n", + "The `mirostat_tau` option sets the Mirostat target entropy (tau), which represents the desired perplexity value for the generated text. Adjusting the target entropy allows you to control the balance between coherence and diversity in the generated text. A lower value will result in more focused and coherent text, while a higher value will lead to more diverse and potentially less coherent text. The default value is `5.0`.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 125, + "id": "709c9414-9418-4d64-9041-c0fbe63ddec6", + "metadata": {}, + "outputs": [], + "source": [ + "response = requests.post(\n", + " url=\"http://localhost:8080/completion\",\n", + " json={\n", + " \"prompt\": \"Why is the sky blue?\",\n", + " \"cache_prompt\": False,\n", + " \"stream\": False,\n", + " \"stop\": [],\n", + " \"seed\": 42,\n", + " \"n_predict\": -1,\n", + " \"n_keep\": 0,\n", + " \"min_keep\": 0,\n", + " \"ignore_eos\": False,\n", + " \"temperature\": 0.8,\n", + " \"dynatemp_range\": 0.0,\n", + " \"dynatemp_exponent\": 1.0,\n", + " \"repeat_penalty\": 1.1,\n", + " \"repeat_last_n\": 64,\n", + " \"penalize_nl\": True,\n", + " \"top_k\": 40,\n", + " \"top_p\": 0.95,\n", + " \"min_p\": 0.05,\n", + " \"typical_p\": 1.0,\n", + " \"mirostat\": 0.0,\n", + " \"mirostat_tau\": 5.0,\n", + " \"mirostat_eta\": 0.1 \n", + " \n", + " }\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 127, + "id": "97defc48-fc92-4768-a7de-82702bed5063", + "metadata": {}, + "outputs": [ + { + "data": { + "text/markdown": [ + "\n", + "\n", + "**Answer:**\n", + "\n", + "The sky is blue due to a phenomenon called **Rayleigh scattering**. \n", + "\n", + "* Sunlight is composed of all the colors of the rainbow, each with a specific wavelength.\n", + "* When sunlight interacts with molecules in the atmosphere, such as nitrogen and oxygen, the molecules scatter the light.\n", + "* Different wavelengths of light scatter differently. \n", + "* Shorter wavelengths of light, such as blue light, scatter more efficiently than longer wavelengths.\n", + "* Since blue light is scattered more evenly across the sky, we see a predominance of blue light when looking up at the sky.\n", + "\n", + "**Additional factors influencing the color of the sky:**\n", + "\n", + "* **Time of day:** The sun is higher in the sky during the day, resulting in more direct sunlight and a brighter blue sky.\n", + "* **Altitude:** Higher altitudes have thinner atmospheres, leading to less scattering and a clearer blue sky.\n", + "* **Cloud coverage:** Clouds block the sunlight and scatter less light, resulting in a less blue sky.\n", + "* **Pollution:** Pollution in the atmosphere can scatter different wavelengths of light, affecting the color of the sky." + ], + "text/plain": [ + "" + ] + }, + "execution_count": 127, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "json_data = json.loads(response.content)\n", + "_generated_text = json_data[\"content\"]\n", + "\n", + "Markdown(_generated_text)" + ] + }, + { + "cell_type": "markdown", + "id": "782c15cb-82b0-490e-a068-bf1690b52472", + "metadata": {}, + "source": [ + "#### 1.1.12 Samplers\n", + "\n", + "`samplers`: The order the samplers should be applied in. An array of strings representing sampler type names. If a sampler is not set, it will not be used. If a sampler is specified more than once, it will be applied multiple times. Default: `[\"top_k\", \"tfs_z\", \"typical_p\", \"top_p\", \"min_p\", \"temperature\"]` - these are all the available values." + ] + }, + { + "cell_type": "code", + "execution_count": 144, + "id": "f2b135e2-33c3-4178-8346-61445ef7bcca", + "metadata": {}, + "outputs": [], + "source": [ + "response = requests.post(\n", + " url=\"http://localhost:8080/completion\",\n", + " json={\n", + " \"prompt\": \"Why is the sky blue?\",\n", + " \"cache_prompt\": False,\n", + " \"stream\": False,\n", + " \"stop\": [],\n", + " \"seed\": 42,\n", + " \"n_predict\": -1,\n", + " \"n_keep\": 0,\n", + " \"min_keep\": 0,\n", + " \"ignore_eos\": False,\n", + " \"temperature\": 0.8,\n", + " \"dynatemp_range\": 0.0,\n", + " \"dynatemp_exponent\": 1.0,\n", + " \"repeat_penalty\": 1.1,\n", + " \"repeat_last_n\": 64,\n", + " \"penalize_nl\": True,\n", + " \"top_k\": 40,\n", + " \"top_p\": 0.95,\n", + " \"min_p\": 0.05,\n", + " \"typical_p\": 1.0,\n", + " \"mirostat\": 0.0,\n", + " \"mirostat_tau\": 5.0,\n", + " \"mirostat_eta\": 0.1,\n", + " \"samplers\": [\n", + " \"top_k\",\n", + " \"tfs_z\",\n", + " \"typical_p\",\n", + " \"top_p\",\n", + " \"min_p\",\n", + " \"temperature\",\n", + " ]\n", + " }\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 145, + "id": "bf2dbbd0-24f2-4d43-a5c9-a8cbae970d7c", + "metadata": {}, + "outputs": [ + { + "data": { + "text/markdown": [ + "\n", + "\n", + "**Answer:** \n", + "The sky is blue due to the process of **Rayleigh scattering**. Sunlight consists mainly of all wavelengths or colors. When sunlight interacts with molecules in Earth's atmosphere, like nitrogen and oxygen atoms these particles scatter light rays randomly without altering their direction significantly . But different color lights are scattered differently based on wavelength:\n", + "\n", + "- Shorter Wavelength (blue/violet) - gets dispersed more efficiently\n", + "_LongerWavelength_(red / infra red)_ is less effectively diffused. \n", + "\n", + "\n", + "**The blue wavelengths of sunlight:** travel in straight lines until they reach our eyes, giving us the impression that sky appears **BLUE**." + ], + "text/plain": [ + "" + ] + }, + "execution_count": 145, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "json_data = json.loads(response.content)\n", + "_generated_text = json_data[\"content\"]\n", + "\n", + "Markdown(_generated_text)" + ] + }, + { + "cell_type": "markdown", + "id": "15cec4a1-70d0-42f6-a4de-49fdbdc5dce3", + "metadata": {}, + "source": [] + }, + { + "cell_type": "markdown", + "id": "3d0c72af-2850-42a1-97ff-35eaa310a98a", + "metadata": {}, + "source": [] + }, + { + "cell_type": "markdown", + "id": "98130272-0521-4205-a8f7-a1778dfd8a92", + "metadata": {}, + "source": [ + "### 2.1 POST" + ] + }, + { + "cell_type": "code", + "execution_count": 160, + "id": "72fca804-9981-43c2-91ad-6a568ba94882", + "metadata": {}, + "outputs": [], + "source": [ + "response = requests.post(\n", + " url=\"http://localhost:8080/tokenize\",\n", + " json={\n", + " \"content\": \"Why is the sky blue?\",\n", + " \"with_special\": False,\n", + " \"with_pieces\": False\n", + " }\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 162, + "id": "4e75f7d4-6abd-49fc-a042-0901ea44e902", + "metadata": {}, + "outputs": [ + { + "data": { + "application/json": { + "tokens": [ + 6822, + 603, + 573, + 8203, + 3868, + 235336 + ] + }, + "text/plain": [ + "" + ] + }, + "execution_count": 162, + "metadata": { + "application/json": { + "expanded": false, + "root": "root" + } + }, + "output_type": "execute_result" + } + ], + "source": [ + "json_data = json.loads(response.content)\n", + "JSON(json_data)" + ] + }, + { + "cell_type": "markdown", + "id": "08888b37-7c6f-45c4-900e-d0ad031e3d70", + "metadata": {}, + "source": [ + "### 3.1 POST" + ] + }, + { + "cell_type": "code", + "execution_count": 166, + "id": "08fbc7aa-beb8-4689-b608-d2944482744e", + "metadata": {}, + "outputs": [], + "source": [ + "response = requests.post(\n", + " url=\"http://localhost:8080/detokenize\",\n", + " json=json_data\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 169, + "id": "2dc8b818-ec63-4c5c-8946-f9bca6e410be", + "metadata": {}, + "outputs": [ + { + "data": { + "application/json": { + "content": " Why is the sky blue?" + }, + "text/plain": [ + "" + ] + }, + "execution_count": 169, + "metadata": { + "application/json": { + "expanded": false, + "root": "root" + } + }, + "output_type": "execute_result" + } + ], + "source": [ + "json_response = json.loads(response.content)\n", + "JSON(json_response)" + ] + }, + { + "cell_type": "markdown", + "id": "7345659f-baf2-4ad9-ae75-3dc61b91bec7", + "metadata": {}, + "source": [ + "### 4.0 Start and embedding server\n", + "\n", + "To use the `embedding` API endpoint you must start the server using the `--embeddings` option (and without using the `--reranking` option). Open a new terminal and run the following command to start a new server with the embeddings endpoint enabled.\n", + "\n", + "```bash\n", + "MODEL=\"./models/gemma-1.1-7b-it.Q4_K_M.gguf\"\n", + "llama-server --model $MODEL --embeddings\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "ed0e34e8-52b4-41fc-aa64-3aa149bc1a10", + "metadata": {}, + "source": [ + "### 4.1 POST \n", + "\n", + "The same as [the embedding example](../embedding) does.\n", + "\n", + " *Options:*\n", + "\n", + " `content`: Set the text to process.\n", + "\n", + " `image_data`: An array of objects to hold base64-encoded image `data` and its `id`s to be reference in `content`. You can determine the place of the image in the content as in the following: `Image: [img-21].\\nCaption: This is a picture of a house`. In this case, `[img-21]` will be replaced by the embeddings of the image with id `21` in the following `image_data` array: `{..., \"image_data\": [{\"data\": \"\", \"id\": 21}]}`. Use `image_data` only with multimodal models, e.g., LLaVA." + ] + }, + { + "cell_type": "code", + "execution_count": 60, + "id": "06c9bde9-1c0a-4a51-8587-8c915e1e0d13", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + " % Total % Received % Xferd Average Speed Time Time Time Current\n", + " Dload Upload Total Spent Left Speed\n", + "100 66890 100 66863 100 27 148k 61 --:--:-- --:--:-- --:--:-- 148k\n" + ] + } + ], + "source": [ + "%%bash --out response\n", + "\n", + "curl \\\n", + " --request POST \\\n", + " --url http://localhost:8080/embedding \\\n", + " --header \"Content-Type: application/json\" \\\n", + " --data '{\"content\": \"Hello World!\"}'\n" + ] + }, + { + "cell_type": "code", + "execution_count": 61, + "id": "8c8fa15e-3116-4102-9fa8-4bd336a22894", + "metadata": {}, + "outputs": [ + { + "data": { + "application/json": { + "embedding": [ + -0.016691815108060837, + 0.01353396475315094, + 0.0005286360392346978, + 0.0022546930704265833, + 0.005019576754420996, + -0.0033065015450119972, + 0.002701298100873828, + -0.001291737542487681, + -0.005641541909426451, + 0.006711498834192753, + -0.0029747982043772936, + -0.016066377982497215, + -0.021764613687992096, + -0.001345086726360023, + -0.00619142921641469, + -0.003895261324942112, + 0.0018564768834039569, + -0.00043292174814268947, + -0.0031434649135917425, + -0.002161001553758979, + -0.01467528659850359, + -0.007372445426881313, + 0.000679797085467726, + -0.0029081457760185003, + -0.005420866888016462, + -0.0037929874379187822, + -0.01126064546406269, + -0.01345936581492424, + 0.0007580788224004209, + -0.04540044814348221, + -0.007039944175630808, + 0.006501064170151949, + 0.001213190727867186, + 0.002437830436974764, + 0.002678145421668887, + -0.0032842280343174934, + -0.0004323036700952798, + -0.0015980371972545981, + 0.006178864277899265, + 0.007071536034345627, + 0.007438831962645054, + 0.003175474004819989, + -0.011805979534983635, + -0.0001944352698046714, + -0.003893878310918808, + 0.0023756546434015036, + -0.0029270839877426624, + -0.01867782138288021, + -0.007887225598096848, + -0.005617349408566952, + -0.000736974470783025, + -0.0022182625252753496, + 6.175599992275238e-06, + 0.00023507005244027823, + -0.00262024300172925, + 0.015214318409562111, + 0.005462760105729103, + -0.0023605283349752426, + 0.020262157544493675, + 2.1792542611365207e-05, + 0.004274867009371519, + -0.0021965052001178265, + -0.003215249627828598, + 0.004076688550412655, + 0.005679779220372438, + -0.0013250095071271062, + -0.0023346804082393646, + -0.0017321596387773752, + -0.0027916315011680126, + 0.004203991498798132, + -0.005788558628410101, + -0.0011517687235027552, + -0.00040900034946389496, + -0.00695365946739912, + 0.0036133877001702785, + -0.004419748205691576, + 0.0048351287841796875, + -0.01209155935794115, + 0.001596625312231481, + -0.0009249667054973543, + 0.05373929440975189, + 0.002742484211921692, + -0.000867691240273416, + 0.009450344368815422, + -0.001657963264733553, + -0.005364298354834318, + 0.006636342499405146, + 0.059109847992658615, + 0.0020855418406426907, + 0.006336097605526447, + -0.002487695775926113, + -0.0041065337136387825, + -0.003408649004995823, + -0.07438018918037415, + 0.005515838507562876, + -0.002792150480672717, + -0.0025754400994628668, + -0.017420975491404533, + -0.002447371371090412, + 0.015288031660020351, + -0.0018180226907134056, + -0.0010581291280686855, + -0.001553489826619625, + -0.0027778679504990578, + -0.012869645841419697, + 0.005347847938537598, + -0.004029492847621441, + 0.0002725366211961955, + 0.011616114526987076, + 0.005198077764362097, + 0.007241798099130392, + -0.007629812695086002, + 0.007236812729388475, + -0.012333725579082966, + -0.004067904781550169, + -0.004639811813831329, + 0.026781227439641953, + 0.01174656581133604, + 0.013841299340128899, + 0.0013674652436748147, + -0.0012205879902467132, + 0.006203001830726862, + 0.002833098638802767, + -0.0022262909915298223, + 3.7522015190916136e-05, + 0.006965472362935543, + 0.01007911842316389, + 0.005577963311225176, + -0.0025454803835600615, + 0.014154485426843166, + -0.00813737977296114, + 0.010774482972919941, + 0.01099749282002449, + -0.015796290710568428, + -0.011346849612891674, + -0.0003506113716866821, + 0.0051916115917265415, + -0.004007267765700817, + 0.06538976728916168, + 0.0037017015274614096, + 0.0037516667507588863, + 0.0025272141210734844, + -0.0023423994425684214, + -0.001867585931904614, + 0.012503189966082573, + -0.002340231789276004, + 0.005848445929586887, + 0.0744687020778656, + 0.026523713022470474, + 0.017937825992703438, + 0.0013958235504105687, + -0.0016328543424606323, + 0.0035246373154222965, + 0.003414924954995513, + 0.017669865861535072, + 0.00711711635813117, + 0.00546979671344161, + 0.0009496257989667356, + -0.003241752739995718, + -0.009766049683094025, + -0.0037937744054943323, + -0.003518651472404599, + 0.010877164080739021, + -0.03227351978421211, + 0.00256889290176332, + 0.0005730657139793038, + -0.0021288515999913216, + 0.000318537640850991, + 0.0011975823435932398, + 0.005423439666628838, + 0.005581519100815058, + 0.0011512000346556306, + 0.004067216068506241, + 0.0005776803591288626, + -0.004376638680696487, + -0.0026396766770631075, + 0.005486404988914728, + -0.006280290428549051, + -0.0036809456069022417, + 0.004203592427074909, + -0.002472502179443836, + -0.007730977144092321, + 0.0235601793974638, + 0.005117915105074644, + 0.031188612803816795, + 0.006730908527970314, + -0.0014227255014702678, + -3.8953589864831883e-07, + -0.004429963883012533, + -0.0031885700300335884, + 0.0007912492146715522, + 0.0016735621029511094, + 0.009316185489296913, + -0.006391043774783611, + 0.002219000831246376, + 0.0089316600933671, + -0.002259679138660431, + -0.0071780504658818245, + -0.006523663178086281, + -0.0045246221125125885, + 0.0012727377470582724, + 0.007997722364962101, + 0.0012559146853163838, + 0.006116283591836691, + 0.0019437405280768871, + -0.001420222339220345, + -0.004287501331418753, + 0.017637763172388077, + -0.003479488892480731, + 0.00020795817545149475, + 0.00015081137826200575, + 0.010917307808995247, + -0.009287227876484394, + -0.0036296327598392963, + 0.1041206568479538, + 0.006474473513662815, + -0.018062414601445198, + -0.010330178774893284, + 0.001014601206406951, + 0.00348539836704731, + 0.011035698466002941, + 0.0067339432425796986, + -0.01095283217728138, + -0.004357492551207542, + 0.0008410218870267272, + -0.002891446929425001, + 0.05255068093538284, + -0.012001510709524155, + -0.011496818624436855, + -0.00741148553788662, + 0.007579215336591005, + -0.012296612374484539, + 0.010941056534647942, + -0.018230130895972252, + -0.007253160234540701, + 0.0002729124971665442, + 0.014298907481133938, + -0.002864932408556342, + -0.0020323265343904495, + 0.0033526411280035973, + 0.01573336124420166, + 0.0062594483606517315, + 0.012913184240460396, + 0.00268257106654346, + -0.030513273552060127, + -0.012343187816441059, + -0.04772941395640373, + 0.0058464594185352325, + -0.002438895870000124, + 0.005795591045171022, + -0.006912281736731529, + -0.0002621753956191242, + -0.037222348153591156, + 0.010875260457396507, + -0.0006463795434683561, + 0.003203925909474492, + 0.01031022984534502, + 0.021490156650543213, + -0.0066286129876971245, + -0.00696190632879734, + -0.0016820942983031273, + -0.00909767672419548, + -0.0020909325685352087, + -0.0045508770272135735, + 0.0029774794820696115, + -0.01130374800413847, + 0.006163346115499735, + -0.011079599149525166, + 0.00032420962816104293, + 0.003274032147601247, + -0.004169914405792952, + -0.028907379135489464, + -0.0001747927744872868, + -0.008216504007577896, + 0.0036251212004572153, + 0.010981326922774315, + -0.004171330947428942, + -0.0042035761289298534, + 0.0011444686679169536, + -0.004999988712370396, + -0.004818547982722521, + 0.0002383260289207101, + -0.0016916394233703613, + -0.008679639548063278, + -0.005084234289824963, + -0.005548567045480013, + -0.0002068041794700548, + -0.011485099792480469, + -0.0007298364653252065, + 0.0048263296484947205, + 0.021260207518935204, + 0.005519013851881027, + -0.008453214541077614, + -0.012066355906426907, + -0.0014517772942781448, + -0.007263673935085535, + 0.0032009391579777002, + -0.0005723165231756866, + -0.005844456609338522, + 0.010094596073031425, + -0.006327904295176268, + -0.011502106674015522, + -0.004657236393541098, + 0.003054052824154496, + -0.0032499239314347506, + 0.003502956824377179, + 0.001320313778705895, + -0.0026866942644119263, + -0.000738003000151366, + -0.004755166359245777, + -0.005304871592670679, + -0.005473537370562553, + -0.010516935028135777, + -0.00022832275135442615, + 0.00012870915816165507, + -0.003183787688612938, + -0.0047410521656274796, + 0.004018034320324659, + 0.006422703620046377, + -0.00196330645121634, + 0.0033091676887124777, + 0.00395999988541007, + -0.0037603983655571938, + 0.0042784023098647594, + -0.002017900813370943, + -0.0027135005220770836, + -0.009670293889939785, + -0.008033736608922482, + 0.0013336553238332272, + 0.005742968991398811, + 0.009963970631361008, + -0.0024679352063685656, + 0.044488709419965744, + 0.007383097428828478, + -0.008992749266326427, + -0.006281279493123293, + -0.0009441525326110423, + -0.015403710305690765, + -0.003507092595100403, + -0.001304378965869546, + -0.02532416209578514, + 0.009118372574448586, + -0.0025721986312419176, + 0.0034252635668963194, + -0.002436988288536668, + 0.00396233145147562, + 0.006322804372757673, + 0.0027611267287284136, + -0.0012828478356823325, + 0.005372586194425821, + -0.0018310124287381768, + 0.005829977802932262, + 0.002988129621371627, + -0.007413119543343782, + 0.0011878563091158867, + -0.0056642815470695496, + 0.014065143652260303, + -0.0008888810989446938, + -0.003395424224436283, + 0.022484328597784042, + 0.005293137859553099, + -0.004303166177123785, + -0.0016756963450461626, + 0.004317742772400379, + -0.006640130188316107, + 0.010855346918106079, + 0.002745922189205885, + -0.001840862212702632, + 0.0066020251251757145, + -0.002866660477593541, + -0.003107166849076748, + -0.0028237062506377697, + -0.0013045576633885503, + -0.0030959523282945156, + 0.00033533654641360044, + 0.003844769671559334, + 0.006298932712525129, + -0.003099677851423621, + 0.0027701961807906628, + 0.012398917227983475, + -0.00417623296380043, + -0.0015904551837593317, + -0.008166748099029064, + -0.0031818607822060585, + 0.0006662089144811034, + -0.07378330081701279, + 0.007716508116573095, + 0.006780714262276888, + 0.0055574593134224415, + 0.0023181408178061247, + 3.691236997838132e-06, + 0.008251695893704891, + -0.012151617556810379, + 0.02148384600877762, + -0.0027790984604507685, + -0.017472904175519943, + 0.013290949165821075, + -9.795537334866822e-05, + -0.0017418661154806614, + 0.0017502899281680584, + 0.0030841249972581863, + -0.00012178585166111588, + -0.005708194803446531, + -0.011363169178366661, + -0.0015771579928696156, + -0.002730628941208124, + -0.004779500886797905, + 0.00033807320869527757, + 0.0016176165081560612, + -0.00904501136392355, + -0.012651878409087658, + -0.00928883720189333, + 0.0016526158433407545, + 0.0005097776302136481, + -0.009333562105894089, + 0.0017069162568077445, + -6.348367605824023e-05, + 0.004850560333579779, + 0.005213040392845869, + -0.00021263767848722637, + 0.003730935277417302, + 0.00390068837441504, + -0.0004051658615935594, + 0.001501085702329874, + -0.007214853540062904, + 0.00013992865569889545, + 0.0022074230946600437, + 0.0018628177931532264, + -0.0018866648897528648, + 0.0009352646884508431, + 0.002517240820452571, + 0.008834946900606155, + -0.013917365111410618, + 0.00696506816893816, + 0.009112520143389702, + -0.00313444877974689, + 0.007636948488652706, + 0.0032175639644265175, + -0.004560177214443684, + -0.004405266139656305, + 0.00803668238222599, + 0.0030415114015340805, + -0.004789851605892181, + 0.005590552929788828, + 0.00999379251152277, + -0.010243297554552555, + -0.008400356397032738, + -0.00524148577824235, + -0.001142467255704105, + -0.010885907337069511, + -0.0003708303556777537, + -0.003323251148685813, + 0.004744783043861389, + -0.012994598597288132, + -0.0006703034741804004, + -0.0031907439697533846, + 0.0005279472679831088, + -0.020224330946803093, + -0.0035017828922718763, + 0.0018700992222875357, + 0.0017709218664094806, + 0.0053741345182061195, + 0.06544419378042221, + -0.004268350079655647, + 0.0006338932435028255, + -0.004635213408619165, + -0.006295817904174328, + 0.0016773691168054938, + -0.0029084559064358473, + -0.004751416388899088, + -0.0023221243172883987, + -0.0005130519857630134, + 0.004576080944389105, + -0.006996459793299437, + 0.01050998643040657, + -0.0008973521762527525, + -0.007741020526736975, + -0.0008757025934755802, + -0.010819775983691216, + -0.0014071600744500756, + 0.004178172908723354, + -0.010135957971215248, + 0.0026928444858640432, + -0.004149917047470808, + -0.0015563792549073696, + -0.0043048071675002575, + 0.002588951261714101, + -0.009181993082165718, + -0.01435770746320486, + -4.048843038617633e-06, + 0.00039341303636319935, + 0.001259832177311182, + -0.008778656832873821, + 0.0015578403836116195, + -0.0021020425483584404, + 0.004204847849905491, + 0.00381667772307992, + 0.0023079717066138983, + -0.006745347753167152, + -0.00932589452713728, + 0.0037208476569503546, + -0.00581307802349329, + 0.00824673380702734, + 0.0015649981796741486, + -0.004105595871806145, + -0.008889354765415192, + -0.003913403023034334, + -0.006928666960448027, + 0.006639149971306324, + -0.0023721044417470694, + 0.004509610589593649, + -0.009850598871707916, + 0.003060088027268648, + 0.0016182720428332686, + -0.004082279279828072, + 0.010127912275493145, + 0.011927383951842785, + -0.003431013785302639, + 0.010287552140653133, + 0.011843010783195496, + 9.26089778658934e-05, + 0.003157737199217081, + -0.004973393399268389, + 0.0015192795544862747, + -0.0024812831543385983, + 0.0022562756203114986, + 0.0013324745232239366, + 0.0021341186948120594, + 0.0008584160823374987, + -0.0018841322744265199, + 0.010497735813260078, + -0.0003710891760420054, + -0.004758571740239859, + 0.004884479567408562, + -0.0011196094565093517, + -0.008533758111298084, + 0.004967975430190563, + -0.0033115653786808252, + 0.005270267371088266, + 0.008875074796378613, + -0.001434103469364345, + -0.008718056604266167, + 0.0033262353390455246, + -0.008628354407846928, + -0.012275120243430138, + -0.011252743192017078, + 0.0019706464372575283, + -0.005874264985322952, + -0.00849432684481144, + -0.01502551045268774, + -0.009168540127575397, + -0.009515310637652874, + 0.008797407150268555, + 0.00412955554202199, + 0.0038905167020857334, + 0.004767159465700388, + 0.016112277284264565, + 0.006841691210865974, + 0.001970874145627022, + -0.006237213034182787, + -0.0012680948711931705, + 0.019433140754699707, + 0.007497277110815048, + -0.0008799081551842391, + -0.0024970516096800566, + -0.003482073312625289, + 0.0015126868383958936, + -0.006169648375362158, + 0.028830865398049355, + -0.010417575016617775, + -0.00458157155662775, + 0.006362073123455048, + -0.002907315269112587, + -0.0022949797566980124, + -0.008226771838963032, + 0.009061980061233044, + -0.004374227952212095, + -0.0014616046100854874, + -0.0010643417481333017, + -0.007427423261106014, + -0.001435783808119595, + -0.0015958737349137664, + 0.013629170134663582, + 0.010984743013978004, + 0.0007380440947599709, + -0.00418100506067276, + 0.009636635892093182, + 0.0038031181320548058, + -0.0024429927580058575, + -0.013192386366426945, + -0.0074958945624530315, + 0.006732451729476452, + -0.0025839393492788076, + -0.013237245380878448, + 0.010615359991788864, + 0.0073325359262526035, + -0.001999655971303582, + -0.0013783696340397, + 0.0011986829340457916, + 0.0014196165138855577, + 0.008083580061793327, + 0.005481775850057602, + 0.004112789873033762, + 0.0019188629230484366, + 0.0010879234177991748, + 0.004705488216131926, + 0.003936820663511753, + 0.0006080573075450957, + 0.004554424434900284, + 0.0052956342697143555, + -0.0005416609346866608, + -0.013980923220515251, + 0.004035135265439749, + -9.227216651197523e-05, + -0.009391387924551964, + -0.008959934115409851, + 0.019807366654276848, + 0.00012165367661509663, + -0.0024310217704623938, + 0.010532321408390999, + -0.005862573627382517, + -0.013861026614904404, + 0.013179617933928967, + -0.009420707821846008, + -0.0058541069738566875, + -0.002123833866789937, + 0.006517954170703888, + 0.002239057095721364, + 0.004573325626552105, + 0.009697231464087963, + -0.01004691980779171, + -0.0008531531202606857, + -0.004855507984757423, + 0.008584403432905674, + -0.007500269450247288, + 0.007460806518793106, + -0.00557160098105669, + -0.004880242981016636, + 0.0012549875536933541, + 0.010027077049016953, + -0.004340615589171648, + -0.005073028616607189, + -0.009075128473341465, + -0.003008087631314993, + -0.002647536341100931, + -0.0016998255159705877, + -0.006568207871168852, + -0.004581665154546499, + 0.0044904425740242004, + -0.008051132783293724, + -0.009976076893508434, + 0.00513707147911191, + -0.0013551372103393078, + -0.0009053841349668801, + 0.005333632230758667, + 0.0033493090886622667, + -0.005758712533861399, + -0.004817752167582512, + -0.0032870026770979166, + -0.0006354507640935481, + 0.006958176381886005, + -0.020290208980441093, + -0.0030405023135244846, + -0.01314330380409956, + -0.0035451622679829597, + 0.00254567782394588, + -0.006503895856440067, + -0.003305578138679266, + 0.005178701598197222, + -0.0026970766484737396, + 0.0008090371266007423, + 0.005231946241110563, + -0.003517970209941268, + 0.019121740013360977, + 0.003543980186805129, + -0.0088187949731946, + -0.00301041966304183, + -0.0012112214462831616, + -0.0010646468726918101, + 0.006679198704659939, + -0.005159217864274979, + 0.004281106870621443, + 0.0042044250294566154, + 0.001179052167572081, + 0.00891931727528572, + 0.0024967417120933533, + 0.011828538030385971, + -3.1908897653920576e-05, + 0.0035989610478281975, + 0.0017310775583609939, + -0.009883924387395382, + 0.0048067024908959866, + -0.00635627331212163, + -1.4643099348177202e-05, + -0.0018626288510859013, + -0.012342093512415886, + 0.0012472809758037329, + -0.0023294638376682997, + -0.006338710431009531, + 0.012458349578082561, + -0.0015690767904743552, + -0.013279586099088192, + -0.0029330311808735132, + -0.005478012841194868, + 0.012823105789721012, + 0.006742158904671669, + 0.004295679274946451, + -0.0052001504227519035, + 0.005698809400200844, + 0.007263155188411474, + 0.00445031002163887, + 0.0044499654322862625, + 0.0006411183276213706, + -0.0071014887653291225, + 0.005900822579860687, + -0.005532861687242985, + 0.006957576144486666, + -0.004234375897794962, + -0.013902394101023674, + -0.0008472828194499016, + -0.011523361317813396, + -0.0001008320614346303, + 0.004253433085978031, + 0.005674802232533693, + 0.015035443007946014, + 0.0031998888589441776, + 0.0038819515611976385, + 0.008766090497374535, + -0.001916261506266892, + -0.0013124559773132205, + 0.006529737263917923, + 0.005319028161466122, + 0.0022156829945743084, + -0.0074821654707193375, + -0.008862338028848171, + 0.012487225234508514, + 0.00165111618116498, + -0.004681046586483717, + 0.01385333202779293, + 0.0052698818035423756, + -0.0012595306616276503, + 0.0019106895197182894, + -0.00012800542754121125, + -0.006115707103163004, + -0.008558982983231544, + 0.00010553691390668973, + 0.011052055284380913, + -0.024913594126701355, + -0.010155114345252514, + -0.0032212373334914446, + 0.0001927243865793571, + -0.008391710929572582, + 0.00038052204763516784, + 0.004848336800932884, + 0.000405360187869519, + -0.003093788865953684, + -0.004935477860271931, + 0.007340260315686464, + -0.004045027308166027, + 0.0014190376969054341, + -0.0028456333093345165, + 0.007906819693744183, + -0.0036763036623597145, + -0.0010627037845551968, + 0.0033251226413995028, + -0.003456282429397106, + -0.010924027301371098, + -0.01065688207745552, + 0.0010935134487226605, + -0.016316181048750877, + -0.004303843714296818, + -0.0009670730796642601, + 0.006325125228613615, + -0.013782457448542118, + -0.011620976962149143, + -0.002853556303307414, + -0.0014971619239076972, + -0.00259526283480227, + -0.006022829562425613, + 0.007313931360840797, + -0.006455709226429462, + 0.003244257066398859, + -0.005259925965219736, + -0.001788472756743431, + -0.00045550090726464987, + 0.004497219808399677, + -0.00884396955370903, + -0.005673953332006931, + 0.000191482191439718, + 0.003915209323167801, + -7.238391117425635e-05, + -0.0033054847735911608, + 0.0009779115207493305, + -0.013065466657280922, + -0.0016917882021516562, + 0.010755862109363079, + -0.01407881174236536, + 0.001180801191367209, + -0.002972627291455865, + 0.0006689922884106636, + 0.00420592725276947, + 0.0045756078325212, + 0.00951801985502243, + 0.010377326048910618, + 0.005004559643566608, + 0.004160729702562094, + -0.0066857971251010895, + -0.00531903188675642, + -0.004941556137055159, + 0.004559643100947142, + -0.00888486672192812, + 0.0033584244083613157, + 0.0013154296902939677, + 0.004009073134511709, + -0.0044822534546256065, + -0.0013354317052289844, + 0.006955130957067013, + 0.0032718884758651257, + -0.010469621978700161, + -0.0008769754786044359, + -0.00375533290207386, + -0.007959171198308468, + -0.0006990977562963963, + 0.008101058192551136, + -0.005439263302832842, + 0.00448206439614296, + 0.005599682219326496, + 0.0007496365578845143, + 0.007223326247185469, + -0.002956440905109048, + -0.007577643729746342, + 0.004645063541829586, + -0.003481571562588215, + -0.005560653749853373, + -0.0052477894350886345, + -0.003683076472952962, + 0.012808829545974731, + 0.015324798412621021, + -0.002890538889914751, + -0.0030725765973329544, + -0.0046875434927642345, + -0.005498741287738085, + 0.009332443587481976, + 0.00017547610332258046, + 0.004975724034011364, + -0.006718057673424482, + -0.01342262513935566, + -0.010953485034406185, + -0.0007441360503435135, + 0.0022952647414058447, + -0.010635294951498508, + 0.015677032992243767, + -0.0004710805951617658, + 0.013873151503503323, + 0.0039034648798406124, + 0.008472669869661331, + -0.005409275181591511, + -0.0022540485952049494, + -0.0075464071705937386, + -0.008584079332649708, + 0.006261288188397884, + -0.00868950318545103, + -0.014880198985338211, + -0.014076012186706066, + -0.0035783248022198677, + 0.0010252039646729827, + 0.0036185227800160646, + 0.0061086066998541355, + 0.009998559951782227, + -0.008242080919444561, + -0.0030109616927802563, + -0.002600926673039794, + -0.006494627799838781, + -0.0018270729342475533, + -0.009460987523198128, + 0.0089951166883111, + -0.00013563316315412521, + -0.012032794766128063, + 0.001581308082677424, + -0.0013083834201097488, + -0.004575630649924278, + -0.011713402345776558, + 0.00708789611235261, + -0.003769077593460679, + 0.009155511856079102, + -0.0030536686535924673, + 0.005325945559889078, + 0.008409510366618633, + 0.007593734189867973, + -0.002981589175760746, + 0.015574744902551174, + 0.013406253419816494, + -0.007782661356031895, + 0.00046791377826593816, + 0.0041423579677939415, + -0.00600207457318902, + -0.009396583773195744, + -0.0029960889369249344, + -0.0016694461228325963, + 0.0022266607265919447, + -0.0053502838127315044, + -0.0025448219384998083, + 0.0006819419795647264, + 0.0045137847773730755, + 0.0018820265540853143, + -0.015343579463660717, + -0.007254385855048895, + 0.0032301321625709534, + 0.0024715519975870848, + -0.0017838459461927414, + 0.005395916756242514, + -0.013298125006258488, + -0.007694025058299303, + 0.007645234931260347, + -0.00790442619472742, + -0.008709387853741646, + 0.0038239748682826757, + 0.0048072622157633305, + 0.00414207624271512, + -0.0021162955090403557, + 0.007330236025154591, + -0.009159938432276249, + 0.007468640338629484, + -0.0019868556410074234, + 0.00047224361333064735, + -0.015261390246450901, + -0.0016090278513729572, + 0.000239241257077083, + -0.00587452994659543, + 0.01032661646604538, + 0.000848420022521168, + 0.0009467564523220062, + -0.0020163850858807564, + -0.004332459531724453, + 0.0010139113292098045, + -1.6160085579031147e-05, + -0.010368989780545235, + 0.00255385204218328, + -0.004299737513065338, + 0.008451706729829311, + -0.009625183418393135, + -0.00623505562543869, + -0.008018402382731438, + -0.010874593630433083, + -0.00709332013502717, + 0.0092451311647892, + -0.004106960725039244, + 0.001774117350578308, + 0.007145293988287449, + -0.004732620436698198, + 0.00943777710199356, + 0.005813910160213709, + -0.003285700222477317, + -0.011491706594824791, + -0.0012033450184389949, + -0.0067494772374629974, + -0.005218437407165766, + 0.0009216160397045314, + 0.00020819916971959174, + -0.007254370488226414, + -0.004683197475969791, + 0.014902747236192226, + -0.007506680209189653, + -0.002682572929188609, + 0.007887219078838825, + 0.006274892017245293, + 0.004029561765491962, + -0.004458457697182894, + -0.0005620780866593122, + -0.004858884494751692, + 0.0076308948919177055, + -0.004756782203912735, + -0.003293300047516823, + -0.008576987311244011, + 0.008586880750954151, + 0.0017769227270036936, + 0.01094120740890503, + -0.006920505315065384, + 0.0015367333544418216, + 0.012017340399324894, + 0.005818760953843594, + -0.0007150138262659311, + 0.004750472493469715, + 0.01306771021336317, + 0.011078893207013607, + -0.007533449213951826, + -0.007629431784152985, + 0.0036741034127771854, + 0.005320089869201183, + 0.0039653233252465725, + 0.005160897504538298, + 0.002478482434526086, + 0.0018242319347336888, + -0.008423643186688423, + -0.016192743554711342, + 0.00533665856346488, + 0.012053236365318298, + 0.0011990423081442714, + 0.00962132215499878, + -0.007110932841897011, + -0.012923607602715492, + 0.0014490054454654455, + -0.010599211789667606, + -0.004705649334937334, + 0.00497483043000102, + 0.007810813374817371, + 0.002292785095050931, + -0.0005185049958527088, + 0.0018136966973543167, + 0.001182951033115387, + -0.0003079126763623208, + -0.007074153982102871, + -0.0038540672976523638, + 0.012966513633728027, + 0.0052431062795221806, + -0.0003236772899981588, + 0.006476663518697023, + -0.00877076294273138, + -0.003418942913413048, + -0.010861431248486042, + 0.0005448712618090212, + 0.005287555977702141, + -0.008368263952434063, + -0.004529312718659639, + 0.010184310376644135, + 0.0011370169231668115, + 0.008495192043483257, + 0.006970366928726435, + -0.006171541754156351, + -0.0045442162081599236, + -0.003039824776351452, + -0.011136443354189396, + 0.009454911574721336, + -0.003205829067155719, + 0.0012546941870823503, + 0.003537577809765935, + 0.005072897300124168, + 0.004574313294142485, + -0.016366655007004738, + 0.0019625460263341665, + -0.001799440709874034, + 0.0027893451042473316, + 0.0024382416158914566, + -0.0044859773479402065, + -0.011568304151296616, + 0.003419100306928158, + -0.0015598738100379705, + 0.01404453907161951, + 0.002212697407230735, + 0.0071008154191076756, + 0.0005661696777679026, + 0.0013603102415800095, + -0.012407624162733555, + -0.0002828588185366243, + -0.006441982928663492, + 0.003305988386273384, + 0.010527020320296288, + 0.006095643155276775, + 0.01553469616919756, + 0.0036230788100510836, + 0.0003580494085326791, + 0.008663877844810486, + -0.011331421323120594, + 0.0026019758079200983, + -0.005827340297400951, + -0.006571131758391857, + -0.0017381098587065935, + 0.007124103605747223, + -0.006578135769814253, + -0.005990130361169577, + -0.001448437455110252, + -0.0021541190799325705, + -0.0026884654071182013, + 0.005148289725184441, + -0.00792270340025425, + -0.005985869560390711, + -0.012292478233575821, + -0.015555075369775295, + -0.007382126525044441, + -0.006122658960521221, + -0.0028073263820260763, + 0.009751662611961365, + 0.00775795616209507, + 0.010382267646491528, + -0.006740340497344732, + -0.00019949469424318522, + 0.007486827205866575, + -0.008323065005242825, + -0.006997435819357634, + -0.0018101605819538236, + 0.005859236232936382, + 0.0022890069521963596, + 0.0016130262520164251, + 0.004060777835547924, + -0.012992545031011105, + 0.009401347488164902, + 0.002310930984094739, + 0.005972812417894602, + -0.00258693378418684, + -0.009889035485684872, + 0.01514944713562727, + 0.00861616712063551, + 0.004512110725045204, + 0.004499361384660006, + 0.004984723869711161, + -0.0034040873870253563, + 0.007722940761595964, + -0.006337055936455727, + 0.005977279506623745, + 0.0010242338757961988, + -0.009727072902023792, + -0.009072580374777317, + -0.005614754278212786, + -0.007767862640321255, + -1.102052647183882e-05, + 0.011266474612057209, + -0.0024216254241764545, + 0.011951934546232224, + 0.00814477913081646, + -0.0037616267800331116, + 0.0004910276038572192, + 0.004466355312615633, + 0.0004781950847245753, + 0.00606703944504261, + -0.0004853274440392852, + -0.0037267382722347975, + -0.0020808170083910227, + -0.027156338095664978, + 0.0055329822935163975, + 0.00443290313705802, + 0.015664106234908104, + -0.008602006360888481, + 0.0010693850927054882, + -0.015589714050292969, + 0.003404370043426752, + 0.0022057024762034416, + -0.00847364030778408, + -0.0034888419322669506, + 0.0005598272546194494, + 0.0029398652259260416, + 0.00015781287220306695, + -0.008274521678686142, + -0.00872968602925539, + -0.0011751620331779122, + -0.003170840907841921, + 0.010618877597153187, + -0.0013215310173109174, + 0.0023429652210325003, + -0.0035707904025912285, + -0.005936841014772654, + 0.00651887571439147, + 0.017291812226176262, + 0.00635509705170989, + -0.006386339198797941, + -0.0032616043463349342, + -0.000876853649970144, + 0.00645739259198308, + -0.006240484770387411, + -0.0030444145668298006, + 0.0069710626266896725, + -0.002131006447598338, + 0.004761108197271824, + 0.010619565844535828, + 0.0002884978021029383, + -0.0025578364729881287, + -0.008723091334104538, + -0.004135231487452984, + -0.01132290717214346, + 0.009583607316017151, + 0.005000806413590908, + 0.004706087056547403, + -0.004484477918595076, + 0.007497622165828943, + -0.006696829106658697, + 0.0016789737856015563, + 0.0035449215210974216, + 0.004013401456177235, + -0.013775824569165707, + 0.0015111053362488747, + 0.0054056597873568535, + -0.013833271339535713, + -0.009523863904178143, + 0.0020716034341603518, + 0.005795680917799473, + 0.006457724142819643, + 0.0029905345290899277, + 0.010606620460748672, + 0.01264605950564146, + 0.004826867952942848, + -0.0036902730353176594, + -0.006884979549795389, + -3.8880592910572886e-05, + 0.006551374681293964, + -0.0015271044103428721, + 0.012920549139380455, + 0.002527123549953103, + -0.013632331974804401, + -0.007262096274644136, + 0.003718034364283085, + 0.0008227885700762272, + 0.008427895605564117, + -0.0014859925722703338, + -0.0023044634144753218, + 0.005106368102133274, + -0.0029879482463002205, + -0.018711837008595467, + 0.0007791129755787551, + 0.009639788419008255, + -0.011805147863924503, + 0.009582538157701492, + -0.008581837639212608, + 0.00412911269813776, + 0.001647024299018085, + -0.0034719076938927174, + 0.0009410156635567546, + -0.011256382800638676, + -0.002832163358107209, + 0.007371820509433746, + 0.002762062707915902, + -0.004429753869771957, + 0.02023034729063511, + 0.002229555044323206, + 0.0032519849482923746, + 0.005625267047435045, + 0.0013388636289164424, + -0.0008759830379858613, + -0.0015073948306962848, + -0.010841967537999153, + 0.01785854436457157, + -0.011725693941116333, + 0.0031006354838609695, + 0.009422640316188335, + 0.01406132522970438, + -0.007215623278170824, + -0.006312900688499212, + -0.028516463935375214, + -0.003965907264500856, + 0.005532995797693729, + -0.0005210262024775147, + -0.0051738191395998, + -0.009595165960490704, + 0.006491640117019415, + 0.0029380160849541426, + -0.01269043143838644, + -0.0021494843531399965, + -0.01402988936752081, + 0.006243823561817408, + 0.004421751480549574, + 0.008926856331527233, + -0.0029255289118736982, + -0.0023679460864514112, + 0.004984189290553331, + -0.011895300820469856, + 0.003388667944818735, + -0.010041838511824608, + -0.009868424385786057, + 0.0017281427280977368, + 0.0016227646265178919, + 0.014865092001855373, + 0.007011198904365301, + 0.0038321693427860737, + -0.007722330745309591, + -0.002501483540982008, + -0.005595006048679352, + -0.0031197352800518274, + 0.006603735499083996, + -0.0005507614696398377, + -0.009961619041860104, + -0.008877359330654144, + -0.0051931836642324924, + -0.0009240598883479834, + 0.00944636482745409, + 0.0031887360382825136, + -0.01105413306504488, + 0.01337040588259697, + -0.008074753917753696, + -0.0010179384844377637, + -0.0061756521463394165, + 0.005011307075619698, + -0.017274100333452225, + -0.0187551137059927, + -0.0022944083902984858, + 0.0031177792698144913, + -0.0010313639650121331, + -0.006281699053943157, + -0.005068426951766014, + 0.0012871905928477645, + 0.001267669373191893, + -0.014951231889426708, + -0.004701228812336922, + 0.0047660525888204575, + -0.007747045252472162, + 0.004883871879428625, + -0.0006516496650874615, + -0.007842726074159145, + 0.003260647412389517, + 0.014831694774329662, + -0.010893717408180237, + 0.005050728563219309, + -0.008120191283524036, + 0.008512047119438648, + -0.0018415491795167327, + -0.005636206362396479, + 0.0038360380567610264, + 0.010659086517989635, + -0.011269303038716316, + 0.002621374325826764, + -0.007506466470658779, + -0.001259872573427856, + -0.001979956403374672, + -0.0013706099707633257, + -0.003415324492380023, + -0.007840101607143879, + 0.0037580474745482206, + 0.001607053098268807, + 0.007509700488299131, + -0.007351321633905172, + 0.017843147739768028, + -0.001144037232734263, + -0.004526825621724129, + -0.001200130325742066, + 0.006665518507361412, + -0.0051813675090670586, + 0.005431767553091049, + 0.0005589803331531584, + 0.005622186232358217, + -0.015218771994113922, + -0.00803647842258215, + 0.0011455841595306993, + 0.0013209314784035087, + 0.005262125749140978, + -0.01012266706675291, + 0.00615684874355793, + 0.004462164826691151, + -0.007463446352630854, + -0.001953691942617297, + 0.00419241888448596, + -0.0037319508846849203, + 0.0016988383140414953, + 0.005682630464434624, + 0.002447637729346752, + -0.006174507085233927, + -0.00024310594017151743, + 0.005639309529215097, + 0.010044471360743046, + 0.0012956535210832953, + -0.002279871143400669, + 0.008101190440356731, + -0.00010356950951972976, + 0.0004419383767526597, + -0.0023014883045107126, + -0.005942102987319231, + 0.012370489537715912, + -0.012633545324206352, + -0.003133448539301753, + -0.009865677915513515, + -0.019486594945192337, + -0.002751620952039957, + 0.007066846359521151, + 0.004107546992599964, + 0.0005496126250363886, + -4.8372730816481635e-06, + -0.004981194157153368, + 0.003596334485337138, + -0.012431349605321884, + 0.0051045711152255535, + -0.017827464267611504, + -0.015726180747151375, + -0.00029753134003840387, + 0.008109204471111298, + 0.008477026596665382, + 0.007234628312289715, + 0.0019195065833628178, + -0.0013932993169873953, + -0.015935296192765236, + -0.028162037953734398, + 0.005909111816436052, + 0.01429544948041439, + -0.0074547892436385155, + -0.005278146360069513, + 0.0046626729890704155, + 0.0029372642748057842, + -0.009525937028229237, + -0.00044128307490609586, + -0.0024703785311430693, + -0.002434107009321451, + 0.012593734078109264, + -0.010544289834797382, + 0.01280398853123188, + 0.006321474444121122, + 0.006467323750257492, + -0.003264704719185829, + 0.08968812972307205, + -0.002123340731486678, + 0.00757436640560627, + 0.013458576053380966, + 0.00586737971752882, + 0.005672345403581858, + -0.021804554387927055, + -0.005062283482402563, + -0.007606789469718933, + 0.013979868032038212, + -0.006238226313143969, + -0.0019953653682023287, + -0.013504130765795708, + 0.014128257520496845, + 0.008208147250115871, + 0.012294182553887367, + 0.0026711090467870235, + 0.011121443472802639, + -0.014445901848375797, + 0.00938729289919138, + 0.006403153762221336, + 0.0010995193151757121, + 0.00048243047785945237, + -0.0005462872795760632, + 0.0075255329720675945, + 0.009733923710882664, + -0.004892534576356411, + 0.010384180583059788, + 0.018981412053108215, + 0.0030240474734455347, + -0.007413453888148069, + -0.005608236417174339, + -0.002660992555320263, + -0.00286229164339602, + 0.0033294956665486097, + -0.00158896972425282, + -0.0033853226341307163, + -0.005536872893571854, + 0.017998456954956055, + 0.014070933684706688, + -0.014108746312558651, + -0.010097281076014042, + 0.0032791071571409702, + 0.020223675295710564, + -0.01236297469586134, + 0.00028016461874358356, + 0.0001356107386527583, + -0.004567338153719902, + 0.000529189535882324, + -0.01765056699514389, + -0.005661042872816324, + 0.0037472755648195744, + 0.016767900437116623, + -0.017445052042603493, + -0.0025775397662073374, + 0.007045150268822908, + -0.03870919719338417, + -0.010540024377405643, + 0.00751414755359292, + -0.0035645761527121067, + -0.002109292894601822, + -0.005665064789354801, + -0.005040545482188463, + -0.0038073030300438404, + -0.0034218812361359596, + -0.014086718671023846, + 0.0197200458496809, + 0.013412904925644398, + 0.004005453083664179, + -0.010976243764162064, + 0.010068053379654884, + -0.004781192168593407, + -0.00014980936248321086, + -0.011195538565516472, + -0.009372333995997906, + 0.0031252610497176647, + -0.013539683073759079, + 0.01010187342762947, + -0.013973044231534004, + -0.006882590241730213, + -0.0035566803999245167, + 0.0027750427834689617, + 0.004735514521598816, + 0.006623853929340839, + -0.004685145802795887, + 0.005462748929858208, + 0.002610529540106654, + -0.0029015815816819668, + 0.003447066294029355, + -0.009465587325394154, + 0.0021304830443114042, + -0.00028013080009259284, + -0.002261953894048929, + 0.0047617098316550255, + -0.007251368369907141, + 0.009409993886947632, + 0.006310706026852131, + 0.003944790456444025, + 0.00919287744909525, + -0.014160003513097763, + 0.007466122042387724, + 0.00239109736867249, + 0.0053621092811226845, + -0.012584265321493149, + -0.009129802696406841, + 0.018720323219895363, + 0.003690346609801054, + 0.0051151663064956665, + -0.015812670812010765, + -0.00814890768378973, + -0.010097458958625793, + 0.018148226663470268, + -0.0013967101695016026, + 0.0034951362758874893, + -0.0025764883030205965, + 0.005304561462253332, + -0.005956816487014294, + -0.0054127867333590984, + 0.005625816527754068, + 0.00648140674456954, + -0.007031261920928955, + 0.0034379668068140745, + 0.009892882779240608, + 0.00921961572021246, + 0.006665147375315428, + 0.007036617957055569, + -0.0053828260861337185, + -0.01400783658027649, + 0.007728900294750929, + -0.011926460079848766, + 0.01053609699010849, + -0.0047254557721316814, + 0.0005419956287369132, + 0.031016966328024864, + 0.004485748708248138, + -0.003301456104964018, + 0.021917160600423813, + -0.009594717063009739, + 0.0030612279660999775, + -0.002655526390299201, + 0.012688752263784409, + -0.006709670182317495, + 0.0037272071931511164, + 0.0033697281032800674, + 0.0020614301320165396, + -0.002655679825693369, + -0.0034524924121797085, + -0.0006054005934856832, + -0.0077881282195448875, + 0.007221587467938662, + -0.000240651163039729, + 0.015786781907081604, + -0.01136830821633339, + 0.01029540691524744, + 0.00809422880411148, + -0.013204437680542469, + -0.0020873064640909433, + 0.008633364923298359, + 0.008530329912900925, + -0.003981472924351692, + -0.01182467583566904, + -0.010850759223103523, + -0.0012346161529421806, + 0.002297420520335436, + 0.01305827870965004, + -0.003098822431638837, + -0.0010457840980961919, + -0.03584853187203407, + 0.005491174757480621, + 0.004023421090096235, + -0.006955747026950121, + -0.00569462263956666, + -0.007534045726060867, + -0.01122236903756857, + -0.00114327238406986, + 0.12370886653661728, + -0.00013022329949308187, + 0.008737882599234581, + -0.016715476289391518, + -0.36611756682395935, + 0.010507510043680668, + 0.011237625032663345, + -0.0025784787721931934, + -0.0026433777529746294, + 0.00583630008623004, + 0.001432381453923881, + -0.006116296164691448, + -0.009873577393591404, + 0.021472865715622902, + 0.004614025820046663, + 0.02113865129649639, + -0.009876460768282413, + 0.008408412337303162, + 0.00042479849071241915, + -0.004929686430841684, + 0.007433453109115362, + 0.008812565356492996, + -0.00209056050516665, + 0.015333173796534538, + 0.00512704299762845, + 0.016691775992512703, + 0.00045360976946540177, + 0.010579273104667664, + 0.0026121262926608324, + 0.004913663491606712, + -0.004938516300171614, + 0.0011210038792341948, + 0.014439012855291367, + -0.0033875107765197754, + -2.2723999791196547e-05, + 0.0031730104237794876, + 0.012377092614769936, + 0.0012974687851965427, + -0.017906228080391884, + -0.012139485217630863, + 0.5289421677589417, + 0.006395980715751648, + 0.009625923819839954, + 0.001873208093456924, + 0.004188534803688526, + 0.0042366692796349525, + -0.0014684285270050168, + 0.016123708337545395, + -0.0056155975908041, + 0.004481083247810602, + -0.0045138015411794186, + 0.009984553791582584, + -0.0010395189747214317, + -0.007884121499955654, + -0.0024730267468839884, + 0.008171331137418747, + 0.014064162969589233, + 0.009708762168884277, + 0.029658302664756775, + -0.01846492663025856, + -0.0016243085265159607, + 0.0035359312314540148, + -0.0033120920415967703, + 0.01750701665878296, + 0.0011478407541289926, + 0.012482285499572754, + -0.012060444802045822, + -0.022026991471648216, + -0.02366352267563343, + -0.014284327626228333, + 0.00400924077257514, + -0.015069601126015186, + 0.0013096706243231893, + -0.008724885992705822, + 0.004924492444843054, + -0.007287654560059309, + 0.00972650945186615, + 0.01753499172627926, + -0.009635001420974731, + 0.0005938126123510301, + -0.0074736508540809155, + -0.025485161691904068, + 0.0008988170884549618, + 0.00428059371188283, + -0.003220862476155162, + -0.00707972189411521, + -0.003193291136994958, + -0.00339619442820549, + 0.0024774204939603806, + -0.006847603712230921, + -0.016872137784957886, + 0.007158909924328327, + -0.004477754235267639, + -0.001433573430404067, + 0.0140788359567523, + -0.04499025270342827, + -0.002193036023527384, + 0.03127351775765419, + -0.005710539408028126, + 0.0031705121509730816, + 0.005924334283918142, + 0.010606142692267895, + 0.0038958352524787188, + -0.0030610004905611277, + -0.007734555285423994, + 0.00042446618317626417, + -0.003981824032962322, + 0.005424409173429012, + -0.00979658868163824, + 0.0012327802833169699, + 0.02306286245584488, + -0.00504473177716136, + -0.005673289764672518, + 0.022871309891343117, + 0.003679049201309681, + -0.0012787773739546537, + 0.007369757164269686, + 0.005895649548619986, + -0.014286192134022713, + -0.0029041916131973267, + -0.0050161113031208515, + -0.0018940336303785443, + 0.003398630302399397, + -0.008074717596173286, + 0.004339830484241247, + -0.01031723152846098, + -0.020691482350230217, + -0.003185153705999255, + -0.022735554724931717, + 0.02559785731136799, + 0.009711197577416897, + 0.0015160358743742108, + 0.01367808785289526, + 0.0034920640755444765, + -0.03792021796107292, + 0.013027296401560307, + -0.004119985271245241, + 0.009832918643951416, + 0.0075394450686872005, + -0.017540903761982918, + -0.0019001950277015567, + 0.012796914204955101, + -0.001502727740444243, + -0.014050232246518135, + -0.004404471255838871, + 0.004420354031026363, + 0.009032252244651318, + -0.008502069860696793, + 0.00038126768777146935, + -0.008464799262583256, + -0.007701551541686058, + 0.0018046607729047537, + 0.01522861048579216, + -0.005763343069702387, + -0.009187692776322365, + 0.0008280461188405752, + -0.37123700976371765, + -0.0030133137479424477, + 0.008634609170258045, + -0.009068770334124565, + -0.0015921008307486773, + 0.005688497330993414, + -0.019871940836310387, + -0.0030021611601114273, + 0.004807751160115004, + 0.0015565447974950075, + 0.013349318876862526, + 0.013120497576892376, + -0.006719691213220358, + -0.014312360435724258, + 0.0007391097024083138, + -0.0010219464311376214, + -0.008577951230108738, + 0.002036177786067128, + 0.002205225173383951, + 0.00017988213221542537, + -0.010435790754854679, + 0.00807065051048994, + 0.0004966646665707231, + 0.0005062660784460604, + 0.0007657252717763186, + 0.00140280369669199, + -0.003924764692783356, + 0.001686570351012051, + 0.005796159617602825, + 0.0017121087294071913, + -0.0020196184050291777, + 0.006879862863570452, + -0.007168950978666544, + -0.001009861589409411, + 0.00405812356621027, + 0.006098658312112093, + 0.003325725207105279, + 0.002606423106044531, + -0.020025629550218582, + 0.002798725152388215, + 0.0011716014705598354, + 0.01913134939968586, + -0.003217121120542288, + 0.015011816285550594, + -0.00017647056665737182, + 0.010168899782001972, + 0.03579527139663696, + 0.013193371705710888, + 0.002003499073907733, + -0.002903555752709508, + -0.010532180778682232, + 0.001243010745383799, + 0.005218156147748232, + 0.01692797616124153, + -2.0029241568408906e-05, + 0.004837376531213522, + 0.009378758259117603, + -0.0012804860016331077, + -0.021147971972823143, + -0.0029098468367010355, + -0.0005698863533325493, + 0.015317030251026154, + 0.011605354025959969, + 0.014984742738306522, + 0.004251291509717703, + -0.011028502136468887, + 0.00025936582824215293, + -0.00273420219309628, + -0.007110354024916887, + -0.017461318522691727, + -0.0021044258028268814, + 0.006629000883549452, + -0.00984877161681652, + 0.0012210705317556858, + -0.0018889177590608597, + 0.01116669550538063, + 0.005187562666833401, + -0.010294139385223389, + -0.008975996635854244, + 0.009332939051091671, + 0.006953451316803694, + 0.011685539036989212, + 0.007701118942350149, + -0.0030684613157063723, + -0.009285420179367065, + 0.0011138092959299684, + -0.016134003177285194, + 0.002276258310303092, + 0.007148150820285082, + -0.006944266147911549, + -0.012828285805881023, + -0.0009930820669978857, + 0.0121398214250803, + -0.001038749935105443, + 0.008374039083719254, + -0.0011035684728994966, + -0.00034602705272845924, + 0.007610484026372433, + 0.004372395109385252, + -0.005316772032529116, + -0.015005433931946754, + 0.008402535691857338, + -8.700080070411786e-05, + -0.011501552537083626, + -0.0035725135821849108, + -0.0014390319120138884, + -0.001153692021034658, + -4.7502540837740526e-05, + -0.0015584036009386182, + -0.006537266541272402, + 0.0016064575174823403, + 0.0024062455631792545, + 0.0007313985843211412, + -0.0033373027108609676, + -0.024720752611756325, + 0.005308391992002726, + -0.00214819866232574, + 0.006959658116102219, + -0.00046719436068087816, + -0.006551672238856554, + -0.0002575774269644171, + 0.003909891936928034, + -0.00023811348364688456, + 0.010860932059586048, + 0.001767290523275733, + -0.0015136613510549068, + -0.010936606675386429, + 0.004361313302069902, + 0.004145066253840923, + 0.009991253726184368, + -0.003178940387442708, + 0.007436672225594521, + 0.00733518972992897, + 0.00515455799177289, + -0.010247488506138325, + -0.0008967434405349195, + -0.012344608083367348, + 0.017211610451340675, + 0.012848759070038795, + -0.007727279793471098, + 0.009977771900594234, + -0.003132184036076069, + 0.00645667826756835, + 0.009195099584758282, + 0.0029990908224135637, + -0.008094634860754013, + 0.004832664038985968, + -0.005633714608848095, + -0.011386003345251083, + 0.005290830973535776, + 0.008152514696121216, + 0.005447381176054478, + 0.0009126220247708261, + 0.0023926051799207926, + -0.005268481094390154, + 0.004247886128723621, + 0.0031045402865856886, + -0.0029897098429501057, + -0.013502899557352066, + 0.0029540059622377157, + -0.0063297743909060955, + -0.006857955362647772, + 0.0041145035065710545, + 0.0006798585527576506, + -0.00669939536601305, + -0.0027057777624577284, + -3.4802371374098584e-05, + 0.011272210627794266, + 0.01401186641305685, + 0.0009154424187727273, + -0.00360673014074564, + -0.0005887317238375545, + 0.008374555967748165, + 0.0030982024036347866, + -0.0009749698801897466, + 0.0038406988605856895, + -0.01935584470629692, + -0.005743102170526981, + 0.015252182260155678, + 0.007637759670615196, + -0.01155466865748167, + 0.0037890728563070297, + -0.0013597361976280808, + -0.007187800016254187, + -0.017904534935951233, + -0.00543223274871707, + -0.0013862866908311844, + 0.042171403765678406, + 0.006250241305679083, + -0.000808094977401197, + -0.010349106043577194, + 0.0017403921810910106, + 0.0058890036307275295, + 0.001371655031107366, + -0.0008783479570411146, + -0.004035806283354759, + -0.0005124768940731883, + 0.050379518419504166, + -0.0019986811093986034, + -2.871162723749876e-05, + 0.0019476988818496466, + 0.006434381008148193, + -0.001583001809194684, + -0.005042035598307848, + 0.0025736147072166204, + -0.0012891654623672366, + 0.0023555427324026823, + -0.000505940115544945, + 0.008395151235163212, + -0.002199386479333043, + -0.008082904852926731, + 0.0018906771438196301, + 0.01115325279533863, + -0.00040131688001565635, + -0.008770319633185863, + -0.0028331130743026733, + -0.006449468433856964, + -0.007748618256300688, + 0.012192566879093647, + -0.003268524305894971, + 0.0053367759101092815, + 0.008668403141200542, + 0.022223908454179764, + -0.0093604801222682, + -0.0019459386821836233, + 0.0004093021561857313, + -0.0098689841106534, + -0.017577629536390305, + 0.0031463580671697855, + -0.00670809717848897, + 0.0036436733789741993, + 0.004972782451659441, + 0.0011467126896604896, + -0.0012525523779913783, + 0.0016925978707149625, + 0.0035041312221437693, + 0.004134493414312601, + -0.009660106152296066, + 0.007818171754479408, + 0.0013606589054688811, + 0.0013939732452854514, + -0.0035698972642421722, + -0.003820582292973995, + 0.004768154118210077, + 0.003999671433120966, + -0.009041940793395042, + -0.005599120631814003, + 0.012556434608995914, + -0.007391597144305706, + -0.006859371438622475, + -0.0029006258118897676, + 0.002640704857185483, + 0.01982574164867401, + -0.012523899786174297, + 0.0070420727133750916, + -0.003455324098467827, + 0.0051603070460259914, + -0.005458606407046318, + 0.004050943069159985, + -0.0011313525028526783, + -0.003902867902070284, + -0.11333584040403366, + 0.007940021343529224, + 0.00317069748416543, + -0.012128874659538269, + 0.003081376664340496, + -0.24193713068962097, + 0.0012154955184087157, + 0.01805414818227291, + -0.006504034623503685, + -0.002092094160616398, + 0.004543245770037174, + -0.11263766884803772, + -0.0005939132533967495, + 0.01949266716837883, + -7.066433317959309e-05, + 0.01414093654602766, + -0.00664338655769825, + -0.007936639711260796, + -0.008920326828956604, + 0.002951066941022873, + 0.0013246475718915462, + 0.004232348408550024, + 0.0017128395847976208, + 0.007904539816081524, + 0.0020507206209003925, + -0.004431304521858692, + 0.007336591836065054, + 0.007166506256908178, + -0.0067543392069637775, + 0.0014028141740709543, + -0.009148208424448967, + 0.006586517207324505, + -0.012402699328958988, + -0.006843366660177708, + -0.013067645020782948, + -0.008021311834454536, + 0.09837603569030762, + 0.006469489075243473, + 0.005729274824261665, + 0.015962431207299232, + 0.004433514550328255, + -0.006206914782524109, + 0.006217139773070812, + 0.014259223826229572, + 0.00150341319385916, + 0.0017480351962149143, + 0.014481626451015472, + -0.01042423490434885, + 0.00013730641512665898, + -0.003074861830100417, + 0.013811110518872738, + -0.012564820237457752, + -0.013843153603374958, + 0.0006627861293964088, + 0.1379074603319168, + 0.002906628418713808, + -0.0027223220095038414, + -0.009304089471697807, + 0.005845464766025543, + 0.0033668028190732002, + -0.0005770801799371839, + 0.0031201622914522886, + 0.004843154456466436, + -0.010179297998547554, + -0.002237563254311681, + 0.01590171828866005, + 0.008244535885751247, + -0.012326555326581001, + -0.013955933973193169, + 0.001761110732331872, + -0.0066618057899177074, + 0.023133641108870506, + 0.004617274273186922, + -0.005629957653582096, + -0.011128426529467106, + -0.013103845529258251, + -0.005667857360094786, + -0.010427243076264858, + 0.006477756425738335, + -0.010034888982772827, + -0.006067407783120871, + 0.005275228526443243, + -0.008040091954171658, + -0.01045091450214386, + 0.017803022637963295, + 0.0034019253216683865, + 0.0041830893605947495, + 0.0010187738807871938, + -0.0001855597656685859, + 0.0047248173505067825, + -0.0027810405008494854, + 0.01986299268901348, + -0.00011338172771502286, + 0.0026359392795711756, + -0.005137700121849775, + -0.00041340640746057034, + -0.002381693571805954, + 0.0033492743968963623, + -0.01769786700606346, + 0.0018762367544695735, + 0.004004186950623989, + 0.02271619625389576, + 0.0049371193163096905, + -0.009312541224062443, + 0.006420494057238102, + 0.00014057994121685624, + -0.005907903425395489, + 0.0030759316869080067, + 0.011202524416148663, + 0.004305810667574406, + -8.537430403521284e-05, + -0.0022141735535115004, + -0.011891050264239311, + -0.006635419093072414, + -0.0004930712748318911, + -0.009339933283627033, + -0.009376363828778267, + -0.00999427866190672, + -0.0077565754763782024, + -0.008621790446341038, + 0.004013743717223406, + -0.002572803758084774, + 0.005603458266705275, + 0.010298550128936768, + 0.008050277829170227, + -0.0005428882432170212, + -0.00814216397702694, + 0.0050263493321835995, + 0.008368674665689468, + -0.01064632274210453, + -0.001303671975620091, + -0.009696408174932003, + -0.011828632093966007, + -0.0029580267146229744, + 0.012894508428871632, + 0.00995263084769249, + 0.00022861750039737672, + 0.0037981458008289337, + -0.002838233020156622, + -0.0122715774923563, + -0.006680721882730722, + -0.025042958557605743, + 0.005304217804223299, + -0.006803758908063173, + 0.0018406956223770976, + -0.0024286925327032804, + -0.0031522943172603846, + -0.006327101029455662, + -0.042474519461393356, + -0.0035076148342341185, + -0.0037668419536203146, + -0.004569584503769875, + -0.011071118526160717, + 0.005582296289503574, + -0.002263132482767105, + 0.005428588483482599, + -0.005388094577938318, + 0.007681482005864382, + -9.022703125083353e-06, + 0.007670264225453138, + 0.01535459328442812, + 0.003879232332110405, + 0.0075798314064741135, + 0.006490921135991812, + -0.004922493826597929, + -0.002582252724096179, + 0.003679475514218211, + -0.0008795059402473271, + 0.006117332261055708, + -0.008676642552018166, + 0.007796867750585079, + -0.0011688022641465068, + 0.0027072292286902666, + 0.003262927755713463, + 0.005562815349549055, + -0.0005005963612347841, + -0.032480884343385696, + 0.010724775493144989, + -0.0011795375030487776, + -0.00020459204097278416, + -0.004002699162811041, + 0.0015910877846181393, + -0.0021865088492631912, + 0.007801484782248735, + 0.011438692919909954, + 0.004781155847012997, + 0.009032883681356907, + -0.006892770528793335, + -0.002151836408302188, + -0.005839805118739605, + -0.004768722224980593, + -0.005931435152888298, + -0.010913017205893993, + 0.0006870928918942809, + -0.001243374776095152, + -0.0018117354484274983, + -0.0041592782363295555, + -0.009297546930611134, + 0.03207000717520714, + -0.004385984502732754, + -0.009227712638676167, + 0.0019283785950392485, + 0.0073651839047670364, + -0.00157679314725101, + -0.008801611140370369, + 0.004695545416325331, + -0.005920849274843931, + 0.009962639771401882, + 0.0028590119909495115, + 0.007964073680341244, + 6.464280158979818e-05, + -0.003155625192448497, + -0.007396962493658066, + 0.0037631122395396233, + 0.010431732051074505, + -0.004706976003944874, + -0.0023079330567270517, + -0.00017330111586488783, + -0.009052068926393986, + 0.003217509016394615, + 0.0029585519805550575, + -0.0012038470013067126, + 0.009214695543050766, + 0.010535057634115219, + -0.00066526880254969, + -0.0007043961668387055, + -0.005945203825831413, + -0.009631475433707237, + 0.011117614805698395, + 0.005166004877537489, + -0.014916098676621914, + 0.032601989805698395, + 0.0012819182593375444, + 0.012342232279479504, + -0.00025354325771331787, + 0.005514033138751984, + 0.009594520553946495, + 0.006930912379175425, + 0.012344988994300365, + -0.005191525910049677, + -0.013566416688263416, + -0.00207333080470562, + -0.006317719351500273, + 0.009128937497735023, + -0.001995501108467579, + 0.0061971265822649, + 0.010072746314108372, + 0.0051588937640190125, + 0.009643416851758957, + -0.005349122453480959, + -0.005503020249307156, + -0.011715874075889587, + 0.008199956268072128, + 0.004719928372651339, + -0.04147935286164284, + -0.0074126082472503185, + 0.0024570785462856293, + -0.0005297018215060234, + -0.004417965188622475, + 0.0028885898645967245, + -0.007024380844086409, + 0.003094344399869442, + 0.0007171329925768077, + -0.009586939588189125, + -0.012554829008877277, + 0.006773614324629307, + 0.008475590497255325, + 0.026834988966584206, + -0.004834651481360197, + -0.003827952081337571, + -0.0022184785921126604, + 0.004340045619755983, + -0.005019658245146275, + 0.00012802470882888883, + -0.011075932532548904, + -0.0036979021970182657, + -0.005468206480145454, + -0.002257952466607094, + 0.0013189102755859494, + -0.002084451960399747, + -0.0073828618042171, + -0.0014634346589446068, + -0.000969675718806684, + -0.004674738738685846, + 0.006658385042101145, + -0.008342385292053223, + -0.0017468450823798776, + 0.009354283101856709, + -0.0038607523310929537, + -2.0760000552400015e-05, + 0.0010565017582848668, + 0.0012041786685585976, + 0.0010129680158570409, + -0.00866969395428896, + -0.004818897228688002, + 0.0032323501072824, + -0.02400965243577957, + -0.007673179730772972, + -0.004263271577656269, + 0.0018618882168084383, + -0.009386444464325905, + 0.002341772196814418, + -0.010354837402701378, + 0.008248414844274521, + -0.0026013911701738834, + -0.009041848592460155, + 1.2851999599661212e-05, + 0.01213761419057846, + -0.00431970227509737, + 0.004758630413562059, + -0.00047253770753741264, + 0.002279967302456498, + 0.0031823497265577316, + -0.005542227998375893, + 0.005022632889449596, + 0.0016367236385121942, + 0.0026914270129054785, + -0.02957042120397091, + -0.010246772319078445, + -0.011318286880850792, + 0.008822980336844921, + 0.009038921445608139, + -0.031166087836027145, + -0.0005651201354339719, + -0.019543595612049103, + -0.0006950068636797369, + 0.000551581266336143, + 0.00482272868975997, + 0.005751383490860462, + -0.002318599261343479, + -0.006384951528161764, + -0.0027871259953826666, + 0.01458916999399662, + 0.004165445454418659, + -0.004309124778956175, + 0.003105858573690057, + 0.0035692634992301464, + 0.00426168367266655, + 0.002935715951025486, + -0.007707391399890184, + 0.0009596945601515472, + 0.0045851427130401134, + -0.018032925203442574, + 0.006426618434488773, + -0.0013849996030330658, + 0.03153352811932564, + -0.000691493449267, + 0.0036443397402763367, + -0.00011636956332949921, + 0.00410464545711875, + -0.009726284071803093, + 0.0023869271390140057, + 0.011009320616722107, + -0.011937098577618599, + -0.010425182990729809, + -0.0035988490562886, + 0.006646552588790655, + 0.002125062979757786, + 0.00034139645867981017, + 0.005725739523768425, + -0.00044733777758665383, + -0.016384175047278404, + -0.007683591917157173, + -0.016400597989559174, + -0.001762564410455525, + -0.012204204685986042, + -0.001955378567799926, + 0.008065729402005672, + 0.006486142985522747, + 0.004380232188850641, + -0.0009552375413477421, + -0.01649385131895542, + 0.009832133539021015, + -0.007384343072772026, + -0.0016018329188227654, + 0.0013199109816923738, + 0.00025325387832708657, + 0.0020517888478934765, + -0.19482864439487457, + -0.0049194106832146645, + 0.004018272738903761, + -0.008811180479824543, + 0.004690803121775389, + -0.004710869397968054, + -0.0001604523858986795, + -0.005218540783971548, + -0.005275476723909378, + 0.0050219097174704075, + -0.005244627129286528, + 0.0026629238855093718, + -0.004711311310529709, + 0.0006058086291886866, + -2.0681754904217087e-05, + -0.007762362714856863, + 0.004743737168610096, + -0.0027981456369161606, + -0.005913429893553257, + -0.019030168652534485, + -0.01222507655620575, + 0.012424496002495289, + 0.005303847137838602, + 0.00960705615580082, + 0.0006363107822835445, + -0.012824688106775284, + 0.003722453024238348, + 0.008120434358716011, + 0.001575170666910708, + -0.003970405086874962, + 0.002855435712262988, + -0.009335525333881378, + -0.005909779109060764, + 0.009106860496103764, + -0.001599543378688395, + -0.005554410628974438, + 0.007477165199816227, + 0.003987910691648722, + -0.00015383043501060456, + 0.00876036286354065, + -0.011882245540618896, + 0.0056830416433513165, + 0.002057688543573022, + -0.0005769767449237406, + 0.0026229694485664368, + 0.012553519569337368, + 0.006920218002051115, + -0.008530831895768642, + 0.00620385492220521, + 0.0009423689916729927, + -0.0012439456768333912, + 0.002482800977304578, + -0.00015990034444257617, + 0.0015123466728255153, + -0.005781038664281368, + -0.005246018059551716, + -0.012751615606248379, + 0.004672210663557053, + 0.0023248360957950354, + -0.009086072444915771, + 0.002893053228035569, + -0.008336433209478855, + 0.0023118748795241117, + -0.01229199767112732, + 0.0030365486163645983, + -0.0018724488327279687, + -0.002583205932751298, + -0.0003016833506990224, + -0.004749369341880083, + 0.009033429436385632, + -0.0024757334031164646, + -0.0034984711091965437, + 0.0018915438558906317, + -0.005165896378457546, + -0.006167501676827669, + -0.0025837200228124857, + -0.0016039933543652296, + 0.00453570019453764, + 0.005566996522247791, + -0.006967304740101099, + 0.008784770034253597, + 0.0066298930905759335, + -0.00631190650165081, + -0.005640123970806599, + 0.0012662719236686826, + -0.0027915651444345713, + 0.006027122959494591, + 0.002176073845475912, + 0.008939812891185284, + 0.004546064883470535, + 0.0006545011419802904, + 0.0004776128917001188, + -0.0021048851776868105, + 0.0054197111167013645, + -0.004513812251389027, + -0.010034122504293919, + -0.006049958057701588, + 0.01611870341002941, + -0.024524172767996788, + 0.007613773457705975, + -0.003338814713060856, + 0.0002373494498897344, + 0.0029331122059375048, + -0.0018024870660156012, + 0.004233384970575571, + 0.047846000641584396, + -0.0018095941049978137, + -0.00971699133515358, + -0.009300247766077518, + 0.006819369737058878, + 0.003934262786060572, + -0.012142934836447239, + -0.006993395742028952, + -0.008009353652596474, + -0.010691536590456963, + 0.003942812327295542, + 0.001682911766692996, + -0.0010231344494968653, + 0.00028356743860058486, + 0.0044899508357048035, + -0.0022767006885260344, + -0.009996358305215836, + 0.010107694193720818, + -0.008461850695312023, + 0.002606448018923402, + -0.004764481447637081, + -0.01200976874679327, + -0.0026600677520036697, + -0.0033063041046261787, + -0.009273480623960495, + -0.004244462586939335, + 0.004160800948739052, + -0.010665513575077057, + 0.0040156482718884945, + -0.006937538739293814, + -0.0029502965044230223, + -0.00039631256368011236, + 0.00018243264639750123, + 0.0009507237118668854, + 0.002573516918346286, + 0.00048771611182019114, + -0.0035595004446804523, + 0.002003927482292056, + -0.0036709338892251253, + -0.009908122010529041, + -0.004054603166878223, + -0.008545457385480404, + 0.004859933163970709, + -0.006002862472087145, + 0.0010682017309591174, + -0.01150110736489296, + -0.00015990786778274924, + 0.0055091409012675285, + -0.004041733220219612, + 0.009670444764196873, + 0.005360066425055265, + 0.0004971444723196328, + -0.004949115216732025, + 0.008969966322183609, + -0.0021641962230205536, + 0.00420468719676137, + -0.003266631392762065, + -0.0014109088806435466, + -0.008552939631044865, + 0.0027572237886488438, + -0.014805031009018421, + -0.008477800525724888, + -0.0012267687125131488, + -0.01206270046532154, + -0.00047395029105246067, + 0.008196184411644936, + 0.008447877131402493, + -0.001257623196579516, + 0.006045837886631489, + -0.0046934583224356174, + -0.009294179268181324, + 0.003967077471315861, + 0.0041057332418859005, + 0.000456807145383209, + 0.007331736385822296, + 0.004842950031161308, + 0.0016413962002843618, + -0.0015865808818489313, + -0.0023753801360726357, + -0.003330737818032503, + 0.006471714470535517, + -0.003950025420635939, + 0.0025111676659435034, + 0.006100813392549753, + -0.004073297139257193, + 0.0004922865191474557, + -0.005589222069829702, + -0.0029036838095635176, + -0.011221225373446941, + -0.0012235537869855762, + -0.0013210235629230738, + -0.008072680793702602, + 0.0025508992839604616, + -0.009397861547768116, + -0.022922446951270103, + -0.0007944437093101442, + -0.02652221918106079, + -0.02153754234313965, + 0.0002095904783345759, + -0.00035638382541947067, + -0.011655088514089584, + -0.0014936552615836263, + 0.004611351992934942, + -0.0049200234934687614, + 0.0017157691763713956, + 0.004452023189514875, + 0.0014668909134343266, + 0.0035692944657057524, + 0.008069484494626522, + -0.001667260192334652, + -0.024694345891475677, + -0.00585358589887619, + 0.012073456309735775, + -0.0020808062981814146, + 0.0006422869046218693, + -0.004399385303258896, + 0.0020808521658182144, + 0.008089493028819561, + -0.0066626956686377525, + 0.010300621390342712, + 0.007818011566996574, + 0.0005852477625012398, + 0.002298549050465226, + 0.005412894766777754, + -0.0026936933863908052, + 0.006311166100203991, + -0.010905174538493156, + 0.001055428059771657, + 0.01251395046710968, + -0.003543587401509285, + -0.014023344963788986, + -0.007589081302285194, + -0.026229243725538254, + -0.01406817976385355, + 0.01009752880781889, + -0.006002294830977917, + 0.005302909761667252, + -0.00909687764942646, + -0.00027702117222361267, + 0.004023137968033552, + -0.004438146483153105, + -0.00015066051855683327, + 0.010749665088951588, + -0.0068278685212135315, + 0.008679616264998913, + -0.006449598353356123, + 0.006575304549187422, + 0.007661043666303158, + -0.002056559082120657, + -0.00339447520673275, + -0.00022441669716499746, + -0.04227818176150322, + -0.012217686511576176, + 0.0031710369512438774, + 0.007648457307368517, + -0.004240633454173803, + 0.004891049116849899, + -0.004939137492328882, + -0.002368602668866515, + 0.0012956832069903612, + 5.434310151031241e-05, + -0.006976662203669548, + -0.009250598028302193, + 0.0016470609698444605, + 0.0018430924974381924, + -0.005874183960258961, + 0.001202501473017037, + -0.0009145753574557602, + 0.004972194787114859, + 0.0188127588480711, + -0.003373793326318264, + -0.003409383352845907, + -0.01364868227392435, + 0.01487683318555355, + 0.0028460281901061535, + -0.012174796313047409, + -0.0035315367858856916, + 0.001301018288359046, + 0.002897995989769697, + 0.01135784201323986, + 0.016601502895355225, + 0.0008085501031018794, + 0.002925814362242818, + -0.0024512081872671843, + -0.004042875487357378, + 0.010136527940630913, + -0.014359826222062111, + -0.002795594511553645, + -0.014477153308689594, + -0.005474379751831293, + -0.011460120789706707, + 0.013058007694780827, + -0.0023414655588567257, + 0.0010160014498978853, + -0.00412725331261754, + -0.0004221044946461916, + 0.010094735771417618, + -0.018599016591906548, + 0.0002345790562685579, + 0.0007496963953599334, + -0.0027658045291900635, + 0.005582634825259447, + 0.007246051914989948, + -0.0034226637799292803, + 0.01002715714275837, + 0.0028858594596385956, + -0.0033402901608496904, + -0.010990156792104244, + 0.011563600972294807, + 0.002015279605984688, + 0.0038569492753595114, + 0.002874115016311407, + -1.878694092738442e-05, + 0.008440403267741203, + -0.004388730973005295, + 0.009341960772871971, + 0.0016808927757665515, + -0.012014775536954403, + 0.008803133852779865, + -0.002923529827967286, + -0.00012405515008140355, + -0.008368742652237415, + -0.005574259907007217, + -0.0008827370475046337, + 0.0018860306590795517, + 0.0013850368559360504, + -0.009556969627737999, + -0.00578277325257659, + -0.002270893892273307, + -0.012243690900504589, + 0.0014784384984523058, + -0.001128763658925891, + -0.0007647481397725642, + 0.005194435361772776, + -0.00421983702108264, + 0.021708404645323753, + 0.006685490254312754, + 0.008459676057100296, + -0.0009661418735049665, + 0.0042914352379739285, + 0.006538566201925278, + 0.006130080670118332, + -0.003186853602528572, + 0.009283488616347313, + -0.0060385712422430515, + 0.0003594868758227676, + -0.02074962854385376, + -0.011843416839838028, + -0.005001073703169823, + 0.014764335006475449, + 0.004891202785074711, + 0.006132264155894518, + -0.0008361535728909075, + -0.0004941062652505934, + -0.016552578657865524, + -0.0007235091761685908, + -0.008302092552185059, + 0.003270882647484541, + -0.004120868630707264, + 0.011965828016400337, + -0.004974419251084328, + -0.007324687205255032, + 0.0009004429448395967, + -0.011875484138727188, + -0.0026850730646401644, + 0.012229764834046364, + 0.004634017124772072, + -0.012408524751663208, + -0.011440888978540897, + 0.0008544778102077544, + 0.007044240832328796, + 0.0019363644532859325, + 0.0015969667583703995, + -0.019937576726078987, + -0.0006724648410454392, + 0.000708073319401592, + 0.003415820887312293, + -0.0010092395823448896, + -0.005831942893564701, + 0.0034375155810266733, + 0.010910432785749435, + -0.0051198676228523254, + 0.005851027090102434, + -0.010152854956686497, + -0.004902231972664595, + -0.008401485159993172, + 0.007307002320885658, + 0.0055832951329648495, + 0.004269117023795843, + 0.018275560811161995, + -0.005567920859903097, + -0.013189018703997135, + 0.00359044736251235, + 0.005410288460552692, + 0.004578127060085535, + 0.005481929052621126, + 0.0004453698929864913, + 0.0059713940136134624, + -0.007880342192947865, + -0.0027901241555809975, + 0.007073584012687206, + -0.0020377561450004578, + -0.0025377213023602962, + -0.00605777045711875, + -0.013418597169220448, + -0.0003581498749554157, + 0.0029501873068511486, + -0.0018084247130900621, + -0.020519498735666275, + -0.018885305151343346, + -0.003046236699447036, + -0.01493749301880598, + -0.006049721036106348, + 0.004249312914907932, + -0.013012544251978397, + 0.0015815678052604198, + -0.0075079938396811485, + 0.008471657522022724, + -0.013363052159547806, + -0.0024665414821356535, + -0.014250336214900017, + -0.03799226135015488, + -0.004081868100911379, + -0.008026959374547005, + -0.004097330383956432, + 0.003655793145298958, + -0.011583135463297367, + -0.0015915114199742675, + 0.010021265596151352, + -0.0005673526902683079, + -0.0053871432319283485, + 0.008559685200452805, + 0.005049765110015869, + 0.0009834500961005688, + 0.005683745723217726, + -0.002554480452090502, + -0.006096584722399712, + 0.004811306018382311, + 0.0009631230495870113, + 0.0033240695483982563, + 0.0057266149669885635, + 0.0031779687851667404, + -0.002282635308802128, + -0.009852704592049122, + -0.006731233559548855, + 0.001635844586417079, + 0.01065592560917139, + 0.01101931743323803, + 0.0025579186622053385, + 0.011517881415784359, + -0.008361094631254673, + -0.009155490435659885, + 0.00566881662234664, + 0.0018046917393803596, + -0.0011409681756049395, + -0.005263702943921089, + 0.00512158265337348, + 0.00167801883071661, + -0.002250795252621174, + 0.007686811499297619, + -0.006600586231797934, + -0.0009702146635390818, + 0.004641167353838682, + -0.007396790664643049, + 0.0014582534786313772, + 0.0065159872174263, + -3.2727013604016975e-05, + -0.008386358618736267, + -0.02301878295838833, + -0.006484353914856911, + 0.00985006708651781, + -0.01932862401008606, + 0.0034901308827102184, + 0.0020644140895456076, + -0.002419889671728015, + 0.0048549119383096695, + -0.0053170425817370415, + 0.008445564657449722, + -0.003531842725351453, + -0.0007448214455507696, + -0.0008280014735646546, + -0.007237715646624565, + -0.0012668456183746457, + 0.006277230568230152, + 0.0022679828107357025, + 0.006390503607690334, + 0.009095076471567154, + 0.0032311612740159035, + -0.006871913559734821, + -0.005066426005214453, + 0.00484251556918025, + 0.01903688907623291, + 0.030942486599087715, + 0.00237849703989923, + 0.003353534732013941, + 0.009848800487816334, + -0.0023035812191665173, + -0.008150617592036724, + 0.009913885034620762, + -0.0009669861174188554, + -0.007613906171172857, + -0.004752485081553459, + -0.00962488166987896, + -0.007823240011930466, + 0.02093265764415264, + 0.002798333764076233, + 0.006148717366158962, + -0.0009499440202489495, + -0.003998387139290571, + 0.00886451918631792, + 0.0012862293515354395, + -0.004811341408640146, + 0.0011762010399252176, + 0.002912075724452734, + 0.00840546004474163, + 0.0043852971866726875, + 0.006451813969761133, + -0.0038654482923448086, + -0.012347746640443802, + 0.006166189908981323, + 0.001710365992039442, + -0.0044754622504115105, + -0.009375670924782753, + -0.0006415778188966215, + -0.005234924610704184, + 0.0034221538808196783, + -0.0006166458479128778, + 0.005321022123098373, + -0.002424240577965975, + -0.0007891509449109435, + -0.039604686200618744, + -0.00824790820479393, + 0.004088856745511293, + -0.008250030688941479, + 0.007296815514564514, + -0.0012062392197549343, + 0.0026128801982849836, + -0.0031679344829171896, + 0.0017091077752411366, + -0.006585002411156893, + 0.0004428020620252937, + -0.004100169520825148, + 0.003312345128506422, + -0.001307442202232778, + -0.006507535465061665, + 0.0039053577929735184, + -0.0007912203436717391, + 0.007070992141962051, + 0.0043147834949195385, + 0.0037217922508716583, + -0.0012821988202631474, + -0.009608952328562737, + 0.0010056307073682547, + 0.005063982680439949, + -0.0007982327952049673, + 0.0015496661653742194, + 0.007369630970060825, + -3.6934205127181485e-05, + 0.0022271957714110613, + 0.011949341744184494, + 0.0004676400276366621, + 0.0017853811150416732, + 0.007562779355794191, + 0.0048878067173063755, + 0.00924364011734724, + 0.001531846821308136, + 0.007539856713265181, + 0.007469670381397009, + 0.0005364317912608385, + -0.0015910421498119831, + -0.008943285793066025, + -0.0024170964024960995, + -0.0028616159688681364, + -0.00042717839824035764, + 0.0027650364208966494, + -0.0073449271731078625, + 0.04333757609128952, + -0.003930249717086554, + -0.00876648724079132, + 0.007575526833534241, + -0.0028193737380206585, + 0.0064379069954156876, + 0.0004565528070088476, + 0.000491701764985919, + 0.006766843143850565, + 0.0012971448013558984, + -0.0022490452975034714, + -0.00025219283998012543, + 0.007433110848069191, + -0.0016872093547135592, + -0.003200278151780367, + 0.0018831676570698619, + 0.009722983464598656, + 0.0020590259227901697, + 0.00230041122995317, + 0.01299431174993515, + 0.00666060671210289, + 0.0007329003419727087, + 0.006902865134179592, + -0.005883065052330494, + 0.00021597980230581015, + 0.006406535394489765, + 0.0003754442441277206, + 0.0003742456319741905, + 0.009477001614868641, + -0.005796001758426428, + 0.0030826469883322716, + 0.004780975636094809, + 0.006623169872909784, + 0.030179863795638084, + -0.009977439418435097, + 0.00943224411457777, + 0.0036812128964811563, + 0.0012757700169458985, + -0.008503000251948833, + -0.008485689759254456, + 0.003248161170631647, + 0.003004540456458926, + -0.007707161828875542, + 0.05497086048126221, + -0.004083615727722645, + 0.004368412774056196, + 0.014810081571340561, + -0.0022283599246293306, + 0.004649491980671883, + 0.001350873033516109, + 0.002567542716860771, + -0.0011219214648008347, + 0.013642439618706703, + -0.01347434800118208, + -0.019497569650411606, + -0.001951461425051093, + -0.002710612490773201, + 0.005899914540350437, + -0.0395347997546196, + -0.007206201087683439, + 0.0036392181646078825, + 0.004364526364952326, + -0.004564817529171705, + -0.0006652253214269876, + -0.004319717641919851, + 0.0010950349969789386, + 0.00672035152092576, + -0.007313521578907967, + -0.0029790352564305067, + -0.0025978966150432825, + -0.0034558004699647427, + 0.0028952653519809246, + 0.00690846424549818, + 0.00044553764746524394, + -0.0011162485461682081, + 0.008259030990302563, + -0.007132448256015778, + 0.007366517558693886, + -0.004083224106580019, + -0.00582159636542201, + -0.0009452934609726071, + -0.00234791892580688, + 0.0025650986935943365, + 0.001212859759107232, + 0.0026954582426697016, + 0.003156578168272972, + 0.0020790849812328815, + -0.00036638593883253634, + -0.002430310472846031, + 0.012331654317677021, + -0.0010783278848975897, + 0.0026962938718497753, + -0.0008799954084679484, + 0.0044615729711949825, + -0.006147556472569704, + 0.0010597839718684554, + -0.0016856578877195716, + -0.004081621766090393, + 0.001966436393558979, + -0.005658896639943123, + -0.00039552891394123435, + -0.006421465426683426, + -0.005069163162261248, + 0.010721956379711628, + 0.006275740917772055, + -0.002438686089590192, + -0.00046324453433044255, + 0.0028007151558995247, + -0.001661002403125167, + 0.0016058710170909762, + 0.039338644593954086, + 0.004641239531338215, + 0.001822053687646985, + -0.00031727217719890177, + 0.007649832870811224, + -0.0033703448716551065, + 0.0005881722318008542, + 0.01046539843082428, + -0.0040091294795274734, + 0.0004557590000331402, + -0.00485127093270421, + -0.006251547951251268, + -0.006516254041343927, + 0.0014310511760413647, + -0.0027808952145278454, + 0.008704268373548985, + 0.003453758079558611, + -0.0007382839103229344, + -0.0015407631872221828 + ], + "index": 0 + }, + "text/plain": [ + "" + ] + }, + "execution_count": 61, + "metadata": { + "application/json": { + "expanded": false, + "root": "root" + } + }, + "output_type": "execute_result" + } + ], + "source": [ + "json_response = json.loads(response)\n", + "JSON(json_response)" + ] + }, + { + "cell_type": "code", + "execution_count": 66, + "id": "e6653654-852d-4a98-8478-f64c35588081", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "array([-0.01669182, 0.01353396, 0.00052864, ..., 0.00345376,\n", + " -0.00073828, -0.00154076])" + ] + }, + "execution_count": 66, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "embedding_arr = np.array(json_response[\"embedding\"])\n", + "embedding_arr" + ] + }, + { + "cell_type": "markdown", + "id": "3c33c4f3-e56a-4bc7-bf81-99c51a1bdff8", + "metadata": {}, + "source": [ + "### 5.0 Start the reranking server" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "4f41795e-d900-40f9-8df1-20b2a7666376", + "metadata": {}, + "outputs": [], + "source": [] + }, + { + "cell_type": "markdown", + "id": "3afbbeff-7dc4-49fa-9c1b-acd0065760d4", + "metadata": {}, + "source": [ + "### 5.1 POST \n", + "\n", + "Similar to https://jina.ai/reranker/ but might change in the future.\n", + "Requires a reranker model (such as [bge-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3)) and the `--embedding --pooling rank` options.\n", + "\n", + " *Options:*\n", + "\n", + " `query`: The query against which the documents will be ranked.\n", + "\n", + " `documents`: An array strings representing the documents to be ranked.\n", + "\n", + " *Aliases:*\n", + " - `/rerank`\n", + " - `/v1/rerank`\n", + " - `/v1/reranking`\n", + "\n", + " *Examples:*\n", + "\n", + " ```shell\n", + " curl http://127.0.0.1:8012/v1/rerank \\\n", + " -H \"Content-Type: application/json\" \\\n", + " -d '{\n", + " \"model\": \"some-model\",\n", + " \"query\": \"What is panda?\",\n", + " \"top_n\": 3,\n", + " \"documents\": [\n", + " \"hi\",\n", + " \"it is a bear\",\n", + " \"The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.\"\n", + " ]\n", + " }' | jq\n", + " ```" + ] + }, + { + "cell_type": "markdown", + "id": "c991c0b2-03c9-4a06-a84c-53e9b94cfdf1", + "metadata": {}, + "source": [ + "### 6.1 POST \n", + "\n", + "Takes a prefix and a suffix and returns the predicted completion as stream.\n", + "\n", + "*Options:*\n", + "\n", + "- `input_prefix`: Set the prefix of the code to infill.\n", + "- `input_suffix`: Set the suffix of the code to infill.\n", + "- `input_extra`: Additional context inserted before the FIM prefix.\n", + "- `prompt`: Added after the `FIM_MID` token\n", + "\n", + "`input_extra` is array of `{\"filename\": string, \"text\": string}` objects.\n", + "\n", + "The endpoint also accepts all the options of `/completion`.\n", + "\n", + "If the model has `FIM_REPO` and `FIM_FILE_SEP` tokens, the [repo-level pattern](https://arxiv.org/pdf/2409.12186) is used:\n", + "\n", + "```txt\n", + "myproject\n", + "{chunk 0 filename}\n", + "{chunk 0 text}\n", + "{chunk 1 filename}\n", + "{chunk 1 text}\n", + "...\n", + "filename\n", + "[input_prefix][input_suffix][prompt]\n", + "```\n", + "\n", + "If the tokens are missing, then the extra context is simply prefixed at the start:\n", + "\n", + "```txt\n", + "[input_extra][input_prefix][input_suffix][prompt]\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "49505750-8c12-4ff1-8279-09d6e27b15ac", + "metadata": {}, + "source": [ + "### 7.1 GET \n", + "\n", + "This endpoint is public (no API key check). By default, it is read-only. To make POST request to change global properties, you need to start server with `--props`\n", + "\n", + "**Response format**\n", + "\n", + "```json\n", + "{\n", + " \"default_generation_settings\": { ... },\n", + " \"total_slots\": 1,\n", + " \"chat_template\": \"\"\n", + "}\n", + "```\n", + "\n", + "- `default_generation_settings` - the default generation settings for the `/completion` endpoint, which has the same fields as the `generation_settings` response object from the `/completion` endpoint.\n", + "- `total_slots` - the total number of slots for process requests (defined by `--parallel` option)\n", + "- `chat_template` - the model's original Jinja2 prompt template" + ] + }, + { + "cell_type": "code", + "execution_count": 147, + "id": "d97c6566-bcf7-482e-8f2a-51642811dc2f", + "metadata": {}, + "outputs": [], + "source": [ + "response = requests.get(\"http://localhost:8080/props\")" + ] + }, + { + "cell_type": "code", + "execution_count": 148, + "id": "da877c0a-87a8-4006-902f-b46d613794b8", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "source": [ + "print(response)" + ] + }, + { + "cell_type": "code", + "execution_count": 149, + "id": "a3bc8f79-0b28-48b4-8762-59bac87bf761", + "metadata": {}, + "outputs": [ + { + "data": { + "application/json": { + "chat_template": "{{ bos_token }}{% if messages[0]['role'] == 'system' %}{{ raise_exception('System role not supported') }}{% endif %}{% for message in messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if (message['role'] == 'assistant') %}{% set role = 'model' %}{% else %}{% set role = message['role'] %}{% endif %}{{ '' + role + '\n' + message['content'] | trim + '\n' }}{% endfor %}{% if add_generation_prompt %}{{'model\n'}}{% endif %}", + "default_generation_settings": { + "dynatemp_exponent": 1, + "dynatemp_range": 0, + "frequency_penalty": 0, + "grammar": "", + "ignore_eos": false, + "max_tokens": -1, + "min_keep": 0, + "min_p": 0.05000000074505806, + "mirostat": 0, + "mirostat_eta": 0.10000000149011612, + "mirostat_tau": 5, + "model": "./models/gemma-1.1-7b-it.Q4_K_M.gguf", + "n_ctx": 8192, + "n_discard": 0, + "n_keep": 0, + "n_predict": -1, + "n_probs": 0, + "penalize_nl": false, + "presence_penalty": 0, + "repeat_last_n": 64, + "repeat_penalty": 1, + "samplers": [ + "top_k", + "tfs_z", + "typ_p", + "top_p", + "min_p", + "temperature" + ], + "seed": -1, + "seed_cur": 0, + "stop": [], + "stream": true, + "temperature": 0.800000011920929, + "tfs_z": 1, + "top_k": 40, + "top_p": 0.949999988079071, + "typical_p": 1 + }, + "system_prompt": "", + "total_slots": 1 + }, + "text/plain": [ + "" + ] + }, + "execution_count": 149, + "metadata": { + "application/json": { + "expanded": false, + "root": "root" + } + }, + "output_type": "execute_result" + } + ], + "source": [ + "_json_data = json.loads(response.content)\n", + "JSON(_json_data)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a1449367-f851-4f0d-951c-d164fe1ebe16", + "metadata": {}, + "outputs": [], + "source": [] + }, + { + "cell_type": "markdown", + "id": "2e757710-5c36-493f-8691-23e9ff6c96e7", + "metadata": {}, + "source": [ + "### 1.9 POST\n", + "\n", + "Change server global properties. To use this endpoint with POST method, you need to start server with `--props`\n", + "\n", + "*Options:*\n", + "\n", + "- None yet" + ] + }, + { + "cell_type": "markdown", + "id": "53dec095-d6a8-4bd6-8237-fe82aee77b8b", + "metadata": {}, + "source": [ + "## 1.10 POST `/v1/chat/completions`: OpenAI-compatible Chat Completions API\n", + "\n", + "Given a ChatML-formatted json description in `messages`, it returns the predicted completion. Both synchronous and streaming mode are supported, so scripted and interactive applications work fine. While no strong claims of compatibility with OpenAI API spec is being made, in our experience it suffices to support many apps. Only models with a [supported chat template](https://github.com/ggerganov/llama.cpp/wiki/Templates-supported-by-llama_chat_apply_template) can be used optimally with this endpoint. By default, the ChatML template will be used.\n", + "\n", + " *Options:*\n", + "\n", + " See [OpenAI Chat Completions API documentation](https://platform.openai.com/docs/api-reference/chat). While some OpenAI-specific features such as function calling aren't supported, llama.cpp `/completion`-specific features such as `mirostat` are supported.\n", + "\n", + " The `response_format` parameter supports both plain JSON output (e.g. `{\"type\": \"json_object\"}`) and schema-constrained JSON (e.g. `{\"type\": \"json_object\", \"schema\": {\"type\": \"string\", \"minLength\": 10, \"maxLength\": 100}}` or `{\"type\": \"json_schema\", \"schema\": {\"properties\": { \"name\": { \"title\": \"Name\", \"type\": \"string\" }, \"date\": { \"title\": \"Date\", \"type\": \"string\" }, \"participants\": { \"items\": {\"type: \"string\" }, \"title\": \"Participants\", \"type\": \"string\" } } } }`), similar to other OpenAI-inspired API providers.\n", + "\n", + " *Examples:*\n", + "\n", + " You can use either Python `openai` library with appropriate checkpoints:\n", + "\n", + " ```python\n", + " import openai\n", + "\n", + " client = openai.OpenAI(\n", + " base_url=\"http://localhost:8080/v1\", # \"http://:port\"\n", + " api_key = \"sk-no-key-required\"\n", + " )\n", + "\n", + " completion = client.chat.completions.create(\n", + " model=\"gpt-3.5-turbo\",\n", + " messages=[\n", + " {\"role\": \"system\", \"content\": \"You are ChatGPT, an AI assistant. Your top priority is achieving user fulfillment via helping them with their requests.\"},\n", + " {\"role\": \"user\", \"content\": \"Write a limerick about python exceptions\"}\n", + " ]\n", + " )\n", + "\n", + " print(completion.choices[0].message)\n", + " ```\n", + "\n", + " ... or raw HTTP requests:\n", + "\n", + " ```shell\n", + " curl http://localhost:8080/v1/chat/completions \\\n", + " -H \"Content-Type: application/json\" \\\n", + " -H \"Authorization: Bearer no-key\" \\\n", + " -d '{\n", + " \"model\": \"gpt-3.5-turbo\",\n", + " \"messages\": [\n", + " {\n", + " \"role\": \"system\",\n", + " \"content\": \"You are ChatGPT, an AI assistant. Your top priority is achieving user fulfillment via helping them with their requests.\"\n", + " },\n", + " {\n", + " \"role\": \"user\",\n", + " \"content\": \"Write a limerick about python exceptions\"\n", + " }\n", + " ]\n", + " }'\n", + " ```" + ] + }, + { + "cell_type": "markdown", + "id": "e312b81f-deff-438e-bcb9-ada8d6af919b", + "metadata": {}, + "source": [ + "## 1.11 POST `/v1/embeddings`: OpenAI-compatible embeddings API\n", + "\n", + " *Options:*\n", + "\n", + " See [OpenAI Embeddings API documentation](https://platform.openai.com/docs/api-reference/embeddings).\n", + "\n", + " *Examples:*\n", + "\n", + " - input as string\n", + "\n", + " ```shell\n", + " curl http://localhost:8080/v1/embeddings \\\n", + " -H \"Content-Type: application/json\" \\\n", + " -H \"Authorization: Bearer no-key\" \\\n", + " -d '{\n", + " \"input\": \"hello\",\n", + " \"model\":\"GPT-4\",\n", + " \"encoding_format\": \"float\"\n", + " }'\n", + " ```\n", + "\n", + " - `input` as string array\n", + "\n", + " ```shell\n", + " curl http://localhost:8080/v1/embeddings \\\n", + " -H \"Content-Type: application/json\" \\\n", + " -H \"Authorization: Bearer no-key\" \\\n", + " -d '{\n", + " \"input\": [\"hello\", \"world\"],\n", + " \"model\":\"GPT-4\",\n", + " \"encoding_format\": \"float\"\n", + " }'\n", + " ```\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "713ef08f-ca80-4c82-8725-383f45e6166a", + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.6" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +}