Skip to content

Commit a3c66a5

Browse files
vvnginxvvnginx
authored
fix: Update tech-specs.md
Click house tuning settings added
1 parent a9fda9a commit a3c66a5

File tree

1 file changed

+116
-0
lines changed

1 file changed

+116
-0
lines changed

‎content/nim/fundamentals/tech-specs.md‎

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -132,6 +132,122 @@ The table below shows the estimated storage requirements for **NGINX OSS**, base
132132
| | 250 | 14 | 4 GiB |
133133
{{</bootstrap-table>}}
134134

135+
#### ClickHouse Tunings {#Clickhouse-tuning}
136+
137+
The default configuration that comes with clickhouse works with NIM efficiently. But if you make any changes to the configuration and clickhouse is running out of memory, please follow below steps.
138+
There are system tables in ClickHouse that provide logging and telemetry mechanisms to monitor and debug the behavior of the database. These logs are not typical event logs that store user activity but are internal logs designed for diagnostics and telemetry related to system operations.
139+
Below is an explanation of three such tables which may cause an out of memory issue:
140+
141+
- **trace_log**: The trace_log table records detailed execution traces and profiling information. It is mainly used for query debugging and performance analysis in ClickHouse. The logs here provide insight into CPU usage and stack traces for specific parts of query execution. Stores stack traces collected by query profilers and inserted into this table. You can modify the settings for trace_log under <trace_log> section in /etc/clickhouse-server/config.xml file. The flush_interval_milliseconds controls the Interval for flushing data from the buffer in memory to the table and the default value is 7500. Any value below default can cause the excessive debug rows captured in the table and eventually eat up more memory.
142+
143+
Default settings for trace_log is as follows
144+
```shell
145+
<trace_log>
146+
<database>system</database>
147+
<table>trace_log</table>
148+
<partition_by>toYYYYMM(event_date)</partition_by>
149+
<flush_interval_milliseconds>7500</flush_interval_milliseconds>
150+
<max_size_rows>1048576</max_size_rows>
151+
<reserved_size_rows>8192</reserved_size_rows>
152+
<buffer_size_rows_flush_threshold>524288</buffer_size_rows_flush_threshold>
153+
<flush_on_crash>false</flush_on_crash>
154+
<symbolize>false</symbolize>
155+
</trace_log>
156+
```
157+
158+
You can use the below command to view the current memory occupied by each table in the clickhouse database using the below command from clickhouse-client.
159+
```shell
160+
SELECT
161+
database,
162+
table,
163+
formatReadableSize(sum(bytes_on_disk)) AS total_size
164+
FROM system.parts
165+
GROUP BY database, table
166+
ORDER BY sum(bytes_on_disk) DESC;
167+
```
168+
If you observe that this table is utilizing more memory, you can configure the TTL to ensure that outdated records are removed after the specified TTL duration. The TTL configuration guarantees that your table does not expand excessively and automatically deletes old records following the TTL.
169+
170+
```shell
171+
ALTER TABLE system.trace_log
172+
MODIFY TTL event_time + INTERVAL 7 DAY;
173+
```
174+
You can also relieve the memory if its running very low using the below command. Change the interval in the command to how many days of records you want to retain and delete the remaining records.
175+
```shell
176+
ALTER TABLE system.trace_log DELETE WHERE event_time < now() - INTERVAL 30 DAY;
177+
```
178+
- **metric_log**: The `system.metric_log` contains history of metrics values from tables system.metrics and system.events, periodically flushed to disk. It serves as a time-series table that periodically records historical data from system.metrics and system.events over time. This table acts like an essential tool providing historical tracking of captured metrics and event data, making it easier to debug performance trends, spikes, or irregularities. Too much of historical data in this table can occupy more memory and make the clickhouse run out of memory if not properly configured.
179+
180+
Default settings for metric_log is as follows
181+
```shell
182+
<metric_log>
183+
<database>system</database>
184+
<table>metric_log</table>
185+
<flush_interval_milliseconds>7500</flush_interval_milliseconds>
186+
<max_size_rows>1048576</max_size_rows>
187+
<reserved_size_rows>8192</reserved_size_rows>
188+
<buffer_size_rows_flush_threshold>524288</buffer_size_rows_flush_threshold>
189+
<collect_interval_milliseconds>1000</collect_interval_milliseconds>
190+
<flush_on_crash>false</flush_on_crash>
191+
</metric_log>
192+
```
193+
194+
You can use the below command to view the current memory occupied by each table in the clickhouse database using the below command from clickhouse-client.
195+
```shell
196+
SELECT
197+
database,
198+
table,
199+
formatReadableSize(sum(bytes_on_disk)) AS total_size
200+
FROM system.parts
201+
GROUP BY database, table
202+
ORDER BY sum(bytes_on_disk) DESC;
203+
```
204+
205+
If it is observed that system.metric_log table is taking more memory, you can set the TTL(Time to Live) so that old records will be deleted after the TTL time. The TTL setting makes sure your table does not grow uncontrollably and delete the old records automatically after the TTL.
206+
```shell
207+
ALTER TABLE system.metric_log
208+
MODIFY TTL event_time + INTERVAL 7 DAY;
209+
```
210+
211+
You can also relieve the memory immediately if its running very low using the below command. Change the interval in the command to how many days of records you want to retain and delete the remaining records.
212+
```shell
213+
ALTER TABLE system.metric_log DELETE WHERE event_time < now() - INTERVAL 30 DAY;
214+
```
215+
Note: We also have a table called nms.metrics table which stores the metrics collected from the data plane in NIM. Make sure you don't delete the records accidentally from nms.metrics table as deleting from nms.metrics will loose the metrics from the data planes.
216+
217+
- **text_log**: The text_log table contains general logging information, including warnings, errors, system messages, and query-processed events. It is a human-readable diagnostic log for operational debugging. The logging level which goes to this table can be limited to the text_log.level server setting as shown in the below xml snippet.
218+
```shell
219+
<text_log>
220+
<database>system</database>
221+
<table>text_log</table>
222+
<flush_interval_milliseconds>7500</flush_interval_milliseconds>
223+
<max_size_rows>1048576</max_size_rows>
224+
<reserved_size_rows>8192</reserved_size_rows>
225+
<buffer_size_rows_flush_threshold>524288</buffer_size_rows_flush_threshold>
226+
<flush_on_crash>false</flush_on_crash>
227+
<level>trace</level>
228+
</text_log>
229+
```
230+
You can use the below command to view the current memory occupied by each table in the clickhouse database using the below command from clickhouse-client.
231+
```shell
232+
SELECT
233+
database,
234+
table,
235+
formatReadableSize(sum(bytes_on_disk)) AS total_size
236+
FROM system.parts
237+
GROUP BY database, table
238+
ORDER BY sum(bytes_on_disk) DESC;
239+
```
240+
241+
If it is observed that this table is taking more memory, you can set the TTL(Time to Live) so that old records will be deleted after the TTL time. The TTL setting makes sure your table does not grow uncontrollably and delete the old records automatically after the TTL.
242+
```shell
243+
ALTER TABLE system.text_log
244+
MODIFY TTL event_time + INTERVAL 7 DAY;
245+
```
246+
You can also relieve the memory immediately if its running very low using the below command. Change the interval in the command to how many days of records you want to retain and delete the remaining records.
247+
```shell
248+
ALTER TABLE system.text_log DELETE WHERE event_time < now() - INTERVAL 30 DAY;
249+
```
250+
135251
## Firewall ports {#firewall}
136252

137253
NGINX Instance Manager and NGINX Agent use the Unix domain socket by default and proxy through the gateway on port `443`.

0 commit comments

Comments
 (0)