|
1 | 1 | --- |
2 | | -title: Operating in the cloud |
| 2 | +title: Cloud operation costs |
3 | 3 | sidebar_position: 1 |
4 | 4 | --- |
5 | 5 |
|
6 | | -Quickwit has only been tested on AWS S3. This page sums up what we have learned from that. |
| 6 | +Quickwit has been tested on Amazon S3. This page sums up what we have learned from that experience. |
7 | 7 |
|
8 | 8 | ## Data transfers costs and latency |
9 | 9 |
|
10 | | -Cloud providers charge for data transfers in and out of their networks. In addition, querying an index from a remote machine adds some extra latency. For those reasons, we recommend that you test and use the Quickwit from an instance located within your cloud provider's network. |
| 10 | +Cloud providers charge for data transfers in and out of their networks. In addition, querying an index from a remote machine adds some extra latency. |
| 11 | +For those reasons, we recommend that you test and use the Quickwit from an instance located within your cloud provider's network. |
11 | 12 |
|
12 | 13 | ## Optimizing bandwidth with wisely chosen instances |
13 | 14 |
|
14 | | -To get the best performance out of Quickwit search from object storage, we recommend picking an instance with high network bandwidth. |
15 | | -In our experience, `c5n.2xlarge` instances offer the bigger bang for your buck. |
| 15 | +We recommend picking instances with high network performance to allow faster downloads from Amazon S3. In our experience, `c5n.2xlarge` instances offer the best bang for your buck. |
16 | 16 |
|
17 | | -## GET/PUT requests costs |
| 17 | +## Requests cost |
18 | 18 |
|
19 | | -A final note on object storage requests costs. These are [quite low](https://aws.amazon.com/s3/pricing/) actually, $0,0004 / 1000 requests for GET and $0.005 / 1000 requests for PUT on AWS S3, so you don't need to worry too much about it. |
| 19 | +A final note on object storage requests costs. These are [quite low](https://aws.amazon.com/s3/pricing/) actually, $0,0004 / 1000 requests for GET and $0.005 / 1000 requests for PUT on AWS S3. |
20 | 20 |
|
21 | | -Indeed when indexing, Quickwit generates [splits](../overview/architecture.md#splits) of 5 millions documents each and then |
22 | | -upload them. As they are composed of 9 files, this generates 9 PUT requests per split. When querying one term, Quickwit only needs to make 3 GET requests per split. |
| 21 | +### PUT requests |
23 | 22 |
|
24 | | -Of course, these requests could add up quickly if you have a large amount of requests. |
| 23 | +During indexing, Quickwit uploads new splits on Amazon S3 and progressively merges them until they reach 10 million documents that we call “mature splits”. Such splits have a typical size between 1GB and 10GB and will usually require 2 PUT requests to be uploaded (1 PUT request / 5GB). |
| 24 | + |
| 25 | +With default indexing parameters `commit_timeout_secs` of 60 seconds and `merge_policy.merge_factor` of 10 and assuming you want to ingest 1 million documents every minute, this will cost you less than $1 / month. |
| 26 | + |
| 27 | +### GET requests |
| 28 | + |
| 29 | +When querying, Quickwit needs to make multiple GET requests: |
| 30 | + |
| 31 | +```jsx |
| 32 | +#num requests = #num splits * ((#num search fields * #num terms * 3) + #num fast fields) |
| 33 | +``` |
| 34 | + |
| 35 | +The above formula assumes that the hotcache is cached, which will be loaded after the first query for every split. |
| 36 | + |
| 37 | +When positions are not enabled, only 2 GET requests will be executed per term. |
| 38 | + |
| 39 | +These requests costs could add up quickly if you have a high number of splits or QPS > 10. |
25 | 40 | Don't hesitate to [contact us ](mailto:[email protected]) if this is the case :). |
0 commit comments