Skip to content

Commit 5f84994

Browse files
PSeitzfmassot
andauthored
update cloud env (#1057)
Co-authored-by: François Massot <[email protected]>
1 parent 4febf28 commit 5f84994

File tree

1 file changed

+25
-10
lines changed

1 file changed

+25
-10
lines changed

docs/administration/cloud-env.md

Lines changed: 25 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,25 +1,40 @@
11
---
2-
title: Operating in the cloud
2+
title: Cloud operation costs
33
sidebar_position: 1
44
---
55

6-
Quickwit has only been tested on AWS S3. This page sums up what we have learned from that.
6+
Quickwit has been tested on Amazon S3. This page sums up what we have learned from that experience.
77

88
## Data transfers costs and latency
99

10-
Cloud providers charge for data transfers in and out of their networks. In addition, querying an index from a remote machine adds some extra latency. For those reasons, we recommend that you test and use the Quickwit from an instance located within your cloud provider's network.
10+
Cloud providers charge for data transfers in and out of their networks. In addition, querying an index from a remote machine adds some extra latency.
11+
For those reasons, we recommend that you test and use the Quickwit from an instance located within your cloud provider's network.
1112

1213
## Optimizing bandwidth with wisely chosen instances
1314

14-
To get the best performance out of Quickwit search from object storage, we recommend picking an instance with high network bandwidth.
15-
In our experience, `c5n.2xlarge` instances offer the bigger bang for your buck.
15+
We recommend picking instances with high network performance to allow faster downloads from Amazon S3. In our experience, `c5n.2xlarge` instances offer the best bang for your buck.
1616

17-
## GET/PUT requests costs
17+
## Requests cost
1818

19-
A final note on object storage requests costs. These are [quite low](https://aws.amazon.com/s3/pricing/) actually, $0,0004 / 1000 requests for GET and $0.005 / 1000 requests for PUT on AWS S3, so you don't need to worry too much about it.
19+
A final note on object storage requests costs. These are [quite low](https://aws.amazon.com/s3/pricing/) actually, $0,0004 / 1000 requests for GET and $0.005 / 1000 requests for PUT on AWS S3.
2020

21-
Indeed when indexing, Quickwit generates [splits](../overview/architecture.md#splits) of 5 millions documents each and then
22-
upload them. As they are composed of 9 files, this generates 9 PUT requests per split. When querying one term, Quickwit only needs to make 3 GET requests per split.
21+
### PUT requests
2322

24-
Of course, these requests could add up quickly if you have a large amount of requests.
23+
During indexing, Quickwit uploads new splits on Amazon S3 and progressively merges them until they reach 10 million documents that we call “mature splits”. Such splits have a typical size between 1GB and 10GB and will usually require 2 PUT requests to be uploaded (1 PUT request / 5GB).
24+
25+
With default indexing parameters `commit_timeout_secs` of 60 seconds and `merge_policy.merge_factor` of 10 and assuming you want to ingest 1 million documents every minute, this will cost you less than $1 / month.
26+
27+
### GET requests
28+
29+
When querying, Quickwit needs to make multiple GET requests:
30+
31+
```jsx
32+
#num requests = #num splits * ((#num search fields * #num terms * 3) + #num fast fields)
33+
```
34+
35+
The above formula assumes that the hotcache is cached, which will be loaded after the first query for every split.
36+
37+
When positions are not enabled, only 2 GET requests will be executed per term.
38+
39+
These requests costs could add up quickly if you have a high number of splits or QPS > 10.
2540
Don't hesitate to [contact us](mailto:[email protected]) if this is the case :).

0 commit comments

Comments
 (0)