Make the client print an error and continue if it gets a 504 timeout

[tomhamer]

On Marqo cloud we often see a timeout on the first request due to the fact that Marqo has to download the required models. In the case when we get the timeout during indexing, the request still gets sent through and its likely the documents got indexed.

Therefore, I recommend during client side batching we just print a warning that there was a timeout and the client should continue with the next batch.

2022-12-09 16:21:11,737 logger:'marqo' INFO starting batch ingestion with batch size 10
Traceback (most recent call last):
File "/Users/thomashamer/opt/miniconda3/lib/python3.9/site-packages/marqo/_httprequests.py", line 128, in __validate
request.raise_for_status()
File "/Users/thomashamer/opt/miniconda3/lib/python3.9/site-packages/requests/models.py", line 960, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 504 Server Error: Gateway Timeout for url: https://zio3sslj5c.execute-api.us-east-1.amazonaws.com/prod/indexes/jesse-raritypunks-big-test/documents?refresh=false&device=cpu&non_tensor_fields=description&non_tensor_fields=attributes

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/Users/thomashamer/Documents/rarity.py", line 128, in
responses = client.index(index_name).add_documents(_d.tolist(), client_batch_size=batch_size, device='cpu', non_tensor_fields=["description", "attributes"])
File "/Users/thomashamer/opt/miniconda3/lib/python3.9/site-packages/marqo/index.py", line 221, in add_documents
return self._generic_add_update_docs(
File "/Users/thomashamer/opt/miniconda3/lib/python3.9/site-packages/marqo/index.py", line 289, in _generic_add_update_docs
return self._batch_request(
File "/Users/thomashamer/opt/miniconda3/lib/python3.9/site-packages/marqo/index.py", line 393, in _batch_request
results = [verbosely_add_docs(i, docs) for i, docs in enumerate(batched)]
File "/Users/thomashamer/opt/miniconda3/lib/python3.9/site-packages/marqo/index.py", line 393, in
results = [verbosely_add_docs(i, docs) for i, docs in enumerate(batched)]
File "/Users/thomashamer/opt/miniconda3/lib/python3.9/site-packages/marqo/index.py", line 378, in verbosely_add_docs
res = self.http.post(path=path_with_query_str, body=docs)
File "/Users/thomashamer/opt/miniconda3/lib/python3.9/site-packages/marqo/_httprequests.py", line 96, in post
return self.send_request(s.post, path, body, content_type)
File "/Users/thomashamer/opt/miniconda3/lib/python3.9/site-packages/marqo/_httprequests.py", line 74, in send_request
return self.__validate(response)
File "/Users/thomashamer/opt/miniconda3/lib/python3.9/site-packages/marqo/_httprequests.py", line 131, in __validate
convert_to_marqo_error_and_raise(response=request, err=err)
File "/Users/thomashamer/opt/miniconda3/lib/python3.9/site-packages/marqo/_httprequests.py", line 145, in convert_to_marqo_error_and_raise
raise MarqoWebError(message=response_msg, code=code, error_type=error_type,
marqo.errors.MarqoWebError: MarqoWebError: MarqoWebError Error message: {"message": "Endpoint request timed out"}
status_code: 504, type: unhandled_error_type, code: unhandled_error, link: