Question and Answers

Question: "Tasks failed with timeout"

Question: "Tasks failed with out of memory"

Question: "Tasks failed with ‘error’: ‘Too many concurrent requests in flight (count: X, allowed: Y).’}"

Cause

PyWren generates too many concurrent requests and hit limitation of maximum concurrent requests.

Steps to resolve

• Try to reduce number of invocations generated by PyWren. For example increasing chunk size when reading from IBM COS will generate less invocations where each invocation process the requested chunk size. This might be tricky, as increasing chunk size, may also need to increase overall memory required for the invocation.
• Increase retry times and numnber of retries. Limitations of this approach is that you may hit maximal running time. This will also make entire flow slower, due to longer retries. Follow 'retry_sleeps' and 'retires' from here.