Question and Answers

Question: "Tasks failed with timeout"

Question: "Tasks failed with out of memory"

Question: "Tasks failed with ‘error’: ‘Too many concurrent requests in flight (count: X, allowed: Y).’}"

Cause

PyWren generates too many concurrent invocations and hit limitation of maximum concurrent invocations.

Steps to resolve

• Try to reduce number of invocations generated by PyWren. For example increasing chunk size when reading from IBM COS will generate less invocations where each invocation process the requested chunk size. This might be tricky, as increasing chunk size, may also need to increase overall memory required for the invocation. Increasing chunk size will also make an invocation to process more data which will increase running time.
• Increase retry times and numnber of retries. Limitation of this approach is that you may hit maximal running time. This will also might make entire flow slower, due to longer retry times. Follow 'retry_sleeps' and 'retires' from here.