Successful task pods without completions or follow up tasks

[SandraGH5]

We are having issues with workflows where task pods complete successfully (output.pbs uploaded with correct content) but somehow these completions are never registered and follow-up tasks don't start.

The web console keeps showing these tasks as running. The only reference in logs that I can find is from FlyteAdmin pod and make little sense to me:

{"json":{"src":"execution_manager.go:1198"},"level":"debug","msg":"failed to emit workflow execution duration stat, couldn't find or create counter","ts":"2021-11-30T15:57:49Z"}

Does anyone have an idea?

Also, found this:

{"json":{"exec_id":"jliy3xzp3j","node":"n13","ns":"flyte-stetson-development","res_ver":"49062079","routine":"worker-13","src":"pre_post_execution.go:128","tasktype":"python-tas
k","wf":"flyte-stetson:development:src.workflows.stetson.stetson_workflow"},"level":"error","msg":"Failed to check if the output file exists. Error: error file @[s3://wolt-dev-f
lyte-flyte-cluster/metadata/propeller/flyte-stetson-development-jliy3xzp3j/n13/data/0/error.pb] is too large [19807025] bytes, max allowed [10485760] bytes","ts":"2021-11-30T16:
10:03Z"}

@bimtauer

[SandraGH5]

Looks like the error message is too large. Try changing the failure back off configuration. Propeller is trying to recover.
https://github.com/flyteorg/flytepropeller/blob/a2bfb996c77172419ea5d1562ce6af38e6eb1119/pkg/controller/nodes/task/pre_post_execution.go#L108

This can be controlled using:

flyte/kustomize/overlays/eks/flyte/config/propeller/core.yaml

Line 34 in 152aa6a

max-delay: 120s

The pod could be shown as complete without any errors because maximum failure retries are configured to be 50. Refer to discussion #1876

If you have a large string with large outputs, you could configure propeller storage to increase the limit, although this is not recommended. It is better to offload using a file.

@kumare3

[kumare3]

Actually in this case the actual outputs generated were too large. So Flyte creates outputs in 2 parts

1. Meta-ouputs and the 2. Raw outputs.
   Meta outputs are primitives and scalar values. While raw outputs, is files, dataframes, directories and datasets.
   Meta outputs should be restricted in size, as they can be visualized in the UI and loaded in the engine. They store references to raw outputs. the raw outputs can have arbitrary size.

For very large outputs, it might be better to use Files, Directories or dataframe abstraction, as that will automatically offload the data to rawoutput paths.

[kumare3]

Also refer to #1876