cadence的workflow执行历史中出现多个open状态的workflow

[Johnite]

如图所示，在workflow的执行历史上我们发现有多个处于open状态的workflow，但是其实际情况是已经执行完成了，并且没有pending的task。查看cadence-server的日志发现存在这样一些error。
{"level":"error","ts":"2021-08-19T03:09:02.880+0800","msg":"Persistent store operation failure","service":"cadence-history","shard-id":1467,"address":"IP:7934","component":"history-cache","wf- id":"clickhouse_sql_user:realtime:orderCnt_24h","wf-run-id":"bf464277-704b-437a-9661-8264d1c8f397","wf-domain-id":"30a01162-e822-41d2-9b40-9636e0dbce4b","store-operation":"update-wf-execution","error": "maximum attempts exceeded to update history","number":9,"logging-call-at":"context.go:1096","stacktrace":"github.com/uber/cadence/common/log/loggerimpl.(*loggerImpl).Error\n\t/Users/didi/go/src/go.uber. org/cadence-0.20.0/common/log/loggerimpl/logger.go:131\ngithub.com/uber/cadence/service/history/execution.(*contextImpl).updateWorkflowExecutionWithRetry\n\t/Users/didi/go/src/go.uber.org/cadence-0.20.0/ service/history/execution/context.go:1096\ngithub.com/uber/cadence/service/history/execution.(*contextImpl).UpdateWorkflowExecutionWithNew\n\t/Users/didi/go/src/go.uber.org/cadence-0.20.0/service/history/execution/context.go:741\ngithub.com/uber/cadence/service/history/execution.(*contextImpl).UpdateWorkflowExecutionAsActive\n\t/Users/didi/go/src/go.uber.org/cadence-0.20.0/service/history/execution/ context.go:584\ngithub.com/uber/cadence/service/history/workflow.updateHelper\n\t/Users/didi/go/src/go.uber.org/cadence-0.20.0/service/history/workflow/util.go:255\ngithub.com/uber/cadence/service/ history/workflow.UpdateWithActionFunc\n\t/Users/didi/go/src/go.uber.org/cadence-0.20.0/service/history/workflow/util.go:147\ngithub.com/uber/cadence/service/history/decision.(*handlerImpl). HandleDecisionTaskStarted\n\t/Users/didi/go/src/go.uber.org/cadence-0.20.0/service/history/decision/handler.go:169\ngithub.com/uber/cadence/service/history.(*historyEngineImpl). RecordDecisionTaskStarted\n\t/Users/didi/go/src/go.uber.org/cadence-0.20.0/service/history/historyEngine.go:1685\ngithub.com/uber/cadence/service/history.(*handlerImpl).RecordDecisionTaskStarted\n\t/ Users/didi/go/src/go.uber.org/cadence-0.20.0/service/history/handler.go:399\ngithub.com/uber/cadence/service/history.ThriftHandler.RecordDecisionTaskStarted\n\t/Users/didi/go/src/go.uber.org/cadence-0.20.0/service/history/thriftHandler.go:169\ngithub.com/uber/cadence/.gen/go/history/historyserviceserver.handler.RecordDecisionTaskStarted\n\t/Users/didi/go/src/go.uber.org/cadence-0.20.0/.gen/go/history/ historyserviceserver/server.go:1239\ngo.uber.org/yarpc/encoding/thrift.thriftUnaryHandler.Handle\n\t/Users/didi/go/pkg/mod/go.uber.org/[email protected]/encoding/thrift/inbound. go:64\ngo.uber.org/yarpc/internal/observability.(*Middleware).Handle\n\t/Users/didi/go/pkg/mod/go.uber.org/[email protected]/internal/observability/middleware.go:166\ngo.uber. org/yarpc/api/middleware.unaryHandlerWithMiddleware.Handle\n\t/Users/didi/go/pkg/mod/go.uber.org/[email protected]/api/middleware/inbound.go:71\ngo.uber.org/yarpc/api/transport. InvokeUnaryHandler\n\t/Users/didi/go/pkg/mod/go.uber.org/[email protected]/api/transport/handler_invoker.go:70\ngo.uber.org/yarpc/transport/tchannel.handler.callHandler\n\t/ Users/didi/go/pkg/mod/go.uber.org/[email protected]/transport/tchannel/handler.go:233\ngo.uber.org/yarpc/transport/tchannel.handler.handle\n\t/Users/didi/go/pkg/mod/go.uber.org/ [email protected]/transport/tchannel/handler.go:121\ngo.uber.org/yarpc/transport/tchannel.handler.Handle\n\t/Users/didi/go/pkg/mod/go.uber.org/[email protected] b2caa40d56b6/transport/tchannel/handler.go:110\ngithub.com/uber/tchannel-go.channelHandler.Handle\n\t/Users/didi/go/pkg/mod/github.com/uber/[email protected]/handlers.go:126\ngithub.com/uber/tchannel- go.(*Connection).dispatchInbound\n\t/Users/didi/go/pkg/mod/github.com/uber/[email protected]/inbound.go:203"}
{"level":"error","ts":"2021-08-19T03:09:02.881+0800","msg":**"Uncategorized error","service":"cadence-history","wf-domain-id":"30a01162-e822-41d2-9b40-9636e0dbce4b","wf-id":"clickhouse_sql_user:realtime: orderCnt_24h","error":"maximum attempts exceeded to update history","l**ogging-call-at":"handler.go:1911","stacktrace":"github.com/uber/cadence/common/log/loggerimpl.(*loggerImpl).Error\n\t/Users/didi/go/ src/go.uber.org/cadence-0.20.0/common/log/loggerimpl/logger.go:131\ngithub.com/uber/cadence/service/history.(*handlerImpl).updateErrorMetric\n\t/Users/didi/go/src/go.uber.org/cadence-0.20.0/service/ history/handler.go:1911\ngithub.com/uber/cadence/service/history.(*handlerImpl).error\n\t/Users/didi/go/src/go.uber.org/cadence-0.20.0/service/history/handler.go:1923\ngithub.com/uber/cadence/service/ history.(*handlerImpl).RecordDecisionTaskStarted\n\t/Users/didi/go/src/go.uber.org/cadence-0.20.0/service/history/handler.go:401\ngithub.com/uber/cadence/service/history.ThriftHandler. RecordDecisionTaskStarted\n\t/Users/didi/go/src/go.uber.org/cadence-0.20.0/service/history/thriftHandler.go:169\ngithub.com/uber/cadence/.gen/go/history/historyserviceserver.handler. RecordDecisionTaskStarted\n\t/Users/didi/go/src/go.uber.org/cadence-0.20.0/.gen/go/history/historyserviceserver/server.go:1239\ngo.uber.org/yarpc/encoding/thrift.thriftUnaryHandler.Handle\n\t/Users/didi/ go/pkg/mod/go.uber.org/[email protected]/encoding/thrift/inbound.go:64\ngo.uber.org/yarpc/internal/observability.(*Middleware).Handle\n\t/Users/didi/go/pkg/mod/go.uber.org/[email protected]/internal/observability/middleware.go:166\ngo.uber.org/yarpc/api/middleware.unaryHandlerWithMiddleware.Handle\n\t/Users/didi/go/pkg/mod/go.uber.org/[email protected]. 20210303193224-b2caa40d56b6/api/middleware/inbound.go:71\ngo.uber.org/yarpc/api/transport.InvokeUnaryHandler\n\t/Users/didi/go/pkg/mod/go.uber.org/[email protected]/api/ transport/handler_invoker.go:70\ngo.uber.org/yarpc/transport/tchannel.handler.callHandler\n\t/Users/didi/go/pkg/mod/go.uber.org/[email protected]/transport/tchannel/handler.go: 233\ngo.uber.org/yarpc/transport/tchannel.handler.handle\n\t/Users/didi/go/pkg/mod/go.uber.org/[email protected]/transport/tchannel/handler.go:121\ngo.uber.org/yarpc/transport/ tchannel.handler.Handle\n\t/Users/didi/go/pkg/mod/go.uber.org/[email protected]/transport/tchannel/handler.go:110\ngithub.com/uber/tchannel-go.channelHandler.Handle\n\t/Users/ didi/go/pkg/mod/github.com/uber/[email protected]/handlers.go:126\ngithub.com/uber/tchannel-go.(*Connection).dispatchInbound\n\t/Users/didi/go/pkg/mod/github.com/uber/[email protected]/inbound.go: 203"}

这是什么原因呢？

[longquanzheng]

Can you describe the workflow to check if it’s actually closed?

Using basic visibility may cause this because sampling is enabled by default.

If that’s the case, you can disable it in the dynamic config
See EnableVisibilitySampling
In https://pkg.go.dev/github.com/uber/[email protected]/common/dynamicconfig#Key

[Johnite]

这个代码文件里面我看是system的配置，是在dynamicconfig的配置中增加：
system. EnableVisibilitySampling

• value: false
  constraints: {}

因为我在这个配置中看到的都是frontend，history，matching，worker的配置，system还真没见过，不太确定是否这样写。

[longquanzheng]

First step:For the open workflows in web UI, did you check if the workflow is actually closed? (Just appeared to be open on the list)

[longquanzheng]

I am disabling it by default from next release #4407