Send func panic

[billowqiu]

Expected behavior

Tell us what should happen

Actual behavior

panic: sync: negative WaitGroup counter
goroutine 2204 [running]:
sync.(*WaitGroup).Add(0x18dbfc0, 0xc0020d8d40)
sync/waitgroup.go:74 +0x105
sync.(*WaitGroup).Done(...)
sync/waitgroup.go:99
github.com/apache/pulsar-client-go/pulsar.(*partitionProducer).Send.func1({0x1965558, 0xc01a1ab1d0}, 0xc01a841160, {0x0, 0x0})
github.com/apache/pulsar-client-go@v0.7.0/pulsar/producer_partition.go:722 +0x7e
github.com/apache/pulsar-client-go/pulsar.(*partitionProducer).ReceivedSendReceipt(0xc001017b00, 0xc017280480)
github.com/apache/pulsar-client-go@v0.7.0/pulsar/producer_partition.go:828 +0x7ef
github.com/apache/pulsar-client-go/pulsar/internal.(*connection).handleSendReceipt(0xc001188420, 0xc017280480)
github.com/apache/pulsar-client-go@v0.7.0/pulsar/internal/connection.go:673 +0xe8
github.com/apache/pulsar-client-go/pulsar/internal.(*connection).internalReceivedCommand(0xc001188420, 0xc006b31680, {0x0, 0x0})
github.com/apache/pulsar-client-go@v0.7.0/pulsar/internal/connection.go:558 +0x14a
github.com/apache/pulsar-client-go/pulsar/internal.(*connection).run(0xc001188420)
github.com/apache/pulsar-client-go@v0.7.0/pulsar/internal/connection.go:415 +0x3ba
github.com/apache/pulsar-client-go/pulsar/internal.(*connection).start.func1()
github.com/apache/pulsar-client-go@v0.7.0/pulsar/internal/connection.go:227 +0x65
created by github.com/apache/pulsar-client-go/pulsar/internal.(*connection).start
github.com/apache/pulsar-client-go@v0.7.0/pulsar/internal/connection.go:223 +0x75

Steps to reproduce

How can we reproduce the issue

System configuration

Pulsar version: x.y

[xujianhai666]

mark, fix later

[wolfstudy]

When this problem is triggered, we can always see the following log information:

level=info msg="Failing 2499 messages" producerID=8 producer_name=xxxxxx
level=info msg="Failing 2499 messages" producerID=8 producer_name=xxxxxx
level=info msg="Failing 2499 messages" producerID=8 producer_name=xxxxxx
....
level=warning msg="[Connection was closed]" cnx="11.9.4.1:60439 -> 11.9.4.3:6650" producerID=6 producer_name=xxx
level=warning msg="[Connection was closed]" cnx="11.9.4.1:60439 -> 11.9.4.3:6650" producerID=6 producer_name=xxx
level=warning msg="[Connection was closed]" cnx="11.9.4.1:60439 -> 11.9.4.3:6650" producerID=6 producer_name=xxx

It seems to happen when the connection is reconnected. In order to catch the error when the connection is reconnected, cherry-pick has made this change #648. Not sure now, when the reconnection fails, the code logic of #648 will take effect, whether it will trigger this issue.

[billowqiu]

So, should we check why this is called multiple times; Is it reasonable to call multiple times