Event Indexing Issue causing UNIQUE constraint violation in Firefly Core

[vecktor]

Summary:

An issue has been identified in the interaction between the Firefly-fabconnect and Firefly Core services. Firefly-fabconnect is not correctly populating the transactionIndex and eventIndex for events within a single block, causing a UNIQUE constraint violation in the Firefly Core events table.

Details:

Firefly Core has a UNIQUE constraint on a set of fields for its events table. Each event processed by Firefly Core should have a unique combination of these fields to avoid conflicts. Firefly-fabconnect is responsible for generating and assigning unique transactionIndex and eventIndex values for each event in a block. However, it has been observed that Firefly-fabconnect is not correctly populating these indices.

Event creation logic in fabconnect:

internal/events/subscription.go:133

event := &eventsapi.EventEntry{
	ChaincodeId:   ccEvent.ChaincodeID,
	BlockNumber:   ccEvent.BlockNumber,
	TransactionId: ccEvent.TxID,
	EventName:     ccEvent.EventName,
	Payload:       ccEvent.Payload,
}

Debugger inspector representation of the event in fabconnect:

event = {*api.EventEntry | 0x1400001b6c0} 
 ChaincodeId = {string} "token-erc-721-v1"
 BlockNumber = {uint64} 81
 TransactionId = {string} "421b2cb8ce1bcc1332712572b63124e5a1fa70e32eb48626703a56f39ec94059"
 TransactionIndex = {int} 0
 EventIndex = {int} 0
 EventName = {string} "TokenMinted"
 Payload = {interface{} | []uint8} len:278, cap:288
 Timestamp = {int64} 1684224393739164000
 SubID = {string} ""

What is received in Firefly Core:

messages = {[]interface{}} len:2, cap:2
 0 = {interface{} | map[string]interface{}} 
  0 = transactionIndex -> 0
  1 = timestamp -> 1684154093804696000
  2 = chaincodeId -> token-erc-721-v1
  3 = blockNumber -> 80
  4 = transactionId -> cfdf3b5bf26014af280ad88d223db65f4efbe59e92f41263ca5006995a9078d1
  5 = eventIndex -> 0
  6 = eventName -> TokenMinted
  7 = payload -> eyJ0byI6Ing1MDk6OkNOPWNyZWF0b3IsT1U9Y2xpZW50OjpDTj1mYWJyaWNfY2Eub3JnMS5leGFtcGxlLmNvbSxPVT1IeXBlcmxl
  8 = subId -> sb-c85b1648-8020-4c68-61fb-cb2f26856e93
 1 = {interface{} | map[string]interface{}} 
  0 = chaincodeId -> token-erc-721-v1
  1 = blockNumber -> 80
  2 = transactionId -> 638aaa7b5ee53edcd104e98f84b4d8c3fb7a311bb90525f0144ac7cda92ea891
  3 = eventIndex -> 0
  4 = eventName -> TokenMinted
  5 = timestamp -> 1684154093804696000
  6 = subId -> sb-c85b1648-8020-4c68-61fb-cb2f26856e93
  7 = transactionIndex -> 0
  8 = payload -> eyJ0byI6Ing1MDk6OkNOPWNyZWF0b3IsT1U9Y2xpZW50OjpDTj1mYWJyaWNfY2Eub3JnMS5leGFtcGxlLmNvbSxPVT1IeXBlcmxl

The error that is thrown:

[2023-05-16T11:09:31.707+03:00] ERROR SQL insert failed: UNIQUE constraint failed: blockchainevents.namespace, blockchainevents.listener_id, blockchainevents.protocol_id sql=[ INSERT INTO blockchainevents (id,source,namespace,name,protocol_id,listener_id,output,info,timestamp,tx_type,tx_id,tx_blockchain_id) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12) ]: UNIQUE constraint failed: blockchainevents.namespace, blockchainevents.listener_id, blockchainevents.protocol_id dbtx=KzK541m0 ns=default pid=48305 role=event-manager

Impact:

This issue is preventing Firefly Core from processing multiple events of the same type within a single block. This could potentially lead to loss of data or incomplete data processing, impacting the overall functionality and data integrity of the system.

Recommended Actions:

The root cause of the issue appears to be with Firefly-fabconnect's event indexing. It is recommended to investigate why Firefly-fabconnect is not correctly populating transactionIndex and eventIndex for each event within a block. Once identified, a fix should be implemented to ensure that each event in a block is assigned a unique pair of indexes. This will allow Firefly Core to process and insert all events correctly without violating the UNIQUE constraint on its events table.

[penumbra23]

I've identified the sporadic place where this issue occurs.

The processNewEvents method digests the events from two possible streams:

• when a new blocks arrives
• when a chaincode event gets emitted

Inside the chaincode event stream, when assembling the event model here the TransactionIndex and EventIndex fields are missing because, by the protobuf spec, these fields don't get emitted. One possible solution would be to fetch the particular block, parse it and find the event and transaction where the ccEvent occurs.

We tried a different approach, to force the blockEventNotifier stream (by commenting out the first if block, leaving only the block notifier stream) and it works.

We ran a 15 minutes load test for Transfer and with a 6-15 tx/block count, everything went successful both on Fabconnect and Firefly.

Still there are some questions, primarily, is that a valid approach? Can we "force" to use the block notifier with some configuration variable? What are the information that we're losing by removing the chaincode events stream?

Any idea? @nguyer @PanGan21

cc: @jurosm