Add ability to try the decoder over multiple ABIs (#239)

* Change store API and add migrations

* Refactor abi loader to handle multiple abi in decoding

* Update documentation

* Update migrations

* Strategy executor returns abi with strategy id

* Add default ABIs as fallback

* Use strict decode to ensure params

---------

Co-authored-by: Anastasia Rodionova <[email protected]>

Author: Ferossgp

.changeset/old-signs-lay.md

@@ -0,0 +1,5 @@
+---
+'@3loop/transaction-decoder': minor
+---
+
+BREAKING! This version changes the public API for ABI store. If you use built in stores evrything should work out of the box. When using SQL store ensure that migrations complete on start. Additionally the AbiLoader will now also return an array of ABIs when accessing the cached data. These changes allows us to run over multiple ABIs when decoding a transaction, instead of failing when the cached ABI is wrong.

apps/docs/src/content/components/effect-abi-store-interface.md

@@ -1,8 +1,23 @@
 ```ts
 export interface AbiStore {
+  /**
+   * Set of resolver strategies grouped by chain id and a `default` bucket.
+   */
   readonly strategies: Record<ChainOrDefault, readonly ContractAbiResolverStrategy[]>
-  readonly set: (key: AbiParams, value: ContractAbiResult) => Effect.Effect<void, never>
+  /**
+   * Persist a resolved ABI or a terminal state for a lookup key.
+   */
+  readonly set: (key: AbiParams, value: CacheContractABIParam) => Effect.Effect<void, never>
+  /**
+   * Fetch all cached ABIs for the lookup key. Implementations may return multiple entries.
+   */
   readonly get: (arg: AbiParams) => Effect.Effect<ContractAbiResult, never>
+  /** Optional batched variant of `get` for performance. */
   readonly getMany?: (arg: Array<AbiParams>) => Effect.Effect<Array<ContractAbiResult>, never>
+  /** Optional helper for marking an existing cached ABI as invalid or changing its status. */
+  readonly updateStatus?: (
+    id: string | number,
+    status: 'success' | 'invalid' | 'not-found',
+  ) => Effect.Effect<void, never>
 }
 ```

apps/docs/src/content/components/vanilla-abi-store-interface.md

@@ -2,6 +2,6 @@
 export interface VanillaAbiStore {
   strategies?: readonly ContractAbiResolverStrategy[]
   get: (key: AbiParams) => Promise<ContractAbiResult>
-  set: (key: AbiParams, val: ContractAbiResult) => Promise<void>
+  set: (key: AbiParams, val: ContractABI) => Promise<void>
 }
 ```

apps/docs/src/content/docs/reference/data-loaders.mdx

@@ -10,10 +10,11 @@ import { Tabs, TabItem } from '@astrojs/starlight/components'
 Data Loaders are mechanisms for retrieving the necessary ABI and Contract Metadata for transaction decoding. They are responsible for:
 
 - Resolving ABIs and Contract Metadata using specified strategies and third-party APIs
-- Automatically batching requests when processing logs and traces in parallel
-- Caching request results in the [Data Store](/reference/data-store)
+- Automatically batching requests and deduplicating identical lookups during parallel log/trace processing
+- Caching results and negative lookups in the [Data Store](/reference/data-store)
+- Applying rate limits, circuit breaking, and adaptive concurrency via a shared request pool
 
-Loop Decoder implements optimizations to minimize API requests to third-party services. For example, when a contract’s ABI is resolved via Etherscan, it is cached in the store. If the ABI is requested again, the store provides the cached version, avoiding redundant API calls.
+Loop Decoder implements optimizations to minimize API requests to third-party services. For example, when a contract’s ABI is resolved via Etherscan, it is cached in the store. If the ABI is requested again, the store provides the cached version, avoiding redundant API calls. When a source returns no result, a `not-found` entry can be stored to skip repeated attempts temporarily.
 
 ## ABI Strategies
 

apps/docs/src/content/docs/reference/data-store.mdx

@@ -67,7 +67,7 @@ The full interface of ABI store is:
 
 <EffectABI />
 
-ABI Store Interface requires 2 methods: `set` and `get` to store and retrieve the ABI of a contract. Optionally, you can provide a batch get method `getMany` for further optimization. Because our API supports ABI fragments, the get method will receive both the contract address and the fragment signature.
+ABI Store Interface requires `set` and `get` to store and retrieve ABIs. Optionally, you can provide a batch get method `getMany` and an `updateStatus` helper used by the decoder to mark ABIs invalid after failed decode attempts. Because our API supports ABI fragments, the get method will receive both the contract address and the fragment signature.
 
 ```typescript
 interface AbiParams {

apps/docs/src/content/docs/welcome/overview.md

@@ -26,7 +26,7 @@ The Transaction Decoder package transforms raw transactions, including calldata,
 
 See the Decoded Transaction section in our [playground](https://loop-decoder-web.vercel.app/) for transaction examples.
 
-The minimal configuration for the Transaction Decoder requires an RPC URL, ABIs, and contract metadata stores, which can be in-memory (see the [Data Store](/reference/data-store/) section). We also provide data loaders for popular APIs like Etherscan (see the full list in the [ABI Strategies](/reference/data-loaders/) section). These loaders request contract metadata and cache it in your specified store. The decoder supports RPCs with both Debug (Geth tracers) and Trace (OpenEthereum/Parity and Erigon tracers) APIs.
+The minimal configuration for the Transaction Decoder requires an RPC URL, ABIs, and contract metadata stores, which can be in-memory (see the [Data Store](/reference/data-store/) section). We also provide data loaders for popular APIs like Etherscan (see the full list in the [ABI Strategies](/reference/data-loaders/) section). These loaders fetch ABIs/metadata, cache results and negative lookups in your store, and avoid querying sources previously marked invalid for a key. The decoder supports RPCs with both Debug (Geth tracers) and Trace (OpenEthereum/Parity and Erigon tracers) APIs.
 
 ### Transaction Interpreter
 

apps/web/src/lib/decode.ts

@@ -19,12 +19,7 @@ import { SqlAbiStore, SqlContractMetaStore } from '@3loop/transaction-decoder/sq
 import { Hex } from 'viem'
 import { DatabaseLive } from './database'
 
-const LogLevelLive = Layer.unwrapEffect(
-  Effect.gen(function* () {
-    const level = LogLevel.Warning
-    return Logger.minimumLogLevel(level)
-  }),
-)
+const LogLevelLive = Logger.minimumLogLevel(LogLevel.Warning)
 
 const AbiStoreLive = Layer.unwrapEffect(
   Effect.gen(function* () {
@@ -85,7 +80,7 @@ export async function decodeTransaction({
     return { decoded: result }
   } catch (error: unknown) {
     const endTime = performance.now()
-    const message = error instanceof Error ? JSON.parse(error.message) : 'Failed to decode transaction'
+    const message = error instanceof Error ? error.message : 'Failed to decode transaction'
     console.log(message)
     console.log(`Failed decode transaction took ${endTime - startTime}ms`)
     return {

packages/transaction-decoder/src/abi-loader.ts

@@ -1,9 +1,10 @@
-import { Effect, Either, RequestResolver, Request, Array, pipe, Data, PrimaryKey, Schema, SchemaAST } from 'effect'
+import { Effect, Either, RequestResolver, Request, Array, Data, PrimaryKey, Schema, SchemaAST } from 'effect'
 import { ContractABI } from './abi-strategy/request-model.js'
-import { Abi } from 'viem'
+import { Abi, erc20Abi, erc721Abi } from 'viem'
 import * as AbiStore from './abi-store.js'
 import * as StrategyExecutorModule from './abi-strategy/strategy-executor.js'
 import { SAFE_MULTISEND_SIGNATURE, SAFE_MULTISEND_ABI, AA_ABIS } from './decoding/constants.js'
+import { errorFunctionSignatures, solidityError, solidityPanic } from './helpers/error.js'
 
 interface LoadParameters {
   readonly chainID: number
@@ -34,7 +35,7 @@ export class EmptyCalldataError extends Data.TaggedError('DecodeError')<
 class SchemaAbi extends Schema.make<Abi>(SchemaAST.objectKeyword) {}
 class AbiLoader extends Schema.TaggedRequest<AbiLoader>()('AbiLoader', {
   failure: Schema.instanceOf(MissingABIError),
-  success: SchemaAbi, // Abi
+  success: Schema.Array(Schema.Struct({ abi: SchemaAbi, id: Schema.optional(Schema.String) })),
   payload: {
     chainID: Schema.Number,
     address: Schema.String,
@@ -68,7 +69,7 @@ const getMany = (requests: Array<AbiStore.AbiParams>) =>
     }
   })
 
-const setValue = (key: AbiLoader, abi: ContractABI | null) =>
+const setValue = (key: AbiLoader, abi: (ContractABI & { strategyId: string }) | null) =>
   Effect.gen(function* () {
     const { set } = yield* AbiStore.AbiStore
     yield* set(
@@ -78,20 +79,23 @@ const setValue = (key: AbiLoader, abi: ContractABI | null) =>
         event: key.event,
         signature: key.signature,
       },
-      abi == null ? { status: 'not-found', result: null } : { status: 'success', result: abi },
+      abi == null
+        ? {
+            type: 'func' as const,
+            abi: '',
+            address: key.address,
+            chainID: key.chainID,
+            signature: key.signature || '',
+            status: 'not-found' as const,
+          }
+        : {
+            ...abi,
+            source: abi.strategyId,
+            status: 'success' as const,
+          },
     )
   })
 
-const getBestMatch = (abi: ContractABI | null) => {
-  if (abi == null) return null
-
-  if (abi.type === 'address') {
-    return JSON.parse(abi.abi) as Abi
-  }
-
-  return JSON.parse(`[${abi.abi}]`) as Abi
-}
-
 /**
  * Data loader for contracts abi
  *
@@ -136,6 +140,7 @@ const getBestMatch = (abi: ContractABI | null) => {
  * - Request pooling with back-pressure handling
  */
 
+// TODO: there is an overfetching, find why
 export const AbiLoaderRequestResolver = RequestResolver.makeBatched((requests: Array<AbiLoader>) =>
   Effect.gen(function* () {
     if (requests.length === 0) return
@@ -146,27 +151,41 @@ export const AbiLoaderRequestResolver = RequestResolver.makeBatched((requests: A
     const requestGroups = Array.groupBy(requests, makeRequestKey)
     const uniqueRequests = Object.values(requestGroups).map((group) => group[0])
 
-    const [remaining, cachedResults] = yield* pipe(
-      getMany(uniqueRequests),
-      Effect.map(
-        Array.partitionMap((resp, i) => {
-          return resp.status === 'empty'
-            ? Either.left(uniqueRequests[i])
-            : Either.right([uniqueRequests[i], resp.result] as const)
-        }),
-      ),
-      Effect.orElseSucceed(() => [uniqueRequests, []] as const),
-    )
+    const allCachedData = yield* getMany(uniqueRequests)
+
+    // Create a map of invalid sources for each request
+    const invalidSourcesMap = new Map<string, string[]>()
+    allCachedData.forEach((abis, i) => {
+      const request = uniqueRequests[i]
+      const invalid = abis
+        .filter((abi) => abi.status === 'invalid')
+        .map((abi) => abi.source)
+        .filter(Boolean) as string[]
+
+      invalidSourcesMap.set(makeRequestKey(request), invalid)
+    })
+
+    const [remaining, cachedResults] = Array.partitionMap(allCachedData, (abis, i) => {
+      // Filter out invalid/not-found ABIs and check if we have any valid ones
+      const validAbis = abis.filter((abi) => abi.status === 'success')
+      return validAbis.length === 0
+        ? Either.left(uniqueRequests[i])
+        : Either.right([uniqueRequests[i], validAbis] as const)
+    })
 
     //  Resolve ABI from the store
     yield* Effect.forEach(
       cachedResults,
-      ([request, abi]) => {
+      ([request, abis]) => {
         const group = requestGroups[makeRequestKey(request)]
-        const bestMatch = getBestMatch(abi)
-        const result = bestMatch ? Effect.succeed(bestMatch) : Effect.fail(new MissingABIError(request))
-
-        return Effect.forEach(group, (req) => Request.completeEffect(req, result), { discard: true })
+        const allMatches = abis.map((abi) => {
+          const parsedAbi = abi.type === 'address' ? (JSON.parse(abi.abi) as Abi) : (JSON.parse(`[${abi.abi}]`) as Abi)
+          return { abi: parsedAbi, id: abi.id }
+        })
+
+        return Effect.forEach(group, (req) => Request.completeEffect(req, Effect.succeed(allMatches)), {
+          discard: true,
+        })
       },
       {
         discard: true,
@@ -191,15 +210,19 @@ export const AbiLoaderRequestResolver = RequestResolver.makeBatched((requests: A
     const response = yield* Effect.forEach(
       remaining,
       (req) => {
-        const allAvailableStrategies = Array.prependAll(strategies.default, strategies[req.chainID] ?? []).filter(
-          (strategy) => strategy.type === 'address',
-        )
+        const invalidSources = invalidSourcesMap.get(makeRequestKey(req)) ?? []
+        const allAvailableStrategies = Array.prependAll(strategies.default, strategies[req.chainID] ?? [])
+          .filter((strategy) => strategy.type === 'address')
+          .filter((strategy) => !invalidSources.includes(strategy.id))
+
+        if (allAvailableStrategies.length === 0) {
+          return Effect.succeed(Either.right(req))
+        }
 
         return strategyExecutor
           .executeStrategiesSequentially(allAvailableStrategies, {
             address: req.address,
             chainId: req.chainID,
-            strategyId: 'address-batch',
           })
           .pipe(
             Effect.tapError(Effect.logDebug),
@@ -220,18 +243,22 @@ export const AbiLoaderRequestResolver = RequestResolver.makeBatched((requests: A
     // NOTE: Secondly we request strategies to fetch fragments
     const fragmentStrategyResults = yield* Effect.forEach(
       notFound,
-      ({ chainID, address, event, signature }) => {
-        const allAvailableStrategies = Array.prependAll(strategies.default, strategies[chainID] ?? []).filter(
-          (strategy) => strategy.type === 'fragment',
-        )
+      (req) => {
+        const invalidSources = invalidSourcesMap.get(makeRequestKey(req)) ?? []
+        const allAvailableStrategies = Array.prependAll(strategies.default, strategies[req.chainID] ?? [])
+          .filter((strategy) => strategy.type === 'fragment')
+          .filter((strategy) => !invalidSources.includes(strategy.id))
+
+        if (allAvailableStrategies.length === 0) {
+          return Effect.succeed(null)
+        }
 
         return strategyExecutor
           .executeStrategiesSequentially(allAvailableStrategies, {
-            address,
-            chainId: chainID,
-            event,
-            signature,
-            strategyId: 'fragment-batch',
+            address: req.address,
+            chainId: req.chainID,
+            event: req.event,
+            signature: req.signature,
           })
           .pipe(
             Effect.tapError(Effect.logDebug),
@@ -244,21 +271,50 @@ export const AbiLoaderRequestResolver = RequestResolver.makeBatched((requests: A
       },
     )
 
-    const strategyResults = Array.appendAll(addressStrategyResults, fragmentStrategyResults)
+    // Create a map to track which requests got results from address strategies
+    const addressResultsMap = new Map<string, (ContractABI & { strategyId: string })[]>()
+    addressStrategyResults.forEach((abis, i) => {
+      const request = remaining[i]
+      if (abis && abis.length > 0) {
+        addressResultsMap.set(makeRequestKey(request), abis)
+      }
+    })
+
+    // Create a map to track which requests got results from fragment strategies
+    const fragmentResultsMap = new Map<string, (ContractABI & { strategyId: string })[]>()
+    fragmentStrategyResults.forEach((abis, i) => {
+      const request = notFound[i]
+      if (abis && abis.length > 0) {
+        fragmentResultsMap.set(makeRequestKey(request), abis)
+      }
+    })
 
-    // Store results and resolve pending requests
+    // Resolve all remaining requests
     yield* Effect.forEach(
-      strategyResults,
-      (abis, i) => {
-        const request = remaining[i]
-        const abi = abis?.[0] ?? null
-        const bestMatch = getBestMatch(abi)
-        const result = bestMatch ? Effect.succeed(bestMatch) : Effect.fail(new MissingABIError(request))
+      remaining,
+      (request) => {
         const group = requestGroups[makeRequestKey(request)]
+        const addressAbis = addressResultsMap.get(makeRequestKey(request)) || []
+        const fragmentAbis = fragmentResultsMap.get(makeRequestKey(request)) || []
+        const allAbis = [...addressAbis, ...fragmentAbis]
+
+        const allMatches = allAbis.map((abi) => {
+          const parsedAbi = abi.type === 'address' ? (JSON.parse(abi.abi) as Abi) : (JSON.parse(`[${abi.abi}]`) as Abi)
+          // TODO: We should figure out how to handle the db ID here, maybe we need to start providing the ids before inserting
+          return { abi: parsedAbi, id: undefined }
+        })
+
+        const cacheEffect =
+          allAbis.length > 0
+            ? Effect.forEach(allAbis, (abi) => setValue(request, abi), {
+                discard: true,
+                concurrency: 'unbounded',
+              })
+            : Effect.void
 
         return Effect.zipRight(
-          setValue(request, abi),
-          Effect.forEach(group, (req) => Request.completeEffect(req, result), { discard: true }),
+          cacheEffect,
+          Effect.forEach(group, (req) => Request.completeEffect(req, Effect.succeed(allMatches)), { discard: true }),
         )
       },
       {
@@ -277,21 +333,31 @@ export const AbiLoaderRequestResolver = RequestResolver.makeBatched((requests: A
 // in a missing Fragment. We treat this issue as a minor one for now, as we expect it to occur rarely on contracts that
 // are not verified and with a non standard events structure.
 
+const errorAbis: Abi = [...solidityPanic, ...solidityError]
+
 export const getAndCacheAbi = (params: AbiStore.AbiParams) =>
   Effect.gen(function* () {
     if (params.event === '0x' || params.signature === '0x') {
       return yield* Effect.fail(new EmptyCalldataError(params))
     }
 
+    if (params.signature && errorFunctionSignatures.includes(params.signature)) {
+      return [{ abi: errorAbis, id: undefined }]
+    }
+
     if (params.signature && params.signature === SAFE_MULTISEND_SIGNATURE) {
-      return yield* Effect.succeed(SAFE_MULTISEND_ABI)
+      return [{ abi: SAFE_MULTISEND_ABI, id: undefined }]
     }
 
     if (params.signature && AA_ABIS[params.signature]) {
-      return yield* Effect.succeed(AA_ABIS[params.signature])
+      return [{ abi: AA_ABIS[params.signature], id: undefined }]
     }
 
-    return yield* Effect.request(new AbiLoader(params), AbiLoaderRequestResolver)
+    const abis = yield* Effect.request(new AbiLoader(params), AbiLoaderRequestResolver)
+    return Array.appendAll(abis, [
+      { abi: erc20Abi, id: undefined },
+      { abi: erc721Abi, id: undefined },
+    ])
   }).pipe(
     Effect.withSpan('AbiLoader.GetAndCacheAbi', {
       attributes: {