Are the .cache directories meant to be kept in version control?

[PaulRBerg]

Question for @razgraf @gavriliumircea

These .cache directories appear to be auto-generated

e.g. https://github.com/sablier-labs/indexers/tree/main/apps/flow-envio/.cache

[razgraf]

Yes, they are. That is where we keep the token-details as they get queried. They're supposed to be used between deployments, so the cache has to be persisted (so we don't query for the same token again next time we deploy the indexer).

[PaulRBerg]

Ok, thanks for confirming.

I recommend renaming the directory to something else because cache directories (especially since it also contains a dot in its name) are typically git ignored and may be reused by other dev tools. For instance, Foundry caches stuff in the ./cache directory.

[PaulRBerg]

I'm reopening this discussion because I want to understand what exactly is being optimized for here.

Is the goal to reduce RPC calls? Is that why we're keeping this folder in version control?

Because if yes, the development overhead of:

• Maintaining the cache logic
• Explaining to each other what this directory is

Does not seem worth it to me, and I thus suggest getting rid of it (entirely, not just from version control). We're not making indexer deployments that often, and when we do it, we should be able to pull all token details from the RPC URLs.

Am I missing something? Besides RPC calls, what does this cache help with, @razgraf?

Edit: I looked at the token-1.json file for Lockup, and I think I now see why this is sort of useful — the cache stores a copy of all ERC-20 tokens streamed on Sablier. However, I'm still skeptical that the development effort is worth the benefits. Do the RPCs freeze or go down when we pull so many tokens' information?

Edit 2: does the fetch occur when running envio codegen or only when deploying?

[razgraf]

Outside of data we can extract from events, we need access to some other onchain values. An example would be the name and symbol of the ERC-20 token.

To get them, we need to call the contracts and ask. While The Graph has a baked in mechanism to query data mid-handler, Envio doesn't. To cover this need, we implemented a system that queries through RPCs on demand (we came up with this solution together with their team).

→ → →

Now, whenever a deployment happens, events will start getting indexed and hundreds (if not more) of tokens will need their details figured out. With simple on-demand queries this caused:

• rate limiting (since lots of requests were sent at once, we're talking millisecond-level spamming)
• delays in indexing and possible out-of-order data (since event B may rely on event A, and event A is still stuck resolving the RPC request from earlier, although this issue may have been a mixed bag and was resolved in the meantime)
• blocking the RAM of the indexer host which caused it to stall or crash

You can check the joint Telegram chat we have for more context.

→ → →

To address this, a caching mechanism was developed to carry results between deployments and skip heavy historical computation which was unnecessary since it has already been done (and no structural mutations took place in the meantime).

While testing, we're using our own more powerful personal machines to build-up this list, which means by the time it gets deployed to the hosted service, we can skip thousands of requests.

---

Is the goal to reduce RPC calls

Basically yes, but the amount of RPC calls needed caused other problems as explained above.

the development overhead

There isn't really much overhead, we implemented it once, a year ago and haven't touched it since. And re. explaining, this discussion will be really helpful.

We're not making indexer deployments that often

We're doing them not that often (as in once or twice a month, depending on the sprints) but on occasions where we're developing new features into them, we can end up deploying 10-20 times until we finalize the sprint. So not really a problem we can leave behind.

Edit 2: does the fetch occur when running envio codegen or only when deploying?

No, fetching only happens when the indexer is started. The pnpm run dev command prepares everything and kick-starts the indexer. Codegen simply generates the typings bound to the schema, just like in the subgraphs.

[PaulRBerg]

Thanks very much for taking the time to explain this. I see now why we should keep the RPC cache. I will just rename it to something else in #92, and DRY-ify it.

The only point I still disagree with is the development overhead. The fact that the code was not changed for a year doesn't mean that there will not be changes from now on OR there will not be a learning curve when a new team member (in this case, me), picks up the code base.

And to clarify, while I consider it overhead, it is worth it per the reason you provided.

[PaulRBerg]

One more question, @razgraf

Does the implementation handle scenarios whereby tokens' metadata change over time? e.g. the token symbol/name changes (because of upgradeability).

[razgraf]

Your observation is correct, it doesn't handle such cases, nor does the implementation in The Graph. This mutation is technically possible but in reality I have never actually seen it happen - so it felt like over-engineering to do it (it would've involved watching every ERC20 contract for updates).

[PaulRBerg]

Yeah we can just nevermind it. I'll simply add an explanatory comment about it.

Admin-controlled tokens like that can take the mumu for now

[PaulRBerg]

Aaand one more :D

I guess this RPC overloading issue is why we need to define 5+ RPCs per chain?

indexers/packages/constants/src/bundles/definitions.ts

Lines 27 to 33 in f23d7f3

	[
	"https://rpc.ankr.com/eth",
	"https://eth.llamarpc.com",
	"https://ethereum.publicnode.com",
	"https://eth-mainnet.public.blastapi.io",
	"https://rpc.tornadoeth.cash/eth",
	],

[razgraf]

Yes ser! Viem has a retry mechanism for RPC queries, but sometimes it doesn't define a lot of RPC fallbacks for every chain, so we have to add some ourselves to be safe(er).

[PaulRBerg]

Got it. It would be nice if chainlist.org provided an npm package for this info ..

[PaulRBerg]

Found this: https://github.com/actuallymentor/chainlist-rpcs

[PaulRBerg]

Another question @razgraf

It looks like the new context.effect API might be an alternative solution.

Based on my reading of the docs, the way this works is that they 'memoize' the web2 request so that if N events all make the same query for obtaining, say, the ERC-20 metadata for USDC on Arbitrum, that request will be executed only once.

More research needed to confirm this but it looks promising!

[DZakh]

Could you share DataEntry you have problems with?

[PaulRBerg]

Thanks so much for the tips!!

Toggle to see `DataEntry`

import * as path from "node:path";
import { sablier } from "@sablier/deployments";
import * as fs from "fs-extra";
import _ from "lodash";
import type { Envio } from "./bindings";
import type { ERC20Metadata } from "./types";

export enum DataCategory {
  Proxender = "proxenders",
  ERC20 = "erc20",
}

type ProxenderInfo = {
  owner: Envio.Address;
};

type ShapeMap = {
  [DataCategory.ERC20]: Record<Envio.Address, ERC20Metadata>;
  [DataCategory.Proxender]: Record<Envio.Address, ProxenderInfo>;
};

export function initDataEntry<C extends DataCategory>(category: C, chainId: number): DataEntry<C> {
  return new DataEntry(category, chainId);
}

export class DataEntry<C extends DataCategory> {
  public readonly file: string;

  private static readonly BASE_DIR = path.join(__dirname, "rpc-data");
  private static readonly ENCODING = "utf8" as const;

  private data: ShapeMap[C] = {};

  constructor(
    public readonly category: C,
    public readonly chainId: number,
  ) {
    const chain = sablier.chains.getOrThrow(chainId);
    this.file = path.join(DataEntry.BASE_DIR, category, `${chain.slug}.json`);
    this.preflight();
    this.load();
  }

  private load() {
    try {
      const raw = fs.readFileSync(this.file, DataEntry.ENCODING);
      this.data = JSON.parse(raw);
    } catch (err) {
      console.error(`Failed reading data from ${this.file}`, err);
      this.data = {};
    }
  }

  private preflight() {
    const fileExists = fs.existsSync(this.file);
    if (!fileExists) {
      fs.writeFileSync(this.file, "{}", DataEntry.ENCODING);
    }
  }

  public read(key: keyof ShapeMap[C]) {
    return this.data[key];
  }

  public save(newData: Partial<ShapeMap[C]>) {
    this.data = _.merge({}, this.data, newData);
    try {
      fs.writeFileSync(this.file, JSON.stringify(this.data), DataEntry.ENCODING);
    } catch (err) {
      console.error(`Failed writing data to ${this.file}`, err);
    }
  }
}

[DZakh]

Forget the second effect for now. We can revisit it later if the external calls will be a bottleneck. There's a bigger problem with the cache logic that it uses syncronous JSON file writes and reads on every effect call anyways.

[PaulRBerg]

Alright, thanks!

I think data caching is an area where Envio could offer something out-of-the-box.

I know it's easier said than done, but I imagine it's a pain point for many users.

[DZakh]

This is true 👍