Error: Max header size for a MIME node exceeded

[alaindeurveilher]

Hello,

Thank you for the great work! But I've got an error message after processing, and all the csv files are empty.

mboxhome yarn start
yarn run v1.22.22
$ gmail-mbox-stats mymail="alain.deurveilher@gmail.com" mboxpath="./Takeout/Mail/All mail Including Spam and Trash.mbox" > stats.txt
node:internal/process/promises:394
    triggerUncaughtException(err, true /* fromPromise */);
    ^

Error: Max header size for a MIME node exceeded
    at MessageSplitter.processLine (/Users/alaind/Apps/mboxhome/node_modules/mailsplit/lib/message-splitter.js:363:31)
    at Immediate.iterateData (/Users/alaind/Apps/mboxhome/node_modules/mailsplit/lib/message-splitter.js:79:33)
    at process.processImmediate (node:internal/timers:491:21) {
  code: 'EMAXLEN'
}

Node.js v22.14.0

FY, I installed the package locally, and created a script in my package.json

{
  "name": "mboxhome",
  "scripts": {
    "start": "gmail-mbox-stats mymail=\"alain.deurveilher@gmail.com\" mboxpath=\"./Takeout/Mail/All mail Including Spam and Trash.mbox\" > stats.txt"
  },
  "dependencies": {
    "gmail-mbox-stats": "^1.2.3"
  }
}

The log ends with this:

gmail-mbox-stats v1.2.3
Created by leodevbro (Levan Katsadze)
* leodevbro@gmail.com
* linkedin.com/in/leodevbro
* github.com/leodevbro
* facebook.com/leodevbro


Started MBOX file analyzation
Start datetime: 2025-02-23_04-31-43
1000
...
78538 - here: 'to' participant address is empty - Search: ===rfc822msgid:91ffe7b1264b0137aba58d080edb1162@www.passion-photos.net=== 'prt.address === str_EMPTY'
78631 - here: 'to' participant address is empty - Search: ===rfc822msgid:3cf8141171b45ee3076b8272b96fcaf5@www.passion-photos.net=== 'prt.address === str_EMPTY'
78635 - here: 'to' participant address is empty - Search: ===rfc822msgid:951732cd7b30e0010a4a17d9cc6a32c5@forumophoto.fr=== 'prt.address === str_EMPTY'
78638 - here: 'to' participant address is empty - Search: ===rfc822msgid:09c67a81d2d30f0d9a27cbac39d5a5d2@www.passion-photos.net=== 'prt.address === str_EMPTY'
78641 - here: 'to' participant address is empty - Search: ===rfc822msgid:d23924c6489f0645b58cb102fe7c5860@forumophoto.fr=== 'prt.address === str_EMPTY'
78643 - here: 'to' participant address is empty - Search: ===rfc822msgid:943e44aad67b53312dd211ebc71220dd@forumophoto.fr=== 'prt.address === str_EMPTY'
78645 - here: 'to' participant address is empty - Search: ===rfc822msgid:7e1d247a4e064937e6bdb1a021657b76@www.passion-photos.net=== 'prt.address === str_EMPTY'
Finished accessing all the mails (79921), but not yet finished proccessing them. Please wait.
79000

So because of the exception, I don' have the rest of the log as well.
For information, it was quite fast, and too about 5 minutes to process all up to that point (Macbook Pro, M4 chip), as the file was last modified at 2025-02-23_04-36-xx

As seen, since the execution of the script expects a relative path, I passed as an argument the relative path to the mbox file, and it seemed to work well.

But as said all the csv files are empty.

Thank you.

[alaindeurveilher]

After some digging, I found out that the issue comes from simpleParser from mailsplit:
zone-eu/mailsplit#19

I don't know is it is possible to increase the maxHeadSize from your code, so I ended up doing that instead, which allowed to bypass the problematic email.s:

updated mailParsingUtils.ts:

export const parseOneMail = async (rawMail: Buffer) => {
  // CHANGES HERE: add a try catch block around simpleParser
  let parsedMail;
  try {
    parsedMail = await simpleParser(rawMail, {});
  } catch (err) {
    console.log("ERROR: simpleParser", step.succeededV, err);
  }
  return parsedMail;
};

export const processOneMail = async (rawMail: Buffer) => {
  const parsedMail = await parseOneMail(rawMail);

  // CHANGES HERE: check if defined before processing
  if (parsedMail) {
    processParsedMail(parsedMail);
  }

  step.succeededV += 1; // count of mails which are processed successfully
  print000SuccessCount();
};

After the changes, the process passes. (I think it was just about 1 email, so I could handle it manually later).

[leodevbro]

Thanks so much for the detailed solution. It's strange, even if attachment size is large, the simpleParser should not have any problem parsing it. The error says "header size", so maybe the mail header text was somehow very large? Did you have a chance to investigate that one mail?

[alaindeurveilher]

No I didn’t. And I don’t know how to identify the email in question. I didn’t start deleting the emails yet, as I had other tasks to do now.
But any advice could help if you have some, to let me know which email is problematic.
If when spotted, it appears to be a non private email (like ads, or something), I could even forward it to you’d like to reproduce.

[leodevbro]

Maybe you could try this:

export const parseOneMail = async (rawMail: Buffer) => {
  let parsedMail;
  try {
    parsedMail = await simpleParser(rawMail, {});
  } catch (err) {
    console.log("ERROR: simpleParser", step.succeededV, err);

    try {
      const investigationMail = await simpleParser(rawMail, {
        maxHtmlLengthToParse: 1 * 1024 * 1024 * 1024, // 1 GB
      });

      const mailId = investigationMail.headers.get("message-id");
      console.log("strange mail id:", mailId); // use this id to find it with the search bar of Gmail website
    } catch (err) {
      console.log("Even 1 GB maxHtmlLengthToParse failed");
    }
  }
  return parsedMail;
};

gmail mail id looks like this:
<CAACv+qDF8Et0=fNiN9C6WRyk05jvhf1=Yp98wKFFjCwJbFc=Tg@mail.gmail.com>

In order to search with Gmail search bar, left side you should add this:
rfc822msgid:

So, the result should look like this:
rfc822msgid:<CAACv+qDF8Et0=fNiN9C6WRyk05jvhf1=Yp98wKFFjCwJbFc=Tg@mail.gmail.com>

it also works if you remove the left arrow and right arrow characters:
rfc822msgid:CAACv+qDF8Et0=fNiN9C6WRyk05jvhf1=Yp98wKFFjCwJbFc=Tg@mail.gmail.com

[alaindeurveilher]

No luck with that. Even with 100 Gb it fails. Maybe we could exploit the rawMail, which is a Buffer, to do something with it?

[leodevbro]

Maybe this is the solution? or at least a possibility to diagnose?

Find this file:
node_modules\mailsplit\lib\message-splitter.js

you can do this change:

'use strict';

const Transform = require('stream').Transform;
const MimeNode = require('./mime-node');

const MAX_HEAD_SIZE = 1 * 1024 * 1024; // temporaily change this to 1 * 1024 * 1024 * 10
const MAX_CHILD_NODES = 1000;

const HEAD = 0x01;
const BODY = 0x02;
...
...
...
...

then let's change our function like this:

export const parseOneMail = async (rawMail: Buffer, trueOrderStep: number) => {
  let parsedMail;
  try {
    parsedMail = await simpleParser(rawMail, {});
  } catch (err) {
    console.log("ERROR: simpleParser, trueOrderStep:", trueOrderStep, err);
  }

  // this will be useful when we will determine the trueOrderStep on which mail parsing fails.
  if (
    parsedMail &&
    trueOrderStep === 0 /* default 0, but change it to the correct number */
  ) {
    const mailId = parsedMail.headers.get("message-id");
    console.log("strange mail id:", mailId);
  }

  return parsedMail;
};

also this function to pass the trueOrderStep:

export const processOneMail = async (
  rawMail: Buffer,
  trueOrderStep: number,
) => {
  const parsedMail = await parseOneMail(rawMail, trueOrderStep);

  if (parsedMail) {
    processParsedMail(parsedMail);
  }
};

also this method which is the source of the trueOrderStep:

mbox.on("data", async (msg) => {
    step.v += 1;

    try {
      processOneMail(msg, step.v); // we just added step.v, nothing more.
    } catch (err) {
      console.log(err);
    }

    step.succeededV += 1;
    print000SuccessCount();
  });

It's important to use trueOrderStep instead of step.succeededV, because succeededV is generated via async operations and it may vary, it will not be always correct order. trueOrderStep will be always correct order, so we will be able to determine the exact step number of mail which is failing.

So, before you change the MAX_HEAD_SIZE , first run my new code to determine the correct step number, and then change the MAX_HEAD_SIZE and also change my code trueOrderStep === 0 zero to correct step number, and run the code again, and I guess it will correctly print the id of the failing mail.

[alaindeurveilher]

I modified in node_modules, the message-splitter.js file to add some info:

switch (this.state) {
  case HEAD: {
    this.node.addHeaderChunk(line);
    if (this.node._headerlen > this.maxHeadSize) {
      let err = new Error('Max header size for a MIME node exceeded: ' + this.node._headerlen + ' > ' + this.maxHeadSize);
      err.code = 'EMAXLEN';
      return next(err);
    }

ERROR: simpleParser 79227 Error: Max header size for a MIME node exceeded: 1048586 > 1048576

I think it's a personal email, in which there is a png attached, but I don't see anything particularly wrong with it.

But, OK. I think I will not spend more time on this, since the problem is in the 3rd party libs in the middle, and I found a way to get my stats already :-)

And I also tried with increasing the value of maxHeadSize (MAX_HEAD_SIZE), and the email is well parsed.

[leodevbro]

1048586 > 1048576

Oh, so it was just 10 bytes more than the limit. Thanks for the investigation, I'll try to update gmail-mbox-stats to easily handle such scenarios.

[alaindeurveilher]

I didn't see your suggestion in the meantime> Maybe I'll try it with the trueOrderStep...

[alaindeurveilher]

Ha! found the email. It's an email I sent, with a video attached, a .mp4 file of 2.8 MB, but without body. I did not write anything in the body at all, I don't know if this is the side effect. Cannot share though, since it's private.

Thank you for your time, it was a fun investigation :-)

[leodevbro]

Oh, cool. I guess the mail is very old, like many years ago when Gmail platform generated mails with a bit different structure and different limits. A header of a mail is not just the body or subject, but many things like "from", "to", timestamps, metadata about everything, metadata about attachments and so on...

[alaindeurveilher]

February 2009.