Strange Behavior (No Body or No Metadata)

[FatihKoz]

Hi, updated my package to latest version 1.17.2 and started my tests, what I am observing can be a bug or design choice but it is below in detail.

I am using the laravel package mainly but also tested this without it too.

When I fetch withBody(), withHeaders(), withFlags(), I can see the attachements (if any) and mail bodies BUT I loose all metadata fields (like date, from, to, uid, subject etc.)

// use DirectoryTree\ImapEngine\Mailbox;
use DirectoryTree\ImapEngine\Laravel\Facades\Imap;

$messages = $inbox->messages()->withFlags()->withBody()->withHeaders()->since($date)->get();

When I do a basic fetch, I can get the metadata but not attachements and bodies.

// use DirectoryTree\ImapEngine\Mailbox;
use DirectoryTree\ImapEngine\Laravel\Facades\Imap;

$messages = $inbox->messages()->since($date)->get();

I am trying to follow available docs but can not find a cause for this, also checking the code via GitHub to get a better view of the relationships and methods...

Do you need any logs for this to debug, if so I can provide IMAP debug logs or laravel error logs without sensitive information.

Thanks for your time and support :)

[FatihKoz]

Below you can see the results of different queries mentioned in the first post.

use DirectoryTree\ImapEngine\Laravel\Facades\Imap;

$messages = $inbox->messages()->withHeaders()->withBody()->withFlags()->since($date)->get();

---

use DirectoryTree\ImapEngine\Laravel\Facades\Imap;

$messages = $inbox->messages()->withHeaders()->withFlags()->since($date)->get();

As you see the difference, adding withBody() simply disables the metadata completely, while providing access to body and attachements.

Removing withBody() disables access to body and attachments but provides all header and metadata sections of the mail itself.

This is the view running with foreach($messages as $email)

@if(filled($email))
  <tr>
    <td>{{ $email->messageId() ?? 'N/A' }}</td>
    <td>{{ optional($email->date())->format('Y.m.d H:i:s') ?? 'N/A' }}</td>
    <td>{{ optional($email->from())->email() ?? 'N/A' }}</td>
    <td>{{ $email->subject() ?? 'N/A' }}</td>
    <td>{{ implode(', ', $email->flags()) ?? 'N/A' }}</td>
    @if(filled($email->html()))
      <td>{!! $email->html() !!}</td>
    @else
      <td>{{ $email->text() ?? 'N/A' }}</td>
    @endif
    <td>{{ count($email->attachments()) ?? 'N/A' }}</td>
  </tr>
@endif

Had to add some failsafe code to let it run on each query without changes.

This is the beginning of the log without ->withBody() usage, did not attached fetched items to save space.

2025-09-29 13:24:59: << * OK The Microsoft Exchange IMAP4 service is ready. 
2025-09-29 13:24:59: >> TAG1 LOGIN "xxx" "xxx" 
2025-09-29 13:24:59: << TAG1 OK LOGIN completed. 
2025-09-29 13:24:59: >> TAG2 LIST "" "INBOX" 
2025-09-29 13:24:59: << * LIST (\Marked \HasNoChildren) "/" INBOX 
2025-09-29 13:24:59: << TAG2 OK LIST completed. 
2025-09-29 13:24:59: >> TAG3 SELECT "INBOX" 
2025-09-29 13:25:00: << * 6 EXISTS 
2025-09-29 13:25:00: << * 0 RECENT 
2025-09-29 13:25:00: << * FLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent) 
2025-09-29 13:25:00: << * OK [PERMANENTFLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent)] Permanent flags 
2025-09-29 13:25:00: << * OK [UNSEEN 1] Is the first unseen message 
2025-09-29 13:25:00: << * OK [UIDVALIDITY 14] UIDVALIDITY value 
2025-09-29 13:25:00: << * OK [UIDNEXT 10] The next unique identifier value 
2025-09-29 13:25:00: << TAG3 OK [READ-WRITE] SELECT completed. 
2025-09-29 13:25:00: >> TAG4 UID SEARCH SINCE 22-Sep-2025 
2025-09-29 13:25:00: << * SEARCH 2 5 6 7 8 9 
2025-09-29 13:25:00: << TAG4 OK SEARCH completed. 
2025-09-29 13:25:00: >> TAG5 UID FETCH 9,8,7,6,5,2 (FLAGS BODY.PEEK[HEADER]) 

And the log of ->withBody() used query.

2025-09-29 13:41:30: << * OK The Microsoft Exchange IMAP4 service is ready. 
2025-09-29 13:41:30: >> TAG1 LOGIN "xxx" "xxx" 
2025-09-29 13:41:30: << TAG1 OK LOGIN completed. 
2025-09-29 13:41:30: >> TAG2 LIST "" "INBOX" 
2025-09-29 13:41:30: << * LIST (\Marked \HasNoChildren) "/" INBOX 
2025-09-29 13:41:30: << TAG2 OK LIST completed. 
2025-09-29 13:41:30: >> TAG3 SELECT "INBOX" 
2025-09-29 13:41:30: << * 6 EXISTS 
2025-09-29 13:41:30: << * 0 RECENT 
2025-09-29 13:41:30: << * FLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent) 
2025-09-29 13:41:30: << * OK [PERMANENTFLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent)] Permanent flags 
2025-09-29 13:41:30: << * OK [UNSEEN 1] Is the first unseen message 
2025-09-29 13:41:30: << * OK [UIDVALIDITY 14] UIDVALIDITY value 
2025-09-29 13:41:30: << * OK [UIDNEXT 10] The next unique identifier value 
2025-09-29 13:41:30: << TAG3 OK [READ-WRITE] SELECT completed. 
2025-09-29 13:41:30: >> TAG4 UID SEARCH SINCE 22-Sep-2025 
2025-09-29 13:41:30: << * SEARCH 2 5 6 7 8 9 
2025-09-29 13:41:30: << TAG4 OK SEARCH completed. 
2025-09-29 13:41:30: >> TAG5 UID FETCH 9,8,7,6,5,2 (FLAGS BODY.PEEK[TEXT] BODY.PEEK[HEADER])

---

Additionally not adding any modifiers like withHeaders() or withBody() results an empty table. Even though 6 items are returned they are empty, no data provided as like nothing is fetched.

$messages = $inbox->messages()->since($date)->get();

Log of latest query

2025-09-29 13:46:56: << * OK The Microsoft Exchange IMAP4 service is ready. 
2025-09-29 13:46:56: >> TAG1 LOGIN "xxx" "xxx" 
2025-09-29 13:46:56: << TAG1 OK LOGIN completed. 2025-09-29 13:46:56: >> TAG2 LIST "" "INBOX" 
2025-09-29 13:46:56: << * LIST (\Marked \HasNoChildren) "/" INBOX 
2025-09-29 13:46:56: << TAG2 OK LIST completed. 
2025-09-29 13:46:56: >> TAG3 SELECT "INBOX" 
2025-09-29 13:46:56: << * 6 EXISTS 
2025-09-29 13:46:56: << * 0 RECENT 
2025-09-29 13:46:56: << * FLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent) 
2025-09-29 13:46:56: << * OK [PERMANENTFLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent)] Permanent flags 
2025-09-29 13:46:56: << * OK [UNSEEN 1] Is the first unseen message 
2025-09-29 13:46:56: << * OK [UIDVALIDITY 14] UIDVALIDITY value 
2025-09-29 13:46:56: << * OK [UIDNEXT 10] The next unique identifier value 
2025-09-29 13:46:56: << TAG3 OK [READ-WRITE] SELECT completed. 
2025-09-29 13:46:56: >> TAG4 UID SEARCH SINCE 22-Sep-2025 
2025-09-29 13:46:56: << * SEARCH 2 5 6 7 8 9 
2025-09-29 13:46:56: << TAG4 OK SEARCH completed.

Any ideas would be great :)

[stevebauman]

Thanks for the detailed issue @FatihKoz!

I'm not actually able to reproduce this. Can you dump the messages you get back in PHP and post them here (with their sensitive details omitted)? You can do so by calling $message->toArray() on each message instance which will show us what parts of the messages are populated:

[
    "uid" => 261255,
    "flags" => ["\Seen"],
    "head" => "...",
    "body" => "...",
]

[stevebauman]

Oh forgot to mention, I'm connecting to my own Microsoft Outlook mailbox as well so I don't think this is a provider issue.

[FatihKoz]

Sure, I can dump and share details when I go back to office (like in 14-15 hours or slightly less)... What I tried before leaving office was kind of strange but worked as expected :) To get all details, I had to hit the server several times in the loop to do a search with $email->uid().

I can do something similar in the controller with only two hits and build up a laravel collection with all data packed together nicely but I do not think this should be the way to go...

• Controller

$mailbox = Imap::mailbox('test');
$inbox = $mailbox->inbox();
$date = Carbon::now()->subDays(7);
$messages = $inbox->messages()->since($date)->withHeaders()->get();

Then passing the $messages to view, which ends up having only the metadata (no body, no attachments)

• View

foreach($messages as $email) {
  $em = Imap::mailbox('test')->inbox()->messages()->withBody()->find($email->uid(), ImapFetchIdentifier::Uid);

  $email->uid();
  $email->subject();
  $email->date()->format('Y.m.d H:i:s');
  $email->from()->email();
  $em->html(); // or $em->text()
  $em->attachments();
}

[FatihKoz]

I think I have a clue, at least I saw the difference when dumping the results.

$messages = $inbox->messages()->since($date)->withHeaders()->get();
>> TAG5 UID FETCH 9,8,7,6,5,2 (FLAGS BODY.PEEK[HEADER])

When using ->withHeaders() alone all metadata is in the head but we have no body as expected.

---

When I change my request to utilize ->withBody() head info changes completely and all metadata is gone.

$messages = $inbox->messages()->since($date)->withHeaders()->withBody()->get();
>> TAG5 UID FETCH 9,8,7,6,5,2 (FLAGS BODY.PEEK[TEXT] BODY.PEEK[HEADER])

---

Using only ->withBody() returns the body as expected with no head info and again all metadata is lost.

$messages_body = $inbox->messages()->since($date)->withBody()->get();

[FatihKoz]

And these are the dumped arrays as requested (for both requests, showing the same message, uid 2)

// $inbox->messages()->since($date)->withHeaders()->get();
[ 
  "uid" => 2
  "flags" => []
  "head" => """
    MIME-Version: 1.0
    Received: from xxx by
     xxx with Microsoft SMTP Server
     xxx id
     xxx via Mailbox Transport; Fri, 26 Sep 2025 14:09:52 +0300
    Received: xxx by
     xxx with Microsoft SMTP Server
     xxx id
     xxx Fri, 26 Sep 2025 14:09:52 +0300
    Received: from xxx by
     xxx with mapi id
     xxx Fri, 26 Sep 2025 14:09:52 +0300
    From: xxx
    To: xxx
    Subject: RE: Test Mail
    Thread-Topic: Test Mail
    Thread-Index: AQHcLtF1oi0XGZ/MSUqMH+UxzSlEMbSlRziA
    Date: Fri, 26 Sep 2025 14:09:52 +0300
    Message-ID: <b4f63b9550ad42dbb48889d0a9d42865@xxx>
    References: <c94975b23afdb28c97534ad1a1a79b2b@xxx>
    In-Reply-To: <c94975b23afdb28c97534ad1a1a79b2b@xxx>
    Accept-Language: en-US
    Content-Language: tr-TR
    X-MS-Exchange-Organization-AuthAs: Internal
    X-MS-Exchange-Organization-AuthMechanism: 04
    X-MS-Exchange-Organization-AuthSource: xxx
    X-MS-Has-Attach:
    X-MS-Exchange-Organization-Network-Message-Id:
    \ta16ff770-f679-4251-841b-08ddfced3757
    X-MS-TNEF-Correlator:
    X-MS-Exchange-Organization-RecordReviewCfmType: 0
    Content-Type: text/html; charset="utf-8"
    Content-Transfer-Encoding: base64
    
    """
  "body" => ""
]

And the one with body (reduced the contents of body to save space)

// $inbox->messages()->since($date)->withBody()->withHeaders()->withFlags()->get();
[
  "uid" => 2
  "flags" => []
  "head" => """
    Content-Type: text/html; charset="utf-8"
    Content-Transfer-Encoding: base64
    
    """
  "body" => """
    bjpzY2hlbWFzLW1pY3Jvc29mdC1jb206b2ZmaWNlOm9mZmljZSIgeG1sbnM6dz0idXJuOnNjaGVt
    YXMtbWljcm9zb2Z0LWNvbTpvZmZpY2U6d29yZCIgeG1sbnM6bT0iaHR0cDovL3NjaGVtYXMubWlj
    """
]

[stevebauman]

Thanks for posting these @FatihKoz. It seems that those last two queries are returning the correct values -- correct me if I'm wrong?

[FatihKoz]

Which correct values ? Sorry but no, when you request headers only you get the metadata (because body and attachements are not there, this can be the expected result), when you request body (either alone or with headers) you loose all metadata (because returned header changes and does not contain basic info like subject, sender, date etc.)

To be honest, I would expect using withHeaders() and withBody() together in the query should return both without removing any data.

[FatihKoz]

And yes, when you do two queries you get correct data.

My understanding from the results is simple;

Using withBody() breaks/replaces the header completely.

Thus I am using two querries (in controller) and then combining them to get a proper view including subject, sender, date/time, body and attacments.

[stevebauman]

Maybe I'm blind but im still not seeing the issue 😅

The first query is only loading headers:

$inbox->messages()->since($date)->withHeaders()->get()

And only the head is populated (correct).

The second query is loading body and headers:

$inbox->messages()->since($date)->withBody()->withHeaders()->withFlags()->get()

Which is correctly returning both body and head of the message.

What do you mean by "metadata"? The head?

[FatihKoz]

Would you please compare the "head" part of the arrays :) they are the same message.

Or kindly try to find the subject in the "head" of second array / second query, ->withBody()->withHeaders()

Metadata; Subject, From, To, Cc, Date/time (as described in your documents)

[FatihKoz]

Second query is NOT returning the header, how can I explain this with different words I have no idea ;)

The dumped arrays even show this clearly :(

[FatihKoz]

Metadata from the docs :) Second query does not have it, using withBody() removes the "head" (header)

Hope it is more clear and understandable now :)

[stevebauman]

Ohhh I see I see, thanks! My mistake -- I thought the two shared snippets were just omitting some of the contents and not actually the full representations of the contents.

I'm unfortunately still not able to reproduce this though. It's almost as if there's some kind of parsing issue with the IMAP response, but I don't see that full response here in the logs you've posted. To debug further I'll need to inspect the entire IMAP response of the message and see if it's being parsed incorrectly. This will of course contain potentially sensitive information so you'd have to anonymize it (but maintain the structure so it's an effective test).

If you want to post that so I can dig in further please do -- otherwise if you'd like to test this yourself without having to anonymize the message data you could clone the repo and add a test case with your logs like this and see if it is a parsing issue:

ImapEngine/tests/Unit/Connection/ImapParserTest.php

Lines 98 to 117 in d74a197

	test('parses a several lines', function () {
	$stream = new FakeStream;
	$stream->open();
	$stream->feed([
	'* 2 FETCH (UID 102)',
	'TAG1 OK FETCH completed',
	'* 3 FETCH (UID 101 RFC822.HEADER {28}',
	'Subject: Foo',
	'Subject: Foo',
	')',
	'* 4 FETCH (UID 102 RFC822.HEADER {14}',
	'Subject: Bar',
	')',
	'TAG2 OK FETCH completed',
	]);

[jbostoen]

I seem to have a similar issue. Indeed, the body is filled; but $oMessage->head() only returns

Content-Type: text/html; charset="utf-8"
Content-ID: Content-Transfer-Encoding: base64

When called like this:

			/** @var Message $oMessage */
			$oMessage = $this->GetFolder()
				->messages()
				->withHeaders()
				->withBody()
				->findOrFail($iOffsetIndex, ImapFetchIdentifier::MessageNumber);

When commenting out the withBody() line, the header info is filled correctly.

Version info:

carbonphp/carbon-doctrine-types  3.2.0   Types to use Carbon in Doctrine
directorytree/imapengine         1.18.0  A fully-featured IMAP library -- without the PHP extension
doctrine/lexer                   3.0.1   PHP Doctrine Lexer parser library that can be used in Top-Down, Recursive Descent Parsers.
egulias/email-validator          4.0.4   A library for validating emails against several RFCs
guzzlehttp/psr7                  2.8.0   PSR-7 message implementation that also provides common utility methods
illuminate/collections           12.33.0 The Illuminate Collections package.
illuminate/conditionable         12.33.0 The Illuminate Conditionable package.
illuminate/contracts             12.33.0 The Illuminate Contracts package.
illuminate/macroable             12.33.0 The Illuminate Macroable package.
laravel/serializable-closure     2.0.5   Laravel Serializable Closure provides an easy and secure way to serialize closures in PHP.
nesbot/carbon                    3.10.3  An API extension for DateTime that supports 281 different languages.
php-di/invoker                   2.3.7   Generic and extensible callable invoker
php-di/php-di                    7.1.1   The dependency injection container for humans
psr/clock                        1.0.0   Common interface for reading the clock.
psr/container                    2.0.2   Common Container Interface (PHP FIG PSR-11)
psr/http-factory                 1.1.0   PSR-17: Common interfaces for PSR-7 HTTP message factories
psr/http-message                 2.0     Common interface for HTTP messages
psr/log                          3.0.2   Common interface for logging libraries
psr/simple-cache                 3.0.0   Common interfaces for simple caching
ralouphie/getallheaders          3.0.3   A polyfill for getallheaders.
symfony/clock                    7.3.0   Decouples applications from the system clock
symfony/deprecation-contracts    3.6.0   A generic function and convention to trigger deprecation notices
symfony/mime                     7.3.4   Allows manipulating MIME messages
symfony/polyfill-iconv           1.33.0  Symfony polyfill for the Iconv extension
symfony/polyfill-intl-idn        1.33.0  Symfony polyfill for intl's idn_to_ascii and idn_to_utf8 functions
symfony/polyfill-intl-normalizer 1.33.0  Symfony polyfill for intl's Normalizer class and related functions
symfony/polyfill-mbstring        1.33.0  Symfony polyfill for the Mbstring extension
symfony/polyfill-php83           1.33.0  Symfony polyfill backporting some PHP 8.3+ features to lower PHP versions
symfony/polyfill-php84           1.33.0  Symfony polyfill backporting some PHP 8.4+ features to lower PHP versions
symfony/polyfill-php85           1.33.0  Symfony polyfill backporting some PHP 8.5+ features to lower PHP versions
symfony/translation              7.3.4   Provides tools to internationalize your application
symfony/translation-contracts    3.6.0   Generic abstractions related to translation
zbateson/mail-mime-parser        3.0.4   MIME email message parser
zbateson/mb-wrapper              2.0.1   Wrapper for mbstring with fallback to iconv for encoding conversion and string manipulation
zbateson/stream-decorators       2.1.1   PHP psr7 stream decorators for mime message part streams

Other info: Microsoft Exchange Online (Office 365) mailbox