Skip to content

Yet another email importer that imports from raw mail text #63

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 28 commits into from
Oct 21, 2025
Merged

Yet another email importer that imports from raw mail text #63

merged 28 commits into from
Oct 21, 2025
+101 −2

Conversation

amatsuda
Copy link
Member

Here's another email importer that imports emails from raw text dump instead of the converted "S3 data".

The biggest addition on this version from the previous one is that it retrieves in-reply-to message-ids (that are dropped in the "S3 data") so we can construct email threads.

amatsuda added 28 commits October 20, 2025 19:39
@amatsuda
Initial implementation of raw mail importer
581e725
@amatsuda
Extract Message.from_mail to a method
352f5f5
@amatsuda
Mail file might not exist or might be a blank file
0917468
@amatsuda
Let's assume that all mail body are encoded in ISO-2022-JP
0ad4d14 
and let's convert all of them to UTF-8
@amatsuda
Use unparsed From value
4e1c01e 
because so far we're just showing them on HTML, not machine-processing
@amatsuda
Mail body may sometimes be broken, but let's proceed anyway
4e62345 
e.g. ruby-dev 1553, 2320, 2321, 2322, 4361, etc.
@amatsuda
Fall back to Mail's default encoding handling
a4d6066 
for some ruby-dev mails, e.g. 2320
@amatsuda
Turned out that it's hard to do this without Kconv...
1e1122c 
e.g. ruby-dev 2655
@amatsuda
Decode subject
6365a1f
@amatsuda
Retrieve parent message-id from in-reply-to, references header
5e4ea8d
@amatsuda
rails g migration add_index_messages_message_id_header
4110f59
@amatsuda
Workaround "string contains null byte" on ruby-dev: 13859
ca6f150
@amatsuda
Some more null bytes
7281ed4
@amatsuda
Properly encode from to UTF-8
7feef35 
this still warns "Encoding conversion failed code converter not found (ISO-2022-JP-2 to UTF-8)"
when fetching `from` from mail, but it seems like it's working anyway
@amatsuda
Mail#from_address can be nil
e9d9a77
@amatsuda
Work around ruby-core: 161 mojibake
5b5a144
@amatsuda
There can be emails without a subject
d5ffd4a 
e.g. ruby-core: 290
@amatsuda
Kconv does everything almost properly, indeed
59f4ee8
@amatsuda
Mail#from_address could return nil
ff38b43 
e.g. ruby-core: 161
@amatsuda
subject can be nil
6ecace8
@amatsuda
Work around "Encoding::CompatibilityError: incompatible character enc…
ff5ff6a 
…odings: UTF-8 and BINARY (ASCII-8BIT)"

on ruby-list: 37565, 38116, 43106
@amatsuda
Workaround broken subject
12e446a 
on ruby-list: 41850, 43710
@amatsuda
message_id could include a broken mojibake char
ebf6152 
e.g. ruby-talk: 10751
@amatsuda
message-id can be nil
2fed221 
e.g. ruby-dev: 93-108
@amatsuda
Perf improvement by not creating AR objects
48bee57
@amatsuda
Report errors at once
2d44e5c
@amatsuda
Perf improvement by not logging debug logs
c92551a
@amatsuda
Mail converts linebreaks to CRLF
7fdc658
@amatsuda amatsuda merged commit 8a29b98 into ruby:main Oct 21, 2025
2 checks passed
@amatsuda amatsuda deleted the import_from_raw_mail branch October 21, 2025 08:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@amatsuda