Skip to content

Commit cc19514

Browse files
nevansnevans
committed
🩹 Workaround invalid Gmail FLAGS response
Gmail allows (or allowed) users to create flags containing some invalid `atom-specials` chars, such as `"]"` or `SP`. Ultimately, this isn't unambiguously parsable. But we can workaround it. Prior to #212, the ResponseParser simply grabbed everything inside the parentheses and then scanned through with a very liberal "flag" regexp. This commit attempts to parse it strictly at first and then falls back to the "quirks mode" parser if that fails. Fixes #241. This also uses `delete_prefix!` to avoid another string allocation when converting flag strings into symbols. ----- Benchmark results: ``` Calculating ------------------------------------- v0.4.7-6-g3f88747 0.2.3 0.3.7 0.4.2 0.4.7 fetch_msg_att_flags_and_uid 53.343k 33.978k 33.562k 34.063k 52.615k i/s - 162.079k times in 3.038415s 4.770048s 4.829303s 4.758267s 3.080481s flags_with_various_flag_types 95.497k 73.549k 73.802k 73.777k 93.892k i/s - 289.950k times in 3.036226s 3.942294s 3.928770s 3.930078s 3.088128s imap.gmail.com allows invalid atom-specials in flags 32.155k 41.518k 38.704k 37.842k ERROR i/s - 100.595k times in 3.128400s 2.422950s 2.599085s 2.658261s 0.000000s list_with_various_flag_capitalizations 42.105k 48.496k 45.108k 31.679k 39.903k i/s - 118.998k times in 2.826218s 2.453753s 2.638075s 3.756420s 2.982145s resp_code_PERMANENTFLAGS_rfc3501_6.3.1_example 56.608k 44.082k 44.029k 46.168k 56.743k i/s - 184.142k times in 3.252913s 4.177284s 4.182247s 3.988496s 3.245177s resp_code_PERMANENTFLAGS_rfc3501_6.3.2_example 62.031k 47.918k 49.941k 56.477k 63.410k i/s - 197.518k times in 3.184188s 4.122017s 3.955044s 3.497315s 3.114928s resp_text_PERMANENTFLAGS_with_various_flag_types 55.224k 41.582k 41.547k 46.085k 54.984k i/s - 178.322k times in 3.229081s 4.288422s 4.292034s 3.869394s 3.243159s rfc3501_7.2.6_FLAGS_response_example 79.586k 61.576k 62.134k 62.640k 79.194k i/s - 268.613k times in 3.375129s 4.362270s 4.323131s 4.288183s 3.391847s Comparison: fetch_msg_att_flags_and_uid v0.4.7-6-g3f88747: 53343.3 i/s 0.4.7: 52614.8 i/s - 1.01x slower 0.4.2: 34062.6 i/s - 1.57x slower 0.2.3: 33978.5 i/s - 1.57x slower 0.3.7: 33561.6 i/s - 1.59x slower flags_with_various_flag_types v0.4.7-6-g3f88747: 95496.8 i/s 0.4.7: 93891.8 i/s - 1.02x slower 0.3.7: 73801.7 i/s - 1.29x slower 0.4.2: 73777.2 i/s - 1.29x slower 0.2.3: 73548.5 i/s - 1.30x slower imap.gmail.com allows invalid atom-specials in flags 0.2.3: 41517.6 i/s 0.3.7: 38704.0 i/s - 1.07x slower 0.4.2: 37842.4 i/s - 1.10x slower v0.4.7-6-g3f88747: 32155.4 i/s - 1.29x slower 0.4.7: 0.0 i/s list_with_various_flag_capitalizations 0.2.3: 48496.3 i/s 0.3.7: 45107.9 i/s - 1.08x slower v0.4.7-6-gd7bf81d-dirty: 42105.0 i/s - 1.15x slower 0.4.7: 39903.5 i/s - 1.22x slower 0.4.2: 31678.6 i/s - 1.53x slower resp_code_PERMANENTFLAGS_rfc3501_6.3.1_example 0.4.7: 56743.3 i/s v0.4.7-6-g3f88747: 56608.3 i/s - 1.00x slower 0.4.2: 46168.3 i/s - 1.23x slower 0.2.3: 44081.8 i/s - 1.29x slower 0.3.7: 44029.4 i/s - 1.29x slower resp_code_PERMANENTFLAGS_rfc3501_6.3.2_example 0.4.7: 63410.1 i/s v0.4.7-6-g3f88747: 62030.9 i/s - 1.02x slower 0.4.2: 56477.0 i/s - 1.12x slower 0.3.7: 49940.8 i/s - 1.27x slower 0.2.3: 47917.8 i/s - 1.32x slower resp_text_PERMANENTFLAGS_with_various_flag_types v0.4.7-6-g3f88747: 55223.8 i/s 0.4.7: 54984.0 i/s - 1.00x slower 0.4.2: 46085.3 i/s - 1.20x slower 0.2.3: 41582.2 i/s - 1.33x slower 0.3.7: 41547.2 i/s - 1.33x slower rfc3501_7.2.6_FLAGS_response_example v0.4.7-6-g3f88747: 79586.0 i/s 0.4.7: 79193.7 i/s - 1.00x slower 0.4.2: 62640.3 i/s - 1.27x slower 0.3.7: 62133.9 i/s - 1.28x slower 0.2.3: 61576.4 i/s - 1.29x slower ```
1 parent bb6ced5 commit cc19514

File tree

2 files changed

+60
-7
lines changed

2 files changed

+60
-7
lines changed

‎lib/net/imap/response_parser.rb

Lines changed: 24 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -231,6 +231,10 @@ module RFC3629
231231
FLAG_PERM_LIST = /\G\((#{FLAG_PERM}(?:#{SP}#{FLAG_PERM})*|)\)/ni
232232
MBX_LIST_FLAGS = /\G (#{MBX_FLAG }(?:#{SP}#{MBX_FLAG })*) /nix
233233

234+
# Gmail allows SP and "]" in flags.......
235+
QUIRKY_FLAG = Regexp.union(/\\?#{ASTRING_CHARS}/n, "\\*")
236+
QUIRKY_FLAGS_LIST = /\G\(( [^)]* )\)/nx
237+
234238
# RFC3501:
235239
# QUOTED-CHAR = <any TEXT-CHAR except quoted-specials> /
236240
# "\" quoted-specials
@@ -1901,22 +1905,35 @@ def address
19011905

19021906
# flag-list = "(" [flag *(SP flag)] ")"
19031907
def flag_list
1904-
match_re(Patterns::FLAG_LIST, "flag-list")[1]
1905-
.split(nil)
1906-
.map! { _1.start_with?("\\") ? _1[1..].capitalize.to_sym : _1 }
1908+
if (match = accept_re(Patterns::FLAG_LIST))
1909+
match[1].split(nil)
1910+
.map! { _1.delete_prefix!("\\") ? _1.capitalize.to_sym : _1 }
1911+
else
1912+
quirky__flag_list "flags-list"
1913+
end
19071914
end
19081915

19091916
# "(" [flag-perm *(SP flag-perm)] ")"
19101917
def flag_perm__list
1911-
match_re(Patterns::FLAG_PERM_LIST, "PERMANENTFLAGS flag-perm list")[1]
1912-
.split(nil)
1913-
.map! { _1.start_with?("\\") ? _1[1..].capitalize.to_sym : _1 }
1918+
if (match = accept_re(Patterns::FLAG_PERM_LIST))
1919+
match[1].split(nil)
1920+
.map! { _1.delete_prefix!("\\") ? _1.capitalize.to_sym : _1 }
1921+
else
1922+
quirky__flag_list "PERMANENTFLAGS flag-perm list"
1923+
end
1924+
end
1925+
1926+
def quirky__flag_list(name)
1927+
match_re(Patterns::QUIRKY_FLAGS_LIST, "quirks mode #{name}")[1]
1928+
.scan(Patterns::QUIRKY_FLAG)
1929+
.map! { _1.delete_prefix!("\\") ? _1.capitalize.to_sym : _1 }
19141930
end
19151931

19161932
# See Patterns::MBX_LIST_FLAGS
19171933
def mbx_list_flags
19181934
match_re(Patterns::MBX_LIST_FLAGS, "mbx-list-flags")[1]
1919-
.split(nil).map! { _1[1..].capitalize.to_sym }
1935+
.split(nil)
1936+
.map! { _1.delete_prefix!("\\"); _1.capitalize.to_sym }
19201937
end
19211938

19221939
# See https://developers.google.com/gmail/imap/imap-extensions

‎test/net/imap/fixtures/response_parser/quirky_behaviors.yml

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -102,3 +102,39 @@
102102
(\"[email protected]\" NIL \"numbers.ryan\" \"satterfield.test\")
103103
(\"[email protected]\" NIL \"keneth_feeney\" \"will-walter.test\"))
104104
NIL NIL NIL \"<aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa@bbbbbbbbbbbbb.ccccccc.PROD.OUTLOOK.COM>\"))\r\n"
105+
106+
imap.gmail.com allows invalid atom-specials in flags:
107+
comment: |
108+
Upstream bug report: https://issuetracker.google.com/issues/315160951
109+
net-imap issue: https://github.com/ruby/net-imap/issues/241
110+
:response: "* FLAGS (\\Answered \\Flagged \\Draft \\Deleted \\Seen $Forwarded $Junk
111+
$MailFlagBit0 $MailFlagBit2 $NotJunk $NotPhishing $Phishing Forwarded
112+
JunkRecorded NotJunk OIB-Seen-INBOX OIB-Seen-Unsubscribe
113+
OIB-Seen-[Google Mail]/Alle Nachrichten)\r\n"
114+
:expected: !ruby/struct:Net::IMAP::UntaggedResponse
115+
name: FLAGS
116+
data:
117+
- :Answered
118+
- :Flagged
119+
- :Draft
120+
- :Deleted
121+
- :Seen
122+
- "$Forwarded"
123+
- "$Junk"
124+
- "$MailFlagBit0"
125+
- "$MailFlagBit2"
126+
- "$NotJunk"
127+
- "$NotPhishing"
128+
- "$Phishing"
129+
- Forwarded
130+
- JunkRecorded
131+
- NotJunk
132+
- OIB-Seen-INBOX
133+
- OIB-Seen-Unsubscribe
134+
- OIB-Seen-[Google
135+
- Mail]/Alle
136+
- Nachrichten
137+
raw_data: "* FLAGS (\\Answered \\Flagged \\Draft \\Deleted \\Seen $Forwarded
138+
$Junk $MailFlagBit0 $MailFlagBit2 $NotJunk $NotPhishing $Phishing Forwarded
139+
JunkRecorded NotJunk OIB-Seen-INBOX OIB-Seen-Unsubscribe OIB-Seen-[Google
140+
Mail]/Alle Nachrichten)\r\n"

0 commit comments

Comments
 (0)