Unsupported ID command variant

[duesee]

IMAP's ID command allows (), and nil to encode an empty parameter list:

id-params-list = "(" [field-value *(SP field-value)] ")" / nil
field-value    = string SP nstring

Some servers don't recognize both variants.

Observed in: Nemesis (GMX, Web.de, ...), and Exchange (See https://github.com/modern-email/defects/issues/12#issuecomment-1845491532)
Reported: No
Status: Unknown
Comment: None
Proposed solution(s):
    Don't send an empty ID command. Note: Proxies may require extra attention!

---

RFC 2971 uses ...

id_params_list ::= "(" #(string SPACE nstring) ")" / nil
    ;; list of field value pairs

... to define the parameter list.

I believe that ::= should be = (according to the current ABNF definition) and that # means "separated by whitespace"¹ (according to the examples).

Question: Does # mean 1* ("at least one") or * ("zero or many")?

¹ I saw it described in other contexts as "separated by comma".

Offending implementation: Nemesis, Exchange

[cketti]

RFC 2971 contains all the necessary references.

4. Formal Syntax

This syntax is intended to augment the grammar specified in [IMAP4rev1] in order to provide for the ID command. This specification uses the augmented Backus-Naur Form (BNF) notation as used in [IMAP4rev1].

The referenced IMAP4rev1 spec (RFC 2060) contains:

9. Formal Syntax

The following syntax specification uses the augmented Backus-Naur Form (BNF) notation as specified in [RFC-822] with one exception; the delimiter used with the "#" construct is a single space (SPACE) and not one or more commas.

The referenced RFC 822 has this to say:

2.7. #RULE: LISTS

A construct "#" is defined, similar to "*", as follows:

<l>#<m>element

indicating at least <l> and at most <m> elements, each separated by one or more commas (","). This makes the usual form of lists very easy; a rule such as '(element *("," element))' can be shown as "1#element". Wherever this construct is used, null elements are allowed, but do not contribute to the count of elements present. That is, "(element),,(element)" is permitted, but counts as only two elements. Therefore, where at least one element is required, at least one non-null element must be present. Default values are 0 and infinity so that "#(element)" allows any number, including zero; "1#element" requires at least one; and "1#2element" allows one or two.

So the original

id_params_list ::= "(" #(string SPACE nstring) ")" / nil

could be written in modern notation as

field-value    = string SP nstring
id-params-list = "(" [field-value *(SP field-value)] ")" / nil

or

id-params-list = "(" [string SP nstring *(SP string SP nstring)] ")" / nil

if you prefer to have it all in one rule.

[duesee]

Thanks! I should have followed the RFC tracks more carefully.

[duesee]

Follow-up question:

The above definition allows ...

• nil -> no list,
• () -> empty (but present) list, and
• ("..." "...") -> list with >=1 values

It's tempting to treat nil and () as an empy vector. But I guess nil has the specific meaning "wants to send no information" and () has the specific meaning "wants to send information but there is none"? Still, I feel there could be providers that are "surprised" by A ID ().

[duesee]

Okay, that was fast ...

$ openssl s_client -crlf -quiet -connect imap.gmx.de:993
* OK [CAPABILITY IMAP4rev1 CHILDREN ENABLE ID IDLE LIST-EXTENDED LIST-STATUS LITERAL- MOVE NAMESPACE SASL-IR SORT SPECIAL-USE THREAD=ORDEREDSUBJECT UIDPLUS UNSELECT WITHIN AUTH=LOGIN AUTH=PLAIN] IMAP server ready H migmx112 27.4 IMAP-1McqiY-1rkrJv1mBW-00ZTj0
A ID nil
* ID ("name" "nemesis" "version" "27.4")
A OK ID completed
B ID ()
B BAD expected string (quoted / literal) instead of ")"

Maybe it should be a Vec<> that is always encoded as nil when empty... Would be more restrictive than the standard, though.

* OK The Microsoft Exchange IMAP4 service is ready. [RgBSADAAUAAyADgAMQBDAEEAMAAyADUANwAuAEQARQBVAFAAMgA4ADEALgBQAFIATwBEAC4ATwBVAFQATABPAE8ASwAuAEMATwBNAA==]
A ID nil
* ID ("name" "Microsoft.Exchange.Imap4.Imap4Server" "version" "15.20")
A OK ID completed
A ID ()
A BAD ID failed

[duesee]

Dovecot, Gmail, Zoho are fine with A ID ().

[cketti]

Most people implementing the ID extension, probably don't spend much time thinking about the empty list case. nil has the advantage that it's used in the example for both the command and the response. So implementers most likely accounted for this case.

As a server I would expect the client to at least send a "name" value. Otherwise why send the ID command? Email clients typically don't display the server response to the user.

[duesee]

Yeah. The reason I'm asking is because I'm not sure how to encode the ID command in imap-types. (Currently working on duesee/imap-codec#392.)

Option<Vec<(key, value)>> would be most precise. But given that GMX and Exchange choke on A ID (), maybe it makes more sense to always encode an empty list as nil. But in this case we don't need the Option<>.

The RFC explicitly mentions proxies: When a proxy removes a particular list item (from a list with only one element), it could maintain the intent "client wanted to send a list" by forwarding (). Forwarding nil instead would mean "client didn't send a list".

[duesee]

I should explain a bit: imap-types wants to be a "type-safe IMAP standard" that strictly encodes the IMAP syntax. Also as a reference. Thus, I feel like I can't be too opinionated. Allowing () (but still always sending nil in a default "quirk mode") would work.

[cketti]

What's written in the standards and what's used in the wild are sometimes very different things. I'm not sure a quirks mode will be enough to bridge that gap. It feels like those should be two different libraries. You probably disagree right now. But I'm sure you'll eventually come to the same conclusion in time 😄

[duesee]

Hehe, you mean we should have an imap-types (ivory tower version) and an imap-types (real-world version)? -- could be!

Current status is: We have an identity proxy that works well with Thunderbird and a few large providers. (That's where some of the defects and fixes in this repo come from.) So, I still hope we can at least get the syntax straight. IMAP operations are a different topic ...

But I see where you are coming from and how you may be right in the end -- even though I hope you are not :-)

Looking forward to the protocol police.

[duesee]

@chibenwa, I think you came across something similar :-)

[chibenwa]

Correct.

It was Office 365 failing if the server answer contains () but suceeded when it contains NIL instead.

The not fun fact was that it was completly crashing the connection...