Chunking 1.1: Split into reliable/ordered mode and unreliable/unordered mode (#153)

The reliable/ordered mode shortens the header to 1 byte and allows for
implementation optimisations. The unreliable/unordered mode is backwards
compatible with version 1.0.

Further changes:

- Add changelog
- Make chunk duplication handling optional
- Wording

Author: lgrahl

Chunking.md

@@ -15,48 +15,77 @@ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
 document are to be interpreted as described in [RFC
 2119](https://tools.ietf.org/html/rfc2119).
 
+## Modes
+
+This specification defines the following two modes:
+
+* **Reliable/Ordered**: Intended for reliable and ordered transmission
+  of chunks. The application that is reassembling chunks into a message
+  MUST ensure that chunks of a message are not reordered. Furthermore,
+  chunks of different messages SHALL NOT be interleaved.
+* **Unreliable/Unordered**: Intended for transmission of chunks where
+  chunks MAY be lost or reordered. Additionally, an implementation MAY
+  optionally be able to handle duplicated chunks.
+
 ## Chunk Size
 
 The term *chunk size* is referring to the resulting size from the
 concatenation of *chunk header* and *chunk data* in bytes.
 
 ## Chunk Header
 
-When converting data to chunks, a 9 byte header MUST be prepended to
-each chunk. This allows for sending a chunk over the network in any
-order.
-
-### Length
-
-The header is 9 bytes long.
+When converting data to chunks, a header MUST be prepended to each
+chunk of the following byte size:
 
-### Format
+* 1 byte **short header** for reliable/ordered mode, and
+* 9 byte **long header** for unreliable/unordered mode.
 
-The header is encoded in binary using network-oriented format (most
-significant byte first, also known as big-endian). It is structured as
-follows:
+Both header variants are encoded in binary using network-oriented format
+(most significant byte first, also known as big-endian).
 
-    |O|IIII|SSSS|
+### Short Header
 
-    - O: Options bit field (1 byte)
-    - I: Message id (4 bytes)
-    - S: Serial number (4 bytes)
+The short header contains a single byte called the *options bit field*:
 
 **Options bit field**
 
 The *options bit field* field is used to encode additional information
-about a chunk. Right now, only the least significant bit is being used.
-The other bits are reserved and MUST be set to `0`.
+about a chunk.
 
     MSB           LSB
     +---------------+
-    |0 0 0 0 0 0 0 E|
+    |R R R R R M M E|
     +---------------+
 
+    - R: Reserved, MUST be 0.
+
+    - M: Mode with the following values:
+
+         11 - reliable/ordered
+         10 - reserved
+         01 - reserved
+         00 - unreliable/unordered
+
     - E: End-of-message, this MUST be set to 1 if this is the
          last chunk of the message. Otherwise, it MUST be set
          to 0.
 
+### Long Header
+
+The long header contains a total of 9 bytes and is structured in the
+following way:
+
+    |O|IIII|SSSS|
+
+    - O: Options bit field (1 byte)
+    - I: Message id (4 bytes)
+    - S: Serial number (4 bytes)
+
+**Options bit field**
+
+Identical to the *options bit field* of the
+[Short Header](#short-header) format.
+
 **Message id**
 
 The *message id* SHALL be any 32 bit unsigned integer. It is RECOMMENDED
@@ -73,27 +102,58 @@ with 0 and MUST be incremented by 1 for every chunk.
 The chunk data MUST be appended to the chunk header. It MUST contain at
 least 1 byte of data.
 
-Every chunk MUST contain up to `chunk size - 9` bytes of data. Only the
-last chunk of a message may contain less than `chunk size - 9` bytes of
-chunk data.
+Every chunk MUST contain up to `chunk size - header size` bytes of
+data unless it is the last chunk of a message in which case it MAY
+contain less than `chunk size - header size` bytes of data.
 
-The chunk data MUST be chunked in a non-overlapping sequential way.
+The chunk data MUST be chunked in a non-overlapping, sequential way.
 
 ## Example
 
-When chunking the byte sequence `12345678` with a chunk size of 12 and
-the message id 42, the data MUST be chunked into the following three
-chunks:
+### Reliable/Ordered Mode
+
+When chunking the byte sequence `12345678` (where each digit is an 8
+bit unsigned integer) with a chunk size of 6, the data is being chunked
+into the following two chunks:
+
+    - First chunk:  `0b00000110 || 0x0102030405`
+    - Second chunk: `0b00000111 || 0x060708`
+
+### Unreliable/Unordered Mode
+
+When chunking the byte sequence `12345678` (where each digit is an 8
+bit unsigned integer) with a chunk size of 12 and the message id 42,
+the data is being chunked into the following three chunks:
 
     - First chunk:  `0b00000000 || 0x0000002a || 0x00000000 || 0x010203`
     - Second chunk: `0b00000000 || 0x0000002a || 0x00000001 || 0x040506`
     - Third chunk:  `0b00000001 || 0x0000002a || 0x00000002 || 0x0708`
 
 ## Unchunking
 
-Implementations MUST support unchunking of chunks that arrive in
-arbitrary order. This is usually done by keeping track of messages and
-the corresponding chunks.
-
+In unordered/unreliable mode, implementations MUST support unchunking of
+chunks that arrive in arbitrary order. This is usually done by keeping
+track of messages and the corresponding chunks.
 In order to prevent memory leaks when chunks are lost in transmission,
 implementations SHOULD provide a way to clean up incomplete messages.
+
+Implementations of the *unreliable/unordered* mode MAY optionally be
+able to handle duplicated chunks.
+
+## Changelog
+
+### 2019-02-06
+
+Version 1.1
+
+**Important**: Backwards compatibility to 1.0 can only be ensured by
+using the unreliable/unordered mode.
+
+* Add reliable/ordered mode with 1 byte header
+
+### 2016-09-29
+
+Version 1.0
+
+* Define unreliable/unordered mode with 9 byte header
+