Merge pull request #5 from dbohdan/master

Extension types, timestamp, UTF-8 fix

.github/workflows/ci.yml

@@ -5,7 +5,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
     - name: Install dependencies
-      run: sudo apt install tcl
+      run: sudo apt install tcl tcllib
     - name: Checkout
       uses: actions/checkout@v1
     - name: Run tests

README

@@ -21,6 +21,7 @@ packerObject pack args
 packerObject reset
 msgpack::unpacker new
 unpackerObject destroy
+unpackerObject set_ext_unpacker ?type? ?script?
 unpackerObject unpack_stream istream callback
 unpackerObject unpack_string istring ?callback?
 msgpack array2list 
@@ -32,7 +33,7 @@ msgpack unpack string
 DESCRIPTION
 ===========
 
-The _msgpack_ package is a pure Tcl implementation the MessagePack object
+The _msgpack_ package is a pure Tcl implementation of the MessagePack object
 serialization library. You can find the code at GitHub:
 <URL:https://github.com/jdc8/msgpack>. MessagePack can be found at
 <URL:http://msgpack.org/>.
@@ -78,6 +79,13 @@ Unpacker class
 
         Destroy the unpacker object.
 
+    unpackerObject set_ext_unpacker ?type? ?script?
+
+        Set the handler for an extension type. When the unpacker encounters
+        extension type type, it will call script with the type and the data as
+        its arguments. Omit script or script and type to get handlers. Set the
+        handler for a type to an empty string to disable it.
+
     unpackerObject unpack_stream istream callback
 
         Unpack data read from the istream argument. The callback command is
@@ -111,23 +119,27 @@ Unpacker class
 Type information found in the unpacked MessagePack objects can be one of the
 following:
 
-    nil
+    array
+
+    bin
 
     boolean
 
-    integer
+    ext
 
     float32
 
     float64
 
-    bin
+    integer
 
-    str
+    map
 
-    array
+    nil
 
-    map
+    str
+
+    timestamp
 
 Values can be nested type/value list.
 
@@ -186,10 +198,34 @@ specifiers and if needed a value. The list below shows the supported types:
         map with the dict size as argument, followed by calling method pack
         keyType and method pack valueType for each key/value pair in the dict.
 
+    ext type bytes
+
+        Add a byte array of a chosen extension type to the packed data.
+
     false
 
         Add a boolean with value false to the packed data.
 
+    fix_ext1 type byte
+
+        Add 1 byte of a chosen extension type to the packed data.
+
+    fix_ext2 type bytes
+
+        Add 2 bytes of a chosen extension type to the packed data.
+
+    fix_ext4 type bytes
+
+        Add 4 bytes of a chosen extension type to the packed data.
+
+    fix_ext8 type bytes
+
+        Add 8 bytes of a chosen extension type to the packed data.
+
+    fix_ext16 type bytes
+
+        Add 16 bytes of a chosen extension type to the packed data.
+
     fix_int8 data
 
         Add an 8 bit integer to the packed data.
@@ -224,7 +260,7 @@ specifiers and if needed a value. The list below shows the supported types:
 
     float32 data
 
-        add a 32-bit float to the packed data.
+        Add a 32-bit float to the packed data.
 
     float64 data
 
@@ -274,6 +310,14 @@ specifiers and if needed a value. The list below shows the supported types:
         Add the map size to the packed data. Must be followed by size pairs of
         calls to method pack to add the keys and values to the packed data.
 
+    microseconds micros
+
+        Add a microsecond timestamp to the packed data as a timestamp96.
+
+    milliseconds millis
+
+        Add a millisecond timestamp to the packed data as a timestamp96.
+
     nil
 
         Add a nil to the packed data.
@@ -293,6 +337,22 @@ specifiers and if needed a value. The list below shows the supported types:
         pack keyType and method pack valueType for each key/value pair in the
         array.
 
+    timestamp32 seconds
+
+        Add a 32-bit unsigned timestamp to the packed data.
+
+    timestamp64 seconds nanoseconds
+
+        Add a 64-bit timestamp (34 bits for seconds, 30 bits for nanoseconds,
+        both unsigned) to the packed data. Nanoseconds must not exceed
+        999999999.
+
+    timestamp96 seconds nanoseconds
+
+        Add a 96-bit timestamp (64 bits for seconds, signed, and 32 bits for
+        nanoseconds, unsigned) to the packed data. Nanoseconds must not exceed
+        999999999.
+
     true
 
         Add a boolean with value true to the packed data.
@@ -378,6 +438,26 @@ unpack utility function:
 
 | {integer 4294967295} {bin {A Utility example}} {map {{integer 3} {str three} {integer 4} {str four}}}
 
+With set_ext_unpacker you can register a handler to unpack custom extension
+types.
+
+| set up [msgpack::unpacker new]
+
+| proc xor {n type data} {
+|     set res {}
+|     foreach b [split $data {}] {
+|         set code [scan $b %c]
+|         append res [format %c [expr { $code ^ $n }]]
+|     }
+
+|     return [list encrypted $res]
+| }
+
+| $up set_ext_unpacker 100 {xor 5}
+| # Prints "{encrypted Hello!}".
+| puts [$up unpack_string [msgpack pack ext 100 M`iij$]]
+| $up destroy
+
 Bugs, ideas, feedback
 =====================
 
@@ -405,5 +485,6 @@ Serialization
 COPYRIGHT
 =========
 
-Copyright (c) Jos Decoster <[email protected]>
+Copyright (c) 2013 Jos Decoster <[email protected]>
+Copyright (c) 2020 D. Bohdan <https://dbohdan.com/>
 

doc/examples.inc

@@ -39,3 +39,22 @@ An using the [cmd unpack] utility function to unpack the data:
 After unpacking, the following list of type/value pairs is returned by the [cmd unpack] utility function:
 
 [example {{integer 4294967295} {bin {A Utility example}} {map {{integer 3} {str three} {integer 4} {str four}}}}]
+
+With [cmd set_ext_unpacker] you can register a handler to unpack custom extension types.
+
+[example {set up [msgpack::unpacker new]
+
+proc xor {n type data} {
+    set res {}
+    foreach b [split $data {}] {
+        set code [scan $b %c]
+        append res [format %c [expr { $code ^ $n }]]
+    }
+
+    return [list encrypted $res]
+}
+
+$up set_ext_unpacker 100 {xor 5}
+# Prints "{encrypted Hello!}".
+puts [$up unpack_string [msgpack pack ext 100 M`iij$]]
+$up destroy}]

doc/msgpack.man

@@ -1,6 +1,7 @@
 [comment {-*- tcl -*- doctools manpage}]
 [manpage_begin msgpack n 2.0.0]
-[copyright {Jos Decoster <[email protected]>}]
+[copyright {2013 Jos Decoster <[email protected]>}]
+[copyright {2020 D. Bohdan <https://dbohdan.com/>}]
 [moddesc {A pure Tcl implementation of the MessagePack object serialization library}]
 [category  {Serialization}]
 [keywords {MessagePack}]
@@ -11,8 +12,8 @@
 [require msgpack [opt 2.0.0]]
 [description]
 
-The [term msgpack] package is a pure Tcl implementation the MessagePack object
-serialization library. You can find the code at GitHub:
+The [term msgpack] package is a pure Tcl implementation of the MessagePack
+object serialization library. You can find the code at GitHub:
 [uri https://github.com/jdc8/msgpack]. MessagePack can be found at
 [uri http://msgpack.org/].
 

doc/packopts.inc

@@ -27,10 +27,34 @@ Add a dict to the packed data. This is equivalent to calling method
 method [method {pack keyType}] and method [method {pack valueType}]
 for each key/value pair in the dict.
 
+[opt_def [method ext] "[arg type] [arg bytes]"]
+
+Add a byte array of a chosen extension type to the packed data.
+
 [opt_def [method false]]
 
 Add a boolean with value [cmd false] to the packed data.
 
+[opt_def [method fix_ext1] "[arg type] [arg byte]"]
+
+Add 1 byte of a chosen extension type to the packed data.
+
+[opt_def [method fix_ext2] "[arg type] [arg bytes]"]
+
+Add 2 bytes of a chosen extension type to the packed data.
+
+[opt_def [method fix_ext4] "[arg type] [arg bytes]"]
+
+Add 4 bytes of a chosen extension type to the packed data.
+
+[opt_def [method fix_ext8] "[arg type] [arg bytes]"]
+
+Add 8 bytes of a chosen extension type to the packed data.
+
+[opt_def [method fix_ext16] "[arg type] [arg bytes]"]
+
+Add 16 bytes of a chosen extension type to the packed data.
+
 [opt_def [method fix_int8] [arg data]]
 
 Add an 8 bit integer to the packed data.
@@ -65,7 +89,7 @@ Add a 64 bit unsigned integer to the packed data.
 
 [opt_def [method float32] [arg data]]
 
-add a 32-bit float to the packed data.
+Add a 32-bit float to the packed data.
 
 [opt_def [method float64] [arg data]]
 
@@ -110,6 +134,14 @@ Add a long long integer to the packed data.
 Add the map size to the packed data. Must be followed by [cmd size] pairs of
 calls to method [method pack] to add the keys and values to the packed data.
 
+[opt_def [method microseconds] [arg micros]]
+
+Add a microsecond timestamp to the packed data as a [method timestamp96].
+
+[opt_def [method milliseconds] [arg millis]]
+
+Add a millisecond timestamp to the packed data as a [method timestamp96].
+
 [opt_def [method nil]]
 
 Add a nil to the packed data.
@@ -129,6 +161,21 @@ Add a Tcl array to the packed data. This is equivalent to calling method
 method [method {pack keyType}] and method [method {pack valueType}]
 for each key/value pair in the array.
 
+[opt_def [method timestamp32] [arg seconds]]
+
+Add a 32-bit unsigned timestamp to the packed data.
+
+[opt_def [method timestamp64] "[arg seconds] [arg nanoseconds]"]
+
+Add a 64-bit timestamp (34 bits for seconds, 30 bits for nanoseconds,
+both unsigned) to the packed data.  Nanoseconds must not exceed 999999999.
+
+[opt_def [method timestamp96] "[arg seconds] [arg nanoseconds]"]
+
+Add a 96-bit timestamp (64 bits for seconds, signed, and 32 bits for
+nanoseconds, unsigned) to the packed data.  Nanoseconds must not exceed
+999999999.
+
 [opt_def [method true]]
 
 Add a boolean with value [cmd true] to the packed data.

doc/unpacker.inc

@@ -11,6 +11,13 @@ implementing the MessagePack unpacking.
 
 Destroy the unpacker object.
 
+[call [arg unpackerObject] [method set_ext_unpacker] [opt [arg type]] [opt [arg script]]]
+
+Set the handler for an extension type.  When the unpacker encounters extension
+type [arg type], it will call [arg script] with the type and the data as its
+arguments.  Omit [arg script] or [arg script] and [arg type] to get handlers.
+Set the handler for a type to an empty string to disable it.
+
 [call [arg unpackerObject] [method unpack_stream] [arg istream] [arg callback]]
 
 Unpack data read from the [arg istream] argument. The [arg callback] command is
@@ -50,15 +57,17 @@ Type information found in the unpacked MessagePack objects can be one of the
 following:
 
 [list_begin options]
-[opt_def nil]
+[opt_def array]
+[opt_def bin]
 [opt_def boolean]
-[opt_def integer]
+[opt_def ext]
 [opt_def float32]
 [opt_def float64]
-[opt_def bin]
-[opt_def str]
-[opt_def array]
+[opt_def integer]
 [opt_def map]
+[opt_def nil]
+[opt_def str]
+[opt_def timestamp]
 [list_end]
 
 Values can be nested type/value list.