Skip to content

Commit a7bfcaf

Browse files
author
Dr. Christian Betz
committed
Merge remote-tracking branch 'MastodonC/franzy-nippy/master' into develop
# Conflicts: # .gitignore # CHANGELOG.md
2 parents eb9328c + 05a2001 commit a7bfcaf

File tree

6 files changed

+267
-0
lines changed

6 files changed

+267
-0
lines changed

README.md

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
# Franzy-Nippy
2+
3+
[Kafka](http://kafka.apache.org/documentation.html) serializer using the excellent serialization library, [Nippy](https://github.com/ptaoussanis/nippy).
4+
5+
Great fit with [Franzy](https://github.com/ymilky/franzy), a Clojure Kafka client, though not required. Feel free to use this serializer with any Kafka client, including via Java, Scala, Groovy, or any other JVM language.
6+
7+
## Why
8+
9+
* You want to compress your data when sending to and from Kafka.
10+
* You want fast de/serialization, via nippy and its underlying implementation.
11+
* You are using a Kafka client such as [Franzy](https://github.com/ymilky/franzy) and need a pluggable, robust serializer.
12+
* You want to serialize Clojure data types with little effort.
13+
* You want seamless serialization, no embedded serialization calls at call sites or `.getBytes` ugly things floating around.
14+
15+
## Docs
16+
17+
* Read the browsable [API](http://ymilky.github.io/franzy-nippy/api/index.html)
18+
* See [Franzy Source](https://github.com/ymilky/franzy) and docs for more information about serializers/deserializers.
19+
* For more information about serializer options, compression, etc, see the official [Nippy](https://github.com/ptaoussanis/nippy) repo.
20+
21+
## Installation
22+
23+
Add the necessary dependency to your project:
24+
25+
```clojure
26+
[ymilky/franzy-nippy "0.0.1"]
27+
```
28+
[![Clojars Project](https://img.shields.io/clojars/v/ymilky/franzy-fressian.svg)](https://clojars.org/ymilky/franzy-fressian)
29+
30+
## Serializing
31+
32+
First, require:
33+
34+
```clojure
35+
(ns my-ns
36+
(:require [franzy.serialization.nippy.serializers :as serializers]))
37+
```
38+
39+
Then use with a producer, such as the one with [Franzy](https://github.com/ymilky/franzy).
40+
41+
```clojure
42+
(let [;;optionally specify via Kakfa Config key - value.serializer using fully qualified class name
43+
pc {:bootstrap.servers ["127.0.0.1"]}
44+
;;Serializes producer record keys, ex: (keyword-serializer) from Franzy
45+
key-serializer (your-key-serializer-type)
46+
;;Serializes producer record values using nippy, call (serializers/nippy-serializer options) to pass nippy options
47+
value-serializer (serializers/nippy-serializer)]
48+
(with-open [p (producer/make-producer pc key-serializer value-serializer)]
49+
;;spray useless data to Kafka using Clojure types
50+
(send-async! "aliens-wearing-curtains" 2262
51+
["s: Who? The Narn or the Centauri? k: yes" "...is dead.
52+
k: We are all Kosh." "If you watch Legend of the Rangers, you will die."]))))
53+
```
54+
55+
## Deserializing
56+
57+
First, require:
58+
59+
```clojure
60+
(ns my-ns
61+
(:require [franzy.serialization.nippy.deserializers :as deserializers]))
62+
```
63+
64+
Then use with a consumer, such as the one with [Franzy](https://github.com/ymilky/franzy).
65+
66+
```clojure
67+
(let [;;optionally specify via Kafka Config key - value.deserializer using fully qualified class name
68+
cc {:bootstrap.servers ["127.0.0.1:9092"]
69+
:group.id "we-are-purple"}
70+
;;Deserializes your record keys, ex: (keyword-deserializer) from Franzy
71+
key-deserializer (your-key-deserializer-type)
72+
;;the value deserializer should be the same kind as the serializer, don't mix and match
73+
;;call (serializers/nippy-deserializer options) to pass nippy options
74+
value-deserializer (deserializers/nippy-deserializer)
75+
topic "aliens-wearing-curtains"
76+
topic-partitions [{:topic topic :partition 2262}]]
77+
(with-open [c (consumer/make-consumer cc key-deserializer value-deserializer)]
78+
(assign-partitions! c topic-partitions)
79+
(seek-to-beginning-offset! c topic-partitions)
80+
(poll! c)))
81+
```
82+
83+
## License
84+
85+
Copyright © 2016 Yossi M. (ymilky).
86+
87+
Distributed under the Eclipse Public License either version 1.0 or (at your option) any later version.
88+
89+
Use at your own risk, I am not responsible or liable. Please give credit if you use pieces of this library or otherwise, it is much appreciated.

project.clj

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
(defproject ymilky/franzy-nippy "0.0.1"
2+
:description "A Kafka Serializer/Deserializer supporting Nippy, and an add-on for Franzy, a Clojure Kafka client."
3+
:url "https://github.com/ymilky/franzy-nippy"
4+
:author "ymilky and others, but see README"
5+
:license {:name "Eclipse Public License"
6+
:url "http://www.eclipse.org/legal/epl-v10.html"}
7+
:repositories {"snapshots" {:url "https://clojars.org/repo"
8+
:username :env
9+
:password :env
10+
:sign-releases false}
11+
"releases" {:url "https://clojars.org/repo"
12+
:username :env
13+
:password :env
14+
:sign-releases false}}
15+
:dependencies [[org.clojure/clojure "1.8.0"]
16+
[org.apache.kafka/kafka-clients "0.9.0.1"]
17+
[com.taoensso/nippy "2.11.1"]]
18+
:plugins [[lein-codox "0.9.4"]]
19+
:codox {:metadata {:doc/format :markdown}
20+
:doc-paths ["README.md"]
21+
:output-path "doc/api"}
22+
:profiles {:dev {:dependencies [[midje "1.7.0"]]
23+
:plugins [[lein-midje "3.2"]
24+
[lein-set-version "0.4.1"]
25+
[lein-update-dependency "0.1.2"]
26+
[lein-pprint "1.1.1"]]}
27+
:reflection-check {:global-vars
28+
{*warn-on-reflection* true
29+
*assert* false
30+
*unchecked-math* :warn-on-boxed}}})
Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
(ns franzy.serialization.nippy.deserializers
2+
(:require [taoensso.nippy :as nippy])
3+
(:import (org.apache.kafka.common.serialization Deserializer)))
4+
5+
(deftype NippyDeserializer [opts]
6+
Deserializer
7+
(configure [_ _ _])
8+
(deserialize [_ _ data]
9+
(nippy/thaw data opts))
10+
(close [_]))
11+
12+
(defn nippy-deserializer
13+
"Nippy deserializer for Apache Kafka.
14+
Use for serializing Kafka values.
15+
16+
> Notes: You may pass any of the built-in nippy options via the opts map, using
17+
the 1-arity version of this function."
18+
(^NippyDeserializer [] (nippy-deserializer nil))
19+
(^NippyDeserializer [opts]
20+
(NippyDeserializer. opts)))
Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
(ns franzy.serialization.nippy.serializers
2+
(:require [taoensso.nippy :as nippy])
3+
(:import (org.apache.kafka.common.serialization Serializer)))
4+
5+
(deftype NippySerializer [opts]
6+
Serializer
7+
(configure [_ _ _])
8+
(serialize [_ _ data]
9+
(nippy/freeze data opts))
10+
(close [_]))
11+
12+
(defn nippy-serializer
13+
"Nippy serializer for Apache Kafka.
14+
Use for serializing Kafka values.
15+
16+
> Notes: You may pass any of the built-in nippy options via the opts map, using
17+
the 1-arity version of this function."
18+
(^NippySerializer [] (nippy-serializer nil))
19+
(^NippySerializer [opts]
20+
(NippySerializer. opts)))

test/franzy/core_test.clj

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
(ns franzy.core-test
2+
(:require [midje.sweet :refer :all]
3+
[franzy-nippy.core :refer :all]))
Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
(ns franzy.serialization.nippy.serialization-tests
2+
(:require [midje.sweet :refer :all]
3+
[franzy.serialization.nippy.serializers :as serializers]
4+
[franzy.serialization.nippy.deserializers :as deserializers])
5+
(:import (org.apache.kafka.common.serialization Deserializer Serializer)
6+
(java.util UUID)))
7+
8+
(facts
9+
"Nippy serializers/deserializers serialize and deserialize Clojure data structures." :serializers
10+
(let [serializer (serializers/nippy-serializer)
11+
deserializer (deserializers/nippy-deserializer)
12+
topic "music-education"
13+
data {:good-bands ["New Order" "Joy Division" "The Cure" "The Smiths" "Pulp" "Jesus and Mary Chain"]
14+
:terrible-bands #{"The Eagles" "Most of American Music in the 90s"}
15+
:things-pretending-to-be-bands `("Justin Bieber" "Kanye West" "Beonce" "Arcade Fire")
16+
:good-year 1984
17+
:essential-album :script-of-the-bridge-by-the-chameleons
18+
:most-overrated "Jennifer Lopez"
19+
:good-music-this-year nil}]
20+
(fact
21+
"A nippy serializer is a Kafka serializer."
22+
(instance? Serializer serializer) => true)
23+
(fact
24+
"A nippy deserializer is a Kafka deserializer."
25+
(instance? Deserializer deserializer) => true)
26+
(fact
27+
"A nippy serializer can serialize a string."
28+
(let [val "Slowdive is under appreciated"]
29+
(.serialize serializer topic val) =not=> nil
30+
(->> (.serialize serializer topic val)
31+
(.deserialize deserializer topic)) => val))
32+
(fact
33+
"A nippy serializer can serialize an number."
34+
(let [val 1982]
35+
(.serialize serializer topic val) =not=> nil
36+
(->> (.serialize serializer topic val)
37+
(.deserialize deserializer topic)) => val))
38+
(fact
39+
"A nippy serializer can serialize a short."
40+
(let [val (short 1)]
41+
(.serialize serializer topic val) =not=> nil
42+
(->> (.serialize serializer topic val)
43+
(.deserialize deserializer topic)) => val))
44+
(fact
45+
"A nippy serializer can serialize a long."
46+
(let [val (long Long/MAX_VALUE)]
47+
(.serialize serializer topic val) =not=> nil
48+
(->> (.serialize serializer topic val)
49+
(.deserialize deserializer topic)) => val))
50+
(fact
51+
"A nippy serializer can serialize an integer."
52+
(let [val (int Integer/MAX_VALUE)]
53+
(.serialize serializer topic val) =not=> nil
54+
(->> (.serialize serializer topic val)
55+
(.deserialize deserializer topic)) => val))
56+
(fact
57+
"A nippy serializer can serialize a UUID."
58+
(let [val (UUID/randomUUID)]
59+
(.serialize serializer topic val) =not=> nil
60+
(->> (.serialize serializer topic val)
61+
(.deserialize deserializer topic)) => val))
62+
(fact
63+
"A nippy serializer can serialize a map."
64+
(let [val {:great-song "porcelain raft - dragonfly"}]
65+
(.serialize serializer topic val) =not=> nil
66+
(->> (.serialize serializer topic val)
67+
(.deserialize deserializer topic)) => val))
68+
(fact
69+
"A nippy serializer can serialize a vector."
70+
(let [val ["DIIV" "Chapterhouse" "Moose" "Ride"]]
71+
(.serialize serializer topic val) =not=> nil
72+
(->> (.serialize serializer topic val)
73+
(.deserialize deserializer topic)) => val))
74+
(fact
75+
"A nippy serializer can serialize a list."
76+
(let [val '("Suede" "Blur" "Elbow" "Cast" "Doves")]
77+
(.serialize serializer topic val) =not=> nil
78+
(->> (.serialize serializer topic val)
79+
(.deserialize deserializer topic)) => val))
80+
(fact
81+
"A nippy serializer can serialize a set."
82+
(let [val #{"Forget That You're Young" "Black Satin" "Hallucinations" "With My Eyes Closed" "Here Comes Mary"}]
83+
(.serialize serializer topic val) =not=> nil
84+
(->> (.serialize serializer topic val)
85+
(.deserialize deserializer topic)) => val))
86+
(fact
87+
"A nippy serializer can serialize a keywords."
88+
(let [val :raveonettes]
89+
(.serialize serializer topic val) =not=> nil
90+
(->> (.serialize serializer topic val)
91+
(.deserialize deserializer topic)) => val))
92+
(fact
93+
"A nippy serializer can serialize a function, but it's stupid to do so."
94+
(let [val (fn [x] (+ 1 x))]
95+
(.serialize serializer topic val) =not=> nil))
96+
(fact
97+
"A nippy serializer should be able to produce the same data in a round trip." :serializers
98+
(->> (.serialize serializer topic data)
99+
(.deserialize deserializer topic)) => data)
100+
(fact
101+
"A nippy serializer should handle nil data, just in case..."
102+
(->> (.serialize serializer topic nil) =not=> nil))
103+
(fact
104+
"A nippy deserializer should handle nil data, just in case..."
105+
(->> (.deserialize deserializer topic nil) =not=> nil))))

0 commit comments

Comments
 (0)