Collections support #758

mosyp · 2017-04-24T17:03:03Z

Fixes #706, #730, #309 and probably something more
This PR's brings collections support for row elements (SQL Arrays, Cassandra Collections)

SQL

Introduced support for SQL Arrays. They're mapped into quill as instances of Seq, which means we can represent them as any type which implements Seq.
Array support by default mixed into Postgres contexts

Cassandra

Introduced support of Cassandra's List, Set and Map. They're mapped into quill as immutable collections.
Introduced CassandraType[T] - marker which signals to quill that type T is already supported by Cassandra codecs so not additional mapping is needed
Introduced CassandraMapper[I, O] - additional mapper to/from CassandraType[T]
Provide implicit mappers for introducing CassandraMapper from MappingEncoding and CassandraType

We can use CassandraType[T] for custom codecs and reuse later on for UDT for example

Checklist

@getquill/maintainers

…raversable encoding to core module

fwbrasil · 2017-04-25T04:47:21Z

@mentegy this is great! Let us know if there's something we can help with.

fwbrasil · 2017-04-25T04:55:24Z

@mentegy it seems that @mxl started working on collections for the cassandra module: #644

mosyp · 2017-04-25T16:15:30Z

@fwbrasil Yes I saw it, but it seems to be inactive now. Also, I have a little bit another design idea as far as you can see from my latest commit. Please look at it, I have created a separate module for cassandra macros. It is already support primitive types (using boxing methods from Predef).
Next step is to create mappers for other cassandra supported AnyRefs types and MappedEncoding. If you have some idea regarding them please advice

fwbrasil · 2017-04-25T16:43:31Z

@mentegy cool, I'll take a look in the evening today

mxl · 2017-04-25T18:52:23Z

I started that PR long ago and did not have time to make much progress so it's nice that @mentegy will implement Cassandra collections support.

fwbrasil · 2017-04-26T04:36:57Z

quill-async-postgres/src/main/scala/io/getquill/context/async/ArrayAsyncEncoding.scala

+ * Base trait for encoding sql arrays via async driver.
+ * We say `array` only in scope of sql driver. In Quill we represent them as instances of Traversable
+ */
+trait ArrayAsyncEncoding extends ArrayEncoding {


Wdyt about having separate traits for encoders and decoders? I'd be more similar to the rest of the encoding classes.

Make sense, will do

fwbrasil · 2017-04-26T04:47:31Z

...-macros/src/main/scala/io/getquill/context/cassandra/encoding/CollectionsEncodingMacro.scala

+
+import scala.reflect.macros.blackbox.{ Context => MacroContext }
+
+trait CollectionsEncodingMacro {


Could you share some context on why you need these macros for cassandra? It's not clear to me

On the first glance it looked easier to implement because of the following:
For Cassandra encoders/decoders we need a some kind of mapper:

boxing/unboxing primitives (this needs to be done manually)

identity function for supported types

mapped encoding

For list and set we can implement something like with sql arrays - collection encoder for each base type. But for map?
One solution is to introduce some type class which will mark supported types and pass each into implicit context. Also another encoders for AnyVals (oh, i completely forgot about anyvals) and primitives. And another one for mapping encoders. This looked a boilerplate for me for the first time and that's why I went into macros.
Having such "CassandraType[T]" will allow as to create generic encoder using cassandra row.set[T]. Also we can create some mechanism to create custom CassandraType[T] in order to create cassandra codecs for it.
Wdyt? I can play around it

…de implicits for each supported type

mosyp

MappedType[T, Cas] is designed to map any scala type of T to already supported type Cas via cassandra codecs.
type CassandraType[T] = MappedType[T, T] - signals that this type T is already supported so no transformations is needed. Both of these type is needed for collection support. Later on we can reuse this for UDT for example

mosyp · 2017-04-26T21:49:56Z

quill-cassandra/src/test/scala/io/getquill/context/cassandra/ListsEncodingSpec.scala

+  }
+
+  /* implicit def mappedEncodingForMappedType[I, O, Cas] is not working!!!
+  "Mapped encoding for mapped type" in {


This issue blocks me from complete cassandra collection support

mosyp · 2017-04-26T21:50:54Z

quill-cassandra/src/main/scala/io/getquill/context/cassandra/encoding/CassandraTypes.scala

+    m2:         MappedEncoding[O, I],
+    mappedType: MappedType[O, Cas]
+  ): MappedType[I, Cas] = MappedType(m1.f.andThen(mappedType.encode), mappedType.decode.andThen(m2.f))
+


This implicits is not working for unknowns reasons for me. Maybe I'm missing something?

…ndra collections

…eq (since it more convenient)

- add tests in sql base module regarding arrays encoding - add decoderUnsafe to MirrorDecoder to reduce ClassTags (removed unnecessary classtags from cassandra collection decoders)

mosyp · 2017-04-29T15:58:00Z

@fwbrasil I think it's completed. Please review.
Also check whereas updated info regarding changes in readme is sufficient.
Thanks!

mosyp · 2017-04-29T17:55:58Z

[tut] compiling: /app/target/tut/README.md
[tut] *** Error reported at /app/target/tut/README.md:706
<console>:28: error: not found: type LocalDate
       case class Person(id: Int, phones: List[String], cards: Vector[Int], dates: Seq[LocalDate])

Seems like tut doesn't like my example with LocalDate in readme. Also I found that #751 has the same problem.
Is there something we can do with tut or should I fix my code example in readme?

mosyp · 2017-05-02T17:38:17Z

@fwbrasil tut still cannot compile, it complains that mirror sql context does not have arrays encoding. Should I introduce arrays encoding for sql mirror context (it means thta for all idioms mirror will encode/decode arrays) or just make tut not compile that line?

fwbrasil · 2017-05-02T18:30:04Z

@mentegy I think it'd be nice if the mirror source also supports arrays, but I don't think it's a blocker. We can create a ticket and work on it later.

mosyp · 2017-05-02T19:56:12Z

@fwbrasil I have removed scala prefix from code examples to make it compile and will create new issue to introduce arrays support to mirror context once this PR will be merged. Thanks

fwbrasil

It's great that we don't need macros for cassandra! Thank you for the contribution, it's an important feature.

mosyp · 2017-05-05T22:04:19Z

@fwbrasil You're welcome!
Also thanks for approving this. Introducing CassandraType[T] will allow me to implement udt support for quill soon

fwbrasil · 2017-05-05T22:58:56Z

@getquill/maintainers let me know if you have objections since this is a relatively large change, I'm planning to merge it tomorrow.

mxl · 2017-05-07T15:39:04Z

Looks awesome 👍

mentegy added 2 commits April 24, 2017 17:10

Add support for Array encoding to JDBC module and support of mapped t…

97de30b

…raversable encoding to core module

Add support for Array encoding to async module

7cb57c0

mosyp changed the title ~~[WIP] Quill collection support~~ [WIP] Quill collections support Apr 24, 2017

mosyp changed the title ~~[WIP] Quill collections support~~ [WIP] Collections support Apr 24, 2017

Add support for cassandra collections of primitive types

dae129a

fwbrasil reviewed Apr 26, 2017

View reviewed changes

mentegy added 4 commits April 26, 2017 19:54

Remove macros for cassandra in favor of MappedCassandraType and provi…

ae11b03

…de implicits for each supported type

Remove generic encoders

9e4dfe1

fix cassandra compilations and tests

021b097

add support for mapped encoding to supported cassandra type

944200f

mosyp commented Apr 26, 2017

View reviewed changes

mentegy added 6 commits April 28, 2017 16:23

Re-think cassandra encoding, implement all major use cases with cassa…

a1d2aef

…ndra collections

Enhance cassandra tests

a49ec87

Refactor sql modules, downgrade array's support from Traversable to S…

95dab6d

…eq (since it more convenient)

Merge branch 'master' into master

cba80c0

- new mirror ctx with arrays in sql base module

37e8fe8

- add tests in sql base module regarding arrays encoding - add decoderUnsafe to MirrorDecoder to reduce ClassTags (removed unnecessary classtags from cassandra collection decoders)

Merge remote-tracking branch 'origin/master'

3ca5aa7

Update README, unify API for custom SQL Arrays encoding

73099db

mosyp force-pushed the master branch from 2eda25f to 73099db Compare May 2, 2017 15:31

Fix tut compilation (comment out mirror arrays example)

b30211e

mosyp force-pushed the master branch from 3eb24a6 to b30211e Compare May 2, 2017 19:32

mosyp changed the title ~~[WIP] Collections support~~ Collections support May 2, 2017

fwbrasil approved these changes May 5, 2017

View reviewed changes

Merge branch 'master' into master

895882c

mxl mentioned this pull request May 7, 2017

[WIP] - Support for Cassandra collections #644

Closed

5 tasks

fwbrasil merged commit c9a4921 into zio:master May 7, 2017

mosyp mentioned this pull request May 7, 2017

Array support for SqlMirrorContext #763

Closed

mxl mentioned this pull request May 8, 2017

Async Postgres Array encoding support #730

Closed

This was referenced May 10, 2017

quill-cassandra - add encoders for collection type columns #309

Closed

Support Cassandra operations on collections #774

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Collections support #758

Collections support #758

mosyp commented Apr 24, 2017 •

edited

Loading

fwbrasil commented Apr 25, 2017

fwbrasil commented Apr 25, 2017

mosyp commented Apr 25, 2017

fwbrasil commented Apr 25, 2017

mxl commented Apr 25, 2017

fwbrasil Apr 26, 2017

mosyp Apr 26, 2017

fwbrasil Apr 26, 2017

mosyp Apr 26, 2017 •

edited

Loading

mosyp left a comment

mosyp Apr 26, 2017

mosyp Apr 26, 2017

mosyp commented Apr 29, 2017

mosyp commented Apr 29, 2017 •

edited

Loading

mosyp commented May 2, 2017

fwbrasil commented May 2, 2017

mosyp commented May 2, 2017

fwbrasil left a comment

mosyp commented May 5, 2017

fwbrasil commented May 5, 2017

mxl commented May 7, 2017


		import scala.reflect.macros.blackbox.{ Context => MacroContext }

		trait CollectionsEncodingMacro {

Collections support #758

Collections support #758

Conversation

mosyp commented Apr 24, 2017 • edited Loading

SQL

Cassandra

Checklist

fwbrasil commented Apr 25, 2017

fwbrasil commented Apr 25, 2017

mosyp commented Apr 25, 2017

fwbrasil commented Apr 25, 2017

mxl commented Apr 25, 2017

fwbrasil Apr 26, 2017

Choose a reason for hiding this comment

mosyp Apr 26, 2017

Choose a reason for hiding this comment

fwbrasil Apr 26, 2017

Choose a reason for hiding this comment

mosyp Apr 26, 2017 • edited Loading

Choose a reason for hiding this comment

mosyp left a comment

Choose a reason for hiding this comment

mosyp Apr 26, 2017

Choose a reason for hiding this comment

mosyp Apr 26, 2017

Choose a reason for hiding this comment

mosyp commented Apr 29, 2017

mosyp commented Apr 29, 2017 • edited Loading

mosyp commented May 2, 2017

fwbrasil commented May 2, 2017

mosyp commented May 2, 2017

fwbrasil left a comment

Choose a reason for hiding this comment

mosyp commented May 5, 2017

fwbrasil commented May 5, 2017

mxl commented May 7, 2017

mosyp commented Apr 24, 2017 •

edited

Loading

mosyp Apr 26, 2017 •

edited

Loading

mosyp commented Apr 29, 2017 •

edited

Loading