Skip to content

Replace typesystem with Spark-/Hive- types #367

@okennedy

Description

@okennedy

Possibly required for #361
Would subsume #333

Mimir has a much cruder internal type system than Spark. In addition to lacking collection types, there's a lot of capabilities (e.g., integer precision) that Mimir just does not support at the moment. Spark already has a well-developed type system, and it would be fantastic if we could replace this part of Mimir with the corresponding parts from Spark.

  • Replace Mimir's Type with Spark's DataType
  • Update or replace Typechecker with corresponding parts of Spark (not sure where this lives... perhaps the Catalyst analyzer
  • Tack on support for unit-based types (Maybe use User-Defined Types?)
  • Move to serialization using foo.json and foo.as[DataType] (make sure to keep track of everywhere this happens)
  • Add an upgrade path to migrate old, encoded types to the new form

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions