Heterogenous and homogenous tuples #364

Author: tonsky

CHANGELOG.md

@@ -1,3 +1,7 @@
+# WIP
+
+- Support heterogeneous (`:db/tupleTypes`) and homogeneous (`:db/tupleType`) tuples #364
+
 # 1.7.8 - Oct 11, 2025
 
 - Skip storage writes for no-op transactions #492 via @jxonas

CLAUDE.md

@@ -0,0 +1,139 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+DataScript is an immutable in-memory database and Datalog query engine implemented in Clojure and ClojureScript. It's designed to run in the browser and JVM, providing a lightweight, ephemeral database with Datalog querying capabilities similar to Datomic.
+
+## Key Commands
+
+### Testing
+
+```bash
+# Run Clojure tests
+./script/test_clj.sh
+# or: clojure -X:dev user/-test-main
+
+# Run ClojureScript tests
+./script/test_cljs.sh
+
+# Run all tests
+./script/test_all.sh
+
+# Run Datomic compatibility tests (requires datomic-free setup)
+./script/test_datomic.sh
+# or: clj -M:datomic
+```
+
+### Benchmarking
+
+```bash
+# Run Clojure benchmarks
+./script/bench_clj.sh
+# or: clojure -A:bench -M -m datascript.bench.datascript
+
+# Run ClojureScript benchmarks
+./script/bench_cljs.sh
+
+# Run Datomic benchmarks (requires datomic-free setup)
+./script/bench_datomic.sh
+
+# Run all benchmarks
+./script/bench_all.sh
+```
+
+### Development
+
+```bash
+# Start a REPL
+./script/repl.sh
+
+# Build ClojureScript release
+lein with-profile test cljsbuild once release
+```
+
+## Architecture Overview
+
+### Core Components
+
+**Database Layer (`datascript.db`):**
+- `Datom` records represent facts as [entity attribute value transaction] tuples
+- `DB` is the immutable database value with three indexes: EAVT, AEVT, AVET
+- Uses persistent sorted sets for index storage (`me.tonsky.persistent-sorted-set`)
+- Transaction IDs start at `0x20000000` (tx0), entity IDs start at 0 (e0)
+- Schema is optional and only needed for special behaviors (cardinality many, unique attributes, ref types)
+
+**Query Engine (`datascript.query`):**
+- Implements Datalog query language
+- Queries are parsed, cached (LRU cache of 100), and executed against relations
+- Supports rules, aggregates, predicates, negation, and multiple data sources
+- `Relation` records store query intermediate results as {attrs tuples}
+- `Context` holds relations, sources, and rules during query execution
+
+**Connection Management (`datascript.conn`):**
+- Connections are atoms holding the latest DB value
+- Access current DB with `@conn`, not `(d/db conn)` as in Datomic
+- Transaction reports include `:db-before`, `:db-after`, `:tx-data`, `:tempids`, `:tx-meta`
+- Supports transaction listeners via `listen!`
+
+**Entity API (`datascript.impl.entity`):**
+- Lazy map-like entities for navigating database
+- Entities cache accessed attributes
+- Reverse references via `_` prefix (`:_ref`)
+- Component references return single entity instead of collection
+
+**Storage (`datascript.storage`):**
+- Protocol-based persistent storage for databases
+- Serializes indexes as persistent sorted sets with addressable chunks
+- Supports custom storage backends via `IStorage` protocol
+
+**Pull API (`datascript.pull-api`, `datascript.pull-parser`):**
+- EQL-style pattern-based entity graph extraction
+- Supports recursive patterns, attribute limits, default values
+
+### Key Differences from Datomic
+
+- Entities use `-1`, `-2` for tempids instead of `#db/id[:db.part/user -100]`
+- No `:db/txInstant` annotations by default
+- Custom query functions in ClojureScript must be passed as source (no `resolve`)
+- Transaction functions called as `[:db.fn/call f args]` where `f` is function reference
+- Additional `:db.fn/retractAttribute` built-in function
+- Schemas are not queryable and attributes don't need pre-declaration
+
+### File Organization
+
+- `src/datascript/` - Core implementation (.cljc files for JVM + JS)
+  - `core.cljc` - Public API entry point
+  - `db.cljc` - Database, datom, and index implementation
+  - `query.cljc` - Datalog query engine
+  - `pull_api.cljc`, `pull_parser.cljc` - Pull API
+  - `storage.clj/.cljs` - Persistent storage (platform-specific)
+  - `conn.cljc` - Connection and transaction handling
+  - `parser.cljc` - Query/rule parsing
+  - `built_ins.cljc` - Built-in query functions and aggregates
+
+- `test/datascript/test/` - Test suite (.cljc files)
+  - Tests use `.cljc` files for cross-platform testing
+  - Acceptance tests demonstrate usage: `test/datascript/test/*.cljc`
+
+- `bench/` - Benchmarking code
+- `bench_datomic/` - Datomic compatibility checks
+- `script/` - Build and test scripts
+
+### Development Notes
+
+- Uses `.cljc` files for cross-platform (JVM + ClojureScript) code
+- Custom macros in `datascript.db` like `defn+` handle CLJS efficiency
+- `*warn-on-reflection*` enabled for JVM performance
+- Storage implementation differs between Clojure (`.clj`) and ClojureScript (`.cljs`)
+- shadow-cljs users must add `:compiler-options {:externs ["datascript/externs.js"]}`
+
+### Aliases (deps.edn)
+
+- `:dev` - Development with REPL tools
+- `:test` - Test dependencies
+- `:bench` - Benchmarking tools (criterium, profiler)
+- `:datomic` - Datomic compatibility testing
+- `:cljs` - ClojureScript compilation
+- `:1.9`, `:1.10`, `:1.11.1`, `:1.12` - Override Clojure versions

docs/tuples.md

@@ -1,7 +1,15 @@
-# Composite tuples
+# Tuples
 
 A tuple is a collection of scalar values, represented in memory as a Clojure vector.
 
+There are three kinds of tuples:
+
+- [Composite tuples](#composite-tuples) (`:db/tupleAttrs`) are derived from other attributes of the same entity and managed by DataScript;
+- [Heterogeneous tuples](#heterogeneous-tuples) (`:db/tupleTypes`) are fixed-length tuples with a declared type per slot, asserted by you;
+- [Homogeneous tuples](#homogeneous-tuples) (`:db/tupleType`) are variable-length tuples with a single declared element type, asserted by you.
+
+# Composite tuples
+
 Composite tuples are applicable:
 
 - when a domain entity has a multi-attribute key;
@@ -100,4 +108,74 @@ If you mark tuple attribute as `:db/unique :db.unique/identity`, you might use i
 ; => {:db/id 4, :a "a", :b "b", :a+b ["a" "b"]}
 ```
 
-Composite tuples are like normal attributes in most cases. Feel free to use them in lookup refs, upserts, queries, index access.
+Composite tuples are like normal attributes in most cases. Feel free to use them in lookup refs, upserts, queries, index access.
+
+# Heterogeneous tuples
+
+Heterogeneous tuples have a fixed length, with a type declared per slot:
+
+```
+(def conn (d/create-conn
+            {:player/location {:db/valueType :db.type/tuple
+                               :db/tupleTypes [:db.type/long :db.type/long]}}))
+```
+
+Unlike composite tuples, you assert and retract them yourself:
+
+```
+(d/transact! conn
+  [{:player/handle "Argent Adept"
+    :player/location [100 0]}])
+```
+
+DataScript validates that the value is a vector of the declared length. `nil` is a legal value for any slot:
+
+```
+(d/transact! conn [[:db/add 1 :player/location [100 nil]]])
+```
+
+Slot types other than `:db.type/ref` are not enforced — DataScript is dynamically typed and treats them as documentation. Datomic’s limits (max 8 elements, strings ≤ 256 chars) are not enforced either. Keep the types in each slot consistent though, otherwise index sorting might fail.
+
+`:db.type/ref` slots are special: entity ids, lookup refs, idents, tempids and `:db/current-tx` in them are resolved the same way ref attribute values are:
+
+```
+(def conn (d/create-conn
+            {:name     {:db/unique :db.unique/identity}
+             :ref+long {:db/valueType :db.type/tuple
+                        :db/tupleTypes [:db.type/ref :db.type/long]}}))
+
+(d/transact! conn
+  [{:db/id -1 :name "Ivan"}
+   [:db/add 2 :ref+long [-1 7]]])           ; tempid, resolves to Ivan’s eid
+
+(d/transact! conn
+  [[:db/add 2 :ref+long [[:name "Ivan"] 8]]]) ; lookup ref works too
+```
+
+Refs inside tuples are resolved on write, but are not full-fledged references otherwise: they don’t show up in reverse index, don’t get retracted when the referenced entity is retracted (the eid stays in the tuple, dangling), can’t be `:db/isComponent`, and entity API returns them as plain numbers, not entities. Same is true in Datomic.
+
+Unlike composite tuples, heterogeneous and homogeneous tuples are not indexed by default. Add `:db/index true` or `:db/unique` if you want AVET access. Unique declared tuples work in upserts and lookup refs, same as composite ones:
+
+```
+(d/entity (d/db conn) [:ref+long [[:name "Ivan"] 8]])
+```
+
+Unlike Datomic, heterogeneous and homogeneous tuples can be `:db.cardinality/many`. Note that in map form multiple values must be wrapped in an extra vector, otherwise a single tuple will be interpreted as multiple values:
+
+```
+(d/transact! conn [{:db/id 1, :locations [[100 0] [200 300]]}]) ; two tuples
+```
+
+# Homogeneous tuples
+
+Homogeneous tuples have variable length and a single type declared for all elements:
+
+```
+(def conn (d/create-conn
+            {:keywords {:db/valueType :db.type/tuple
+                        :db/tupleType :db.type/keyword}}))
+
+(d/transact! conn [[:db/add 1 :keywords [:a :b :c]]])
+```
+
+Everything said about heterogeneous tuples applies: only the vector shape is validated, `:db.type/ref` element type gets full ref resolution in every slot, indexing is opt-in. Note that vectors sort by length first, so `index-range` over different-length tuples is not lexicographic.

script/nrepl.sh

@@ -0,0 +1,5 @@
+#!/bin/bash
+set -o errexit -o nounset -o pipefail
+cd "`dirname $0`/.."
+
+clojure -Sdeps '{:deps {nrepl/nrepl {:mvn/version "1.3.1"}}}' -M:1.12:dev -m nrepl.cmdline