Skip to content

Commit 67acc56

Browse files
[Chore] Monorepo and Drift (#50)
* move sqlite_async to packages * monorepo updates * move drift package * fix actions * more test fixes * Clean up pubspec and README * Clean up readme * Add detailed usage instructions * Remove older packages links * Fix drift repository url * Fix pana analyze packages * Revert pana script * Fix melos pana analyze --------- Co-authored-by: Mugi Khan <[email protected]>
1 parent ea80324 commit 67acc56

File tree

106 files changed

+1877
-172
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

106 files changed

+1877
-172
lines changed

.github/workflows/release.yml

+4-4
Original file line numberDiff line numberDiff line change
@@ -21,11 +21,11 @@ jobs:
2121

2222
- uses: dart-lang/setup-dart@v1
2323

24-
- name: Install dependencies
25-
run: dart pub get
24+
- name: Install Melos
25+
run: dart pub global activate melos
2626

27-
- name: Compile WebWorker
28-
run: dart compile js -o assets/db_worker.js -O0 lib/src/web/worker/worker.dart
27+
- name: Install dependencies
28+
run: melos prepare
2929

3030
- name: Set tag name
3131
id: tag

.github/workflows/test.yaml

+13-12
Original file line numberDiff line numberDiff line change
@@ -12,18 +12,20 @@ jobs:
1212
- uses: actions/checkout@v3
1313
- uses: dart-lang/setup-dart@v1
1414

15+
- name: Install Melos
16+
run: dart pub global activate melos
1517
- name: Install dependencies
16-
run: dart pub get
18+
run: melos prepare
1719
- name: Check formatting
18-
run: dart format --output=none --set-exit-if-changed .
20+
run: melos format:check:packages
1921
- name: Lint
20-
run: dart analyze
22+
run: melos analyze:packages
2123
- name: Publish dry-run
22-
run: dart pub publish --dry-run
24+
run: melos publish --dry-run --yes
2325
- name: Check publish score
2426
run: |
2527
dart pub global activate pana
26-
dart pub global run pana --no-warning --exit-code-threshold 0
28+
melos analyze:packages:pana
2729
2830
test:
2931
runs-on: ubuntu-latest
@@ -51,18 +53,17 @@ jobs:
5153
with:
5254
sdk: ${{ matrix.dart_sdk }}
5355

56+
- name: Install Melos
57+
run: dart pub global activate melos
58+
5459
- name: Install dependencies
55-
run: dart pub get
60+
run: melos prepare
5661

5762
- name: Install SQLite
5863
run: |
5964
./scripts/install_sqlite.sh ${{ matrix.sqlite_version }} ${{ matrix.sqlite_url }}
60-
mkdir -p assets && curl -LJ https://github.com/simolus3/sqlite3.dart/releases/download/sqlite3-2.4.3/sqlite3.wasm -o assets/sqlite3.wasm
61-
62-
- name: Compile WebWorker
63-
run: dart compile js -o assets/db_worker.js -O0 lib/src/web/worker/worker.dart
6465
6566
- name: Run Tests
6667
run: |
67-
export LD_LIBRARY_PATH=./sqlite-autoconf-${{ matrix.sqlite_version }}/.libs
68-
dart test -p vm,chrome
68+
export LD_LIBRARY_PATH=$(pwd)/sqlite-autoconf-${{ matrix.sqlite_version }}/.libs
69+
melos test

.gitignore

+1
Original file line numberDiff line numberDiff line change
@@ -17,5 +17,6 @@ assets
1717
test-db
1818
sqlite-autoconf-*
1919
doc
20+
*.iml
2021

2122
build

DEVELOPING.md

-8
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,5 @@
11
# Developing Instructions
22

3-
## Testing
4-
5-
Running tests for the `web` platform requires some preparation to be executed. The `sqlite3.wasm` and `db_worker.js` files need to be available in the Git ignored `./assets` folder.
6-
7-
See the [test action](./.github/workflows/test.yaml) for the latest steps.
8-
9-
On your local machine run the commands from the `Install SQLite`, `Compile WebWorker` and `Run Tests` steps.
10-
113
## Releases
124

135
Web worker files are compiled and uploaded to draft Github releases whenever tags matching `v*` are pushed. These tags are created when versioning. Releases should be manually finalized and published when releasing new package versions.

README.md

+7-92
Original file line numberDiff line numberDiff line change
@@ -2,100 +2,15 @@
22

33
High-performance asynchronous interface for SQLite on Dart & Flutter.
44

5-
[SQLite](https://www.sqlite.org/) is small, fast, has a lot of built-in functionality, and works
6-
great as an in-app database. However, SQLite is designed for many different use cases, and requires
7-
some configuration for optimal performance as an in-app database.
8-
9-
The [sqlite3](https://pub.dev/packages/sqlite3) Dart bindings are great for direct synchronous access
10-
to a SQLite database, but leaves the configuration up to the developer.
11-
12-
This library wraps the bindings and configures the database with a good set of defaults, with
13-
all database calls being asynchronous to avoid blocking the UI, while still providing direct SQL
14-
query access.
15-
16-
## Features
17-
18-
- All operations are asynchronous by default - does not block the main isolate.
19-
- Watch a query to automatically re-run on changes to the underlying data.
20-
- Concurrent transactions supported by default - one write transaction and many multiple read transactions.
21-
- Uses WAL mode for fast writes and running read transactions concurrently with a write transaction.
22-
- Direct synchronous access in an isolate is supported for performance-sensitive use cases.
23-
- Automatically convert query args to JSON where applicable, making JSON1 operations simple.
24-
- Direct SQL queries - no wrapper classes or code generation required.
25-
26-
See this [blog post](https://www.powersync.co/blog/sqlite-optimizations-for-ultra-high-performance),
27-
explaining why these features are important for using SQLite.
28-
29-
## Installation
30-
31-
```sh
32-
dart pub add sqlite_async
33-
```
34-
35-
For flutter applications, additionally add `sqlite3_flutter_libs` to include the native SQLite
36-
library.
37-
38-
For other platforms, see the [sqlite3 package docs](https://pub.dev/packages/sqlite3#supported-platforms).
39-
40-
Web is currently not supported.
5+
| package | build | pub | likes | popularity | pub points |
6+
|----------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------| ------- | ------- |
7+
| sqlite_async | [![build](https://github.com/powersync-ja/sqlite_async.dart/actions/workflows/test.yaml/badge.svg?branch=main)](https://github.com/powersync-ja/sqlite_async.dart/actions?query=workflow%3Atest) | [![pub package](https://img.shields.io/pub/v/sqlite_async.svg)](https://pub.dev/packages/sqlite_async) | [![likes](https://img.shields.io/pub/likes/powersync?logo=dart)](https://pub.dev/packages/sqlite_async/score) | [![popularity](https://img.shields.io/pub/popularity/sqlite_async?logo=dart)](https://pub.dev/packages/sqlite_async/score) | [![pub points](https://img.shields.io/pub/points/sqlite_async?logo=dart)](https://pub.dev/packages/sqlite_async/score)
8+
| drift_sqlite_async | [![build](https://github.com/powersync-ja/sqlite_async.dart/actions/workflows/test.yaml/badge.svg?branch=main)](https://github.com/powersync-ja/sqlite_async/actions?query=workflow%3Atest) | [![pub package](https://img.shields.io/pub/v/drift_sqlite_async.svg)](https://pub.dev/packages/drift_sqlite_async) | [![likes](https://img.shields.io/pub/likes/drift_sqlite_async?logo=dart)](https://pub.dev/packages/drift_sqlite_async/score) | [![popularity](https://img.shields.io/pub/popularity/drift_sqlite_async?logo=dart)](https://pub.dev/packages/drift_sqlite_async/score) | [![pub points](https://img.shields.io/pub/points/drift_sqlite_async?logo=dart)](https://pub.dev/packages/drift_sqlite_async/score)
419

4210
## Getting Started
4311

44-
```dart
45-
import 'package:sqlite_async/sqlite_async.dart';
46-
47-
final migrations = SqliteMigrations()
48-
..add(SqliteMigration(1, (tx) async {
49-
await tx.execute(
50-
'CREATE TABLE test_data(id INTEGER PRIMARY KEY AUTOINCREMENT, data TEXT)');
51-
}));
52-
53-
void main() async {
54-
final db = SqliteDatabase(path: 'test.db');
55-
await migrations.migrate(db);
56-
57-
// Use execute() or executeBatch() for INSERT/UPDATE/DELETE statements
58-
await db.executeBatch('INSERT INTO test_data(data) values(?)', [
59-
['Test1'],
60-
['Test2']
61-
]);
62-
63-
// Use getAll(), get() or getOptional() for SELECT statements
64-
var results = await db.getAll('SELECT * FROM test_data');
65-
print('Results: $results');
66-
67-
// Combine multiple statements into a single write transaction for:
68-
// 1. Atomic persistence (all updates are either applied or rolled back).
69-
// 2. Improved throughput.
70-
await db.writeTransaction((tx) async {
71-
await tx.execute('INSERT INTO test_data(data) values(?)', ['Test3']);
72-
await tx.execute('INSERT INTO test_data(data) values(?)', ['Test4']);
73-
});
74-
75-
await db.close();
76-
}
77-
```
78-
79-
# Web
80-
81-
Note: Web support is currently in Beta.
82-
83-
Web support requires Sqlite3 WASM and web worker Javascript files to be accessible via configurable URIs.
84-
85-
Default URIs are shown in the example below. URIs only need to be specified if they differ from default values.
86-
87-
The compiled web worker files can be found in our Github [releases](https://github.com/powersync-ja/sqlite_async.dart/releases)
88-
The `sqlite3.wasm` asset can be found [here](https://github.com/simolus3/sqlite3.dart/releases)
89-
90-
Setup
91-
92-
```Dart
93-
import 'package:sqlite_async/sqlite_async.dart';
12+
This monorepo uses [melos](https://melos.invertase.dev/) to handle command and package management.
9413

95-
final db = SqliteDatabase(
96-
path: 'test.db',
97-
options: SqliteOptions(
98-
webSqliteOptions: WebSqliteOptions(
99-
wasmUri: 'sqlite3.wasm', workerUri: 'db_worker.js')));
14+
To configure the monorepo for development run `melos prepare` after cloning.
10015

101-
```
16+
For detailed usage, check out the inner [sqlite_async](https://github.com/powersync-ja/sqlite_async.dart/tree/main/packages/sqlite_async) and [drift_sqlite_async](https://github.com/powersync-ja/sqlite_async.dart/tree/main/packages/drift_sqlite_async) packages.

melos.yaml

+46
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
name: sqlite_async_monorepo
2+
3+
packages:
4+
- packages/**
5+
6+
scripts:
7+
prepare: melos bootstrap && melos prepare:compile:webworker && melos prepare:sqlite:wasm
8+
9+
prepare:compile:webworker:
10+
description: Compile Javascript web worker distributable
11+
run: dart compile js -o assets/db_worker.js -O0 packages/sqlite_async/lib/src/web/worker/worker.dart
12+
13+
prepare:sqlite:wasm:
14+
description: Download SQLite3 WASM binary
15+
run: dart run ./scripts/sqlite3_wasm_download.dart
16+
17+
format:
18+
description: Format Dart code.
19+
run: dart format .
20+
21+
format:check:packages:
22+
description: Check formatting of Dart code in packages.
23+
run: dart format --output none --set-exit-if-changed packages
24+
25+
analyze:packages:
26+
description: Analyze Dart code in packages.
27+
run: dart analyze packages --fatal-infos
28+
29+
analyze:packages:pana:
30+
description: Analyze Dart packages with Pana
31+
exec: dart pub global run pana --no-warning --exit-code-threshold 0
32+
packageFilters:
33+
noPrivate: true
34+
35+
test:
36+
description: Run tests in a specific package.
37+
run: dart test -p chrome,vm
38+
exec:
39+
concurrency: 1
40+
packageFilters:
41+
dirExists:
42+
- test
43+
# This tells Melos tests to ignore env variables passed to tests from `melos run test`
44+
# as they could change the behaviour of how tests filter packages.
45+
env:
46+
MELOS_TEST: true
+4
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
2+
## 0.1.0-alpha.1
3+
4+
Initial release.

packages/drift_sqlite_async/LICENSE

+21
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2024 Journey Mobile, Inc.
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

packages/drift_sqlite_async/README.md

+62
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
# drift_sqlite_async
2+
3+
`drift_sqlite_async` allows using drift on an sqlite_async database - the APIs from both can be seamlessly used together in the same application.
4+
5+
Supported functionality:
6+
1. All queries including select, insert, update, delete.
7+
2. Transactions and nested transactions.
8+
3. Table updates are propagated between sqlite_async and Drift - watching queries works using either API.
9+
4. Select queries can run concurrently with writes and other select statements.
10+
11+
12+
## Usage
13+
14+
Use `SqliteAsyncDriftConnection` to create a DatabaseConnection / QueryExecutor for Drift from the sqlite_async SqliteDatabase:
15+
16+
```dart
17+
@DriftDatabase(tables: [TodoItems])
18+
class AppDatabase extends _$AppDatabase {
19+
AppDatabase(SqliteConnection db) : super(SqliteAsyncDriftConnection(db));
20+
21+
@override
22+
int get schemaVersion => 1;
23+
}
24+
25+
Future<void> main() async {
26+
// The sqlite_async db
27+
final db = SqliteDatabase(path: 'example.db');
28+
// The Drift db
29+
final appdb = AppDatabase(db);
30+
}
31+
```
32+
33+
A full example is in the `examples/` folder.
34+
35+
For details on table definitions and using the database, see the [Drift documentation](https://drift.simonbinder.eu/).
36+
37+
## Transactions and concurrency
38+
39+
sqlite_async uses WAL mode and multiple read connections by default, and this
40+
is also exposed when using the database with Drift.
41+
42+
Drift's transactions use sqlite_async's `writeTransaction`. The same locks are used
43+
for both, preventing conflicts.
44+
45+
Read-only transactions are not currently supported in Drift.
46+
47+
Drift's nested transactions are supported, implemented using SAVEPOINT.
48+
49+
Select statements in Drift use read operations (`getAll()`) in sqlite_async,
50+
and can run concurrently with writes.
51+
52+
## Update notifications
53+
54+
sqlite_async uses SQLite's update_hook to detect changes for watching queries,
55+
and will automatically pick up changes made using Drift. This also includes any updates from custom queries in Drift.
56+
57+
Changes from sqlite_async are automatically propagated to Drift when using SqliteAsyncDriftConnection.
58+
These events are only sent while no write transaction is active.
59+
60+
Within Drift's transactions, Drift's own update notifications will still apply for watching queries within that transaction.
61+
62+
Note: There is a possibility of events being duplicated. This should not have a significant impact on most applications.
+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
targets:
2+
$default:
3+
builders:
4+
drift_dev:
5+
options:
6+
fatal_warnings: true
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
import 'package:drift/drift.dart';
2+
import 'package:drift_sqlite_async/drift_sqlite_async.dart';
3+
import 'package:sqlite_async/sqlite_async.dart';
4+
5+
part 'main.g.dart';
6+
7+
class TodoItems extends Table {
8+
@override
9+
String get tableName => 'todos';
10+
11+
IntColumn get id => integer().autoIncrement()();
12+
TextColumn get description => text()();
13+
}
14+
15+
@DriftDatabase(tables: [TodoItems])
16+
class AppDatabase extends _$AppDatabase {
17+
AppDatabase(SqliteConnection db) : super(SqliteAsyncDriftConnection(db));
18+
19+
@override
20+
int get schemaVersion => 1;
21+
}
22+
23+
Future<void> main() async {
24+
final db = SqliteDatabase(path: 'example.db');
25+
26+
// Example where the schema is managed manually
27+
await db.execute(
28+
'CREATE TABLE IF NOT EXISTS todos(id integer primary key, description text)');
29+
30+
final appdb = AppDatabase(db);
31+
32+
// Watch a query on the Drift database
33+
appdb.select(appdb.todoItems).watch().listen((todos) {
34+
print('Todos: $todos');
35+
});
36+
37+
// Insert using the Drift database
38+
await appdb
39+
.into(appdb.todoItems)
40+
.insert(TodoItemsCompanion.insert(description: 'Test Drift'));
41+
42+
// Insert using the sqlite_async database
43+
await db.execute('INSERT INTO todos(description) VALUES(?)', ['Test Direct']);
44+
45+
await Future.delayed(const Duration(milliseconds: 100));
46+
47+
await appdb.close();
48+
await db.close();
49+
}

0 commit comments

Comments
 (0)