update README.md for 0.2.0 (#290)

* update README.md for 0.2.0

things to. verify:
- [ ] how to upgrade rill developer
- [ ]

* Adding example project command to cli

* Adding README.md to .prettierignore

Co-authored-by: Aditya Hegde <[email protected]>

.prettierignore

@@ -25,3 +25,4 @@ log
 .prettierignore
 Dockerfile
 Duckdb-Dockerfile
+README.md

README.md

@@ -23,12 +23,6 @@ Nodejs version 16+ installed locally: https://nodejs.org/en/download/. Check you
 node -v
 ```
 
-Clone this repository to your local machine:
-
-```
-git clone https://github.com/rilldata/rill-developer.git
-```
-
 On Ubuntu, you'll also need to make sure you have `g++` installed in order to compile DuckDB from source during the installation steps below (please note that compiling DuckDB may take a while):
 
 ```
@@ -37,33 +31,26 @@ sudo apt install g++
 
 # Install Locally
 
-Change directories to the local Rill Developer repository
-
-```
-cd /path/to/rill-developer
-```
+To install locally, you can use `npm` to globally install Rill Developer. This will give you a CLI to start the server. 
 
-Run npm to install dependencies and build the application.
-Part of this step involves compiling DuckDB which can be time consuming to complete (it may take approximately five minutes or more, depending on your machine).
-Please be patient.
+Note: this install command involves compiling DuckDB which can be time consuming to complete (it may take approximately five minutes or more, depending on your machine). Please be patient!
 
 ```
-npm install
-npm run build
+npm install -g @rilldata/rill
 ```
 
 # Quick Start Example
 
 If you are looking for a fast way to get started you can run our quick start example script. This script initializes a project, downloads an [OpenSky Network dataset](https://zenodo.org/record/6325961#.YjDFvhDMI0Q), and imports the data. The Rill Developer UI will be available at http://localhost:8080.
 
 ```
-bash scripts/example-project.sh
+rill initialize-example-project
 ```
 
 If you close the example project and want to restart it, you can do so by running:
 
 ```
-npm run cli --silent -- start --project ../rill-developer-example
+rill start
 ```
 
 # Creating Your Own Project
@@ -75,25 +62,25 @@ If you want to go beyond this example, you can also create a project using your
 Initialize your project in the Rill Developer directory.
 
 ```
-npm run cli --silent -- init
+rill init
 ```
 
 ## Import Your Data
 
 Import datasets of interest into the Rill Developer [duckDB](https://duckdb.org/docs/sql/introduction) database to make them available. We currently support .parquet, .csv, and .tsv.
 
 ```
-npm run cli --silent -- import-table /path/to/data_1.parquet
-npm run cli --silent -- import-table /path/to/data_2.csv
-npm run cli --silent -- import-table /path/to/data_3.tsv
+rill import-table /path/to/data_1.parquet
+rill import-table /path/to/data_2.csv
+rill import-table /path/to/data_3.tsv
 ```
 
 ## Start Your Project
 
 Start the User Interface to interact with your imported tables and revisit projects you have created.
 
 ```
-npm run cli --silent -- start
+rill start
 ```
 
 The Rill Developer UI will be available at http://localhost:8080.
@@ -107,37 +94,35 @@ Rill Developer is powered by duckDB. Please visit their documentation for insigh
 Rill Developer will be evolving quickly! If you want an updated version, you can pull in the latest changes and rebuild the application. Once you have rebuilt the application you can restart your project to see the new experience.
 
 ```
-git pull origin main
-npm run build
-npm run cli --silent -- start
+npm install -g @rilldata/rill
 ```
 
 # Helpful Hints
 
-You can specify a new project folder by including the --project option.
+Rill works best if you have `cd`ed into the project directory, since it assumes that you are in a project directory already. But you can also specify a new project folder by including the --project option.
 
 ```
-npm run cli --silent -- init --project /path/to/a/new/project
-npm run cli --silent -- import-table /path/to/data_1.parquet --project /path/to/a/new/project
-npm run cli --silent -- start --project /path/to/a/new/project
+rill init --project /path/to/a/new/project
+rill import-table /path/to/data_1.parquet --project /path/to/a/new/project
+rill start --project /path/to/a/new/project
 ```
 
 By default the table name will be a sanitized version of the dataset file name. You can specify a name using the --name option.
 
 ```
-npm run cli --silent -- import-table  /path/to/data_1.parquet --name my_table
+rill import-table  /path/to/data_1.parquet --name my_table
 ```
 
 If you have added a table to Rill Developer that you want to drop, you can do so using the --drop-table option.
 
 ```
-npm run cli --silent -- drop-table my_table
+rill drop-table my_table
 ```
 
 If you have a dataset that is delimited by a character other than a comma or tab, you can use the --delimiter option. DuckDB can also attempt to automatically detect the delimiter, so it is not strictly necessary.
 
 ```
-npm run cli --silent -- import-table /path/to/data_4.txt --delimiter "|"
+rill import-table /path/to/data_4.txt --delimiter "|"
 ```
 
 You can connect to an existing duckdb database by passing --db with path to the db file.
@@ -146,20 +131,20 @@ Similarly, any changes made by Rill Developer will modify the database.
 Make sure to have only one connection open to the database, otherwise there will be some unexpected issues.
 
 ```
-npm run cli --silent -- init --db /path/to/duckdb/file
+rill init --db /path/to/duckdb/file
 ```
 
 You can also copy over the database so that there are no conflicts and overrides to the source.
 Pass --copy along with --db to achieve this.
 
 ```
-npm run cli --silent -- init --db /path/to/duckdb/file --copy
+rill init --db /path/to/duckdb/file --copy
 ```
 
 If you would like to see information on all the available CLI commands, you can use the help option.
 
 ```
-npm run cli --silent -- --help
+rill --help
 ```
 
 # Troubleshooting

package.json

@@ -1,6 +1,6 @@
 {
   "name": "@rilldata/rill",
-  "version": "0.1.0",
+  "version": "0.2.0",
   "type": "module",
   "scripts": {
     "dev": "ts-node-dev --quiet --project tsconfig.node.json src/dev.ts",

src/cli/DataModelerCliCommand.ts

@@ -92,10 +92,7 @@ export abstract class DataModelerCliCommand {
     await dataModelerService.init();
   }
 
-  protected async run(
-    cliRunArgs: CliRunArgs,
-    ...args: Array<any>
-  ): Promise<void> {
+  public async run(cliRunArgs: CliRunArgs, ...args: Array<any>): Promise<void> {
     await this.init(cliRunArgs);
     await this.sendActions(...args);
     await this.teardown();

src/cli/ExampleProjectCommand.ts

@@ -0,0 +1,60 @@
+import { DataModelerCliCommand } from "$cli/DataModelerCliCommand";
+import { Command } from "commander";
+import { execSync } from "node:child_process";
+import { ImportTableCommand } from "$cli/ImportTableCommand";
+import { InitCommand } from "$cli/InitCommand";
+import { StartCommand } from "$cli/StartCommand";
+import { ExpressServer } from "$server/ExpressServer";
+
+export class ExampleProjectCommand extends DataModelerCliCommand {
+  public getCommand(): Command {
+    return this.applyCommonSettings(
+      new Command("initialize-example-project"),
+      "Initialize example project."
+    ).action((opts, command: Command) => {
+      let { project } = command.optsWithGlobals();
+      if (!project) project = process.cwd() + "/rill-developer-example";
+
+      return this.createExampleProject(project);
+    });
+  }
+
+  protected async sendActions(): Promise<void> {
+    // no-op
+  }
+
+  public async createExampleProject(project: string): Promise<void> {
+    console.log(`Initializing the project example project ${project} ...`);
+    await new InitCommand().createProjectAndRun({}, project);
+
+    console.log("Downloading dataset for example project...");
+    execSync(
+      `curl -s http://pkg.rilldata.com/rill-developer-example/data/flightlist.zip ` +
+        `--output ${project}/flightlist.zip`,
+      { stdio: "inherit" }
+    );
+    execSync(`unzip ${project}/flightlist.zip ` + `-d ${project}/`, {
+      stdio: "inherit",
+    });
+
+    console.log("Importing example dataset into the project...");
+    await new ImportTableCommand().run(
+      {
+        projectPath: project,
+      },
+      `${project}/data/flightlist_2022_02.csv`,
+      {}
+    );
+
+    return new StartCommand().run({
+      projectPath: project,
+      shouldInitState: false,
+      shouldSkipDatabase: false,
+      profileWithUpdate: true,
+    });
+  }
+
+  protected async teardown(): Promise<void> {
+    // do not teardown as this will have a perpetual server
+  }
+}

src/cli/InitCommand.ts

@@ -26,24 +26,29 @@ export class InitCommand extends DataModelerCliCommand {
         const { project } = command.optsWithGlobals();
 
         const projectPath = project ?? process.cwd();
-        InitCommand.makeDirectoryIfNotExists(projectPath);
-        this.alreadyInitialised = existsSync(`${projectPath}/state`);
 
-        if (
-          !this.alreadyInitialised &&
-          !InitCommand.verifyDuckDbPath(opts.db, opts.copy, projectPath)
-        ) {
-          console.log(`Failed to initialize project under ${projectPath}`);
-          return;
-        }
-
-        return this.run({
-          projectPath,
-          duckDbPath: opts.copy ? undefined : opts.db,
-        });
+        return this.createProjectAndRun(opts, projectPath);
       });
   }
 
+  public createProjectAndRun(opts, projectPath: string) {
+    InitCommand.makeDirectoryIfNotExists(projectPath);
+    this.alreadyInitialised = existsSync(`${projectPath}/state`);
+
+    if (
+      !this.alreadyInitialised &&
+      !InitCommand.verifyDuckDbPath(opts.db, opts.copy, projectPath)
+    ) {
+      console.log(`Failed to initialize project under ${projectPath}`);
+      return;
+    }
+
+    return this.run({
+      projectPath,
+      duckDbPath: opts.copy ? undefined : opts.db,
+    });
+  }
+
   protected async sendActions(): Promise<void> {
     if (!this.alreadyInitialised) {
       // add a single model.

src/cli/data-modeler-cli.ts

@@ -6,6 +6,7 @@ import { ImportTableCommand } from "$cli/ImportTableCommand";
 import { StartCommand } from "$cli/StartCommand";
 import { InfoCommand } from "$cli/InfoCommand";
 import { DropTableCommand } from "$cli/DropTableCommand";
+import { ExampleProjectCommand } from "$cli/ExampleProjectCommand";
 
 const program = new Command();
 
@@ -27,6 +28,7 @@ program
   StartCommand,
   DropTableCommand,
   InfoCommand,
+  ExampleProjectCommand,
 ].forEach((CommandClass) =>
   program.addCommand(new CommandClass().getCommand())
 );

src/common/database-service/DuckDBClient.ts

@@ -20,10 +20,18 @@ export class DuckDBClient {
   protected onCallback: () => void;
   protected offCallback: () => void;
 
-  public constructor(private readonly databaseConfig: DatabaseConfig) {}
+  // this is a singleton class because
+  // duckdb doesn't work well with multiple connections to same db from same process
+  // if we ever need to have different connections modify this to have a map of database to instance
+  private static instance: DuckDBClient;
+  private constructor(private readonly databaseConfig: DatabaseConfig) {}
+  public static getInstance(databaseConfig: DatabaseConfig) {
+    if (!this.instance) this.instance = new DuckDBClient(databaseConfig);
+    return this.instance;
+  }
 
   public async init(): Promise<void> {
-    if (this.databaseConfig.skipDatabase) return;
+    if (this.databaseConfig.skipDatabase || this.db) return;
     // we can later on swap this over to WASM and update data loader
     this.db = new duckdb.Database(this.databaseConfig.databaseName);
     this.db.exec("PRAGMA threads=32;PRAGMA log_query_path='./log';");

src/server/serverFactory.ts

@@ -36,7 +36,7 @@ try {
 }
 
 export function databaseServiceFactory(config: RootConfig) {
-  const duckDbClient = new DuckDBClient(config.database);
+  const duckDbClient = DuckDBClient.getInstance(config.database);
   const databaseDataLoaderActions = new DatabaseDataLoaderActions(
     config.database,
     duckDbClient

test/utils/modify-db.ts

@@ -3,7 +3,7 @@ import { DuckDBClient } from "$common/database-service/DuckDBClient";
 import { DatabaseConfig } from "$common/config/DatabaseConfig";
 
 (async () => {
-  const duckDbClient = new DuckDBClient(
+  const duckDbClient = DuckDBClient.getInstance(
     new DatabaseConfig({ databaseName: process.argv[2] })
   );
   await duckDbClient.init();