Skip to content
Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
237 changes: 230 additions & 7 deletions docs/02-tutorials/01-intro/07-storage-set-up.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -4,16 +4,239 @@ title: 6. Set-up Storage
tags: [tutorial]
---

> :construction_worker: writing in progress
The blindnet devkit storage can be handled via two components:

A [Data Access Component](/docs/storage/data-access) is here to transform requests from the [Privacy Computation Engine](/docs/computation) to efficiently execute automated operations on your data storage.
1. a [Data Access Component](/docs/storage/data-access) (DAC)
2. a storage service, with optional help from a devkit storage connector library (currently, only Java is supported, via the [Java Storage Connector library](https://github.com/blindnet-io/storage-connector-java))

You can choose to use the [Capture Component](/docs/data-capture) and directly store the captured data in blindnet's preconfigured storage, or your own storage service.
You can choose to use the [Capture Component](/docs/data-capture) and directly store the captured data in blindnet's preconfigured storage, or your own storage service. You can also directly connect the Data Access Component to your own storage service without using the Capture Component. For the sake of simplicity, we'll choose this option for this tutorial.

You can also directly connect the Data Access Component to your own storage service without using the Capture Component. For see sake of simplicity, we'll choose this option for this tutorial.
For that, we'll set up a dedicated demo storage service in Java, and connect it to the Data Access Component using the [Java Storage Connector library](https://github.com/blindnet-io/storage-connector-java).

For that, we'll set up a dedicated demo storage service in Java, and connect it to the Capture Component using the [Java Storage Connector library](https://github.com/blindnet-io/storage-connector-java).
## Pre-requisites

The following steps require **git**, a **Java Development Kit**, **sbt**, and **docker** to be installed on your machine.

Refer to the [associated section](https://blindnet.dev/docs/tutorials/intro/PCE-set-up#pre-requisites) of the first step of this tutorial for more information.

Additionnally, [Maven](https://maven.apache.org/install.html) is also required in order to build the associated demo.


## Running the Data Access Component Locally

The [Data Access Component](/docs/storage/data-access) acts as a proxy between the [Privacy Computation Engine](/docs/computation) and your storage service, transforming requests from the PCE to efficiently execute automated operations on your data storage.

To run it locally, you first need to clone the associated repository:

```bash
git clone [email protected]:blindnet-io/data-access-component.git
```

Then, build the project using sbt:

```bash
cd data-access-component
sbt assembly
```

And finally, run the docker image:

```bash
docker build -t blindnet/dac .
docker run --network host -it blindnet/dac
```

The following message should be displayed, showing the DAC has been correctly started:

```text
[io-compute-1] INFO org.http4s.blaze.channel.nio1.NIO1SocketServerGroup - Service bound to address /127.0.0.1:8028
[io-compute-1] INFO org.http4s.blaze.server.BlazeServerBuilder -
_ _ _ _ _
| |_| |_| |_ _ __| | | ___
| ' \ _| _| '_ \_ _(_-<
|_||_\__|\__| .__/ |_|/__/
|_|
[io-compute-1] INFO org.http4s.blaze.server.BlazeServerBuilder - http4s v0.23.14 on blaze v0.23.12 started at http://127.0.0.1:8028/
```

Your DAC instance is now ready to receive requests from a PCE.

:::danger Work In Progress

Currently, the PCE doesn't allow to customize the URL of the DAC to use. As a consequence, your local PCE will still send requests to the DAC instance we provide by default in a staging environement, and use the associated demo storage service.

:::

## Storage Connector Demo

### Set Up

For the purpose of this tutorial, we provide a demo implementation of a storage service.

Code for this demo can be fund in the `example` directory in the librarie's repository: [github.com/blindnet-io/storage-connector-java > `example`](https://github.com/blindnet-io/storage-connector-java/tree/develop/example).
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Code for this demo can be fund in the `example` directory in the librarie's repository: [github.com/blindnet-io/storage-connector-java > `example`](https://github.com/blindnet-io/storage-connector-java/tree/develop/example).
Code for this demo can be found in the `example` directory in the library repository: [github.com/blindnet-io/storage-connector-java > `example`](https://github.com/blindnet-io/storage-connector-java/tree/develop/example).


Clone this resporitoy:
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Clone this resporitoy:
Clone this repository:


```bash
git clone
```

Then, build the library and the demo project using Maven:

```bash
cd data-access-component
mvn package
```

Finally, go to this example directory to run the associated docker image:

```bash
cd example/
docker build -t blindnet/connector-demo .
docker run -e BN_CONNECTOR_ENDPOINT=http://localhost:8028 --network host -it blindnet/connector-demo
```

### How It Works

This demo directly uses the Java Storage Connector library to define mappings between [selectors](/docs/computation/configuration#selectors) defined in the PCE and the actual storage, leading to the following code in the `example/src/main/java/io/blindnet/storageconnector/example/Main.java` file:

```java
StorageConnector.create()
.setDataRequestHandler(new MappingRequestHandler.Builder<User>()
.setSubjectMapper(Database.users::findByEmail)
.addSelectorType("CONTACT.EMAIL", User::email)
.addSelectorType("NAME", User::fullName)
.addSelectorTypeBinary("OTHER-DATA.PROOF", u -> BinaryData.fromArray(u.proof()))
.build())
.start();
```

Let's explore together how it works so you can reproduce it in your project.

#### Setting up a StorageConnector

The `StorageConnector` class is the center of the Java Storage Connector library, instanciated via the `create()` method.

```java
import io.blindnet.storageconnector.StorageConnector;

StorageConnector sc = StorageConnector.create();
```

This class allows you to establish a transparent connection with the PCE to answer in the most efficient way every request the PCE sends.
This connection is only active after calling the `start()` method.

```java
StorageConnector.create().start();
```

Yet, the default implementation of `StorageConnector` doesn't make any assumption about your selectors or storage.
Meaning the `StorageConnector` in the above example will only have a `DefaultDataRequestHandler`, which denies all incoming requests.

We therefore need to define a custom `DataRequestHandler` to define how the storage connector should answer such requests.

#### Defining a Data Request Handler

For this, we use the `setDataRequestHandler()` method to replace the default data request handler with the provided implementation of the `DataRequestHandler` interface.

In this example (and in most cases), we only need to define the mapping between the selectors defined in the PCE configuration and our ORM classes, using the `MappingRequestHandler`'s `Builder`.

In the example project, we start from an ORM defined for the `User` beforehand in `example/src/main/java/io/blindnet/storageconnector/example/User.java`:

```java
public record User(String firstName, String lastName, String email, byte[] proof) {
public String fullName() {
return firstName() + " " + lastName();
}

public interface Dao {
@SqlUpdate("CREATE TABLE users (first_name varchar, last_name varchar, email varchar, proof varbinary)")
void createTable();

@SqlUpdate("INSERT INTO users (first_name, last_name, email, proof) VALUES (?, ?, ?, ?)")
void insert(String firstName, String lastName, String email, byte[] proof);

@SqlQuery("SELECT * FROM users WHERE email=?")
User findByEmail(String email);
}

public static class Mapper implements RowMapper<User> {
@Override
public User map(ResultSet rs, StatementContext ctx) throws SQLException {
return new User(
rs.getString("first_name"),
rs.getString("last_name"),
rs.getString("email"),
rs.getBytes("proof")
);
}
}
}
```

:::info

This example uses `jdbi`, but the Storage Connector library is compatible with any ORM or ORM-less storage solution, even project-specific ones.

:::

Leading to the following builder:

```java
import io.blindnet.storageconnector.handlers.mapping.MappingRequestHandler;

Builder<User> userMappingBuilder = new MappingRequestHandler.Builder<User>();
```

In the end (after configuring the mapping as shown in the following step), we'll only neeed to call the `Builder.build()` method to generate the instance of `MappingRequestHandler`.

```java
StorageConnector storageConnector = StorageConnector.create();

Builder<User> userMappingBuilder = new MappingRequestHandler.Builder<User>();

// configuration

MappingRequestHandler userRequestHandler = userMappingBuilder.build();

storageConnector.setDataRequestHandler(userRequestHandler).start();
```

#### Mapping Simple Entries

First, we start by associating our mapper with a specific stored data.
For that, we use the `Builder.setSubjectMapper()` method, which only needs a function returning the desired object based on a given selector (here, the user's email).
Here, we can directly use a method from our ORM definition.

```java
userMappingBuilder.setSubjectMapper(Database.users::findByEmail)
```

Then, we need to define the associations between PCE selectors and stored data.
As we already refered to the user's email here, we'll start with the associated selector, `CONTACT.EMAIL`, and map it to the equivalent attribute, `User.email`.


```java
userMappingBuilder.addSelectorType("CONTACT.EMAIL", User::email)
```

And we'll proceed in the same exact way for the user's name:

```java
userMappingBuilder.addSelectorType("NAME", User::fullName)
```

##### Mapping Binary Data

The Java Storage Connector library also provides all the methods you need to map binary data in order to handle them in the more efficient way.

Here, the demonstration app also captures a proof of identity document (pdf, image, etc.) for every user, associating it with the custom `OTHER-DATA.PROOF` selector.
The `BinaryData` class will help you to represent such data easily in order to map it to a selector using the `addSelectorTypeBinary()` method.

In this example, as the `proof` entry is stored as a `byte[]`, we use the `fromArray` method:

```java
userMappingBuilder.addSelectorTypeBinary("OTHER-DATA.PROOF", u -> BinaryData.fromArray(u.proof()))
```

<!-- FIXME: set up a dedicated project and detail development and release steps -->

Code for this demo stage service can be found in the `example` directory in the librarie's repository: [github.com/blindnet-io/storage-connector-java > `example`](https://github.com/blindnet-io/storage-connector-java/tree/develop/example).