Initial commit

Author: kmilodenisglez

.env.example

@@ -0,0 +1,24 @@
+# Local directories where inputs and outputs are found
+# When running on the refinement service, files will be mounted to the /input and /output directory of the container
+INPUT_DIR=input
+OUTPUT_DIR=output
+
+# This key is derived from the user file's original encryption key, automatically injected into the container by the refinement service
+# When developing locally, use any value for testing.
+REFINEMENT_ENCRYPTION_KEY=0x1234
+
+# Schema configuration
+SCHEMA_NAME=Google Drive Analytics
+SCHEMA_VERSION=0.0.1
+SCHEMA_DESCRIPTION=Schema for the Google Drive DLP, representing some basic analytics of the Google user
+SCHEMA_DIALECT=sqlite
+
+# IPFS configuration
+# Required if using https://pinata.cloud (IPFS pinning service)
+PINATA_API_KEY=your_pinata_api_key_here
+PINATA_API_SECRET=your_pinata_api_secret_here
+
+# Public IPFS gateway URL for accessing uploaded files
+# Recommended to use your own dedicated IPFS gateway to avoid congestion / rate limiting
+# Example: "https://ipfs.my-dao.org/ipfs" (Note: won't work for third-party files)
+IPFS_GATEWAY_URL=https://gateway.pinata.cloud/ipfs

.github/workflows/build-and-release.yml

@@ -0,0 +1,72 @@
+name: Build and Release
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+permissions:
+  contents: write
+
+jobs:
+  build-and-release:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+
+      - name: Set up Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: '3.11'
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v2
+
+      - name: Build Docker image
+        uses: docker/build-push-action@v4
+        with:
+          context: .
+          load: true
+          tags: |
+            refiner:${{ github.run_number }}
+            refiner:latest
+          cache-from: type=gha
+          cache-to: type=gha,mode=max
+
+      - name: Export image to file
+        run: |
+          docker save refiner:latest | gzip > refiner-${{ github.run_number }}.tar.gz
+
+      - name: Generate release body
+        run: |
+          echo "Image SHA256: $(sha256sum refiner-${{ github.run_number }}.tar.gz | cut -d' ' -f1)" >> release_body.txt
+
+      - name: Upload image
+        uses: actions/upload-artifact@v4
+        with:
+          name: refiner-image
+          path: refiner-${{ github.run_number }}.tar.gz
+
+      - name: Create Release and Upload Assets
+        uses: softprops/action-gh-release@v1
+        if: github.event_name == 'push' && github.ref == 'refs/heads/main'
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        with:
+          tag_name: v${{ github.run_number }}
+          name: Release v${{ github.run_number }}
+          body_path: release_body.txt
+          draft: false
+          prerelease: false
+          files: |
+            ./refiner-${{ github.run_number }}.tar.gz
+
+      - name: Log build result
+        if: always()
+        run: |
+          if [ ${{ job.status }} == "success" ]; then
+            echo "Build and release completed successfully"
+          else
+            echo "Build and release failed"
+          fi

.gitignore

@@ -0,0 +1,12 @@
+*.pem
+*.tar.gz
+__pycache__/
+*.pyc
+
+/input/user.json
+/output/*
+!/output/.gitkeep
+.env
+
+.idea/
+.DS_Store

Dockerfile

@@ -0,0 +1,10 @@
+FROM python:3.12-slim
+
+WORKDIR /app
+
+COPY . /app
+
+# Install any needed packages specified in requirements.txt
+RUN pip install --no-cache-dir -r requirements.txt
+
+CMD ["python", "-m", "refiner"]

LICENSE

@@ -0,0 +1,8 @@
+The MIT License (MIT)
+Copyright © 2024 Corsali, Inc. dba Vana
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

README.md

@@ -0,0 +1,107 @@
+# Vana Data Refinement Template
+
+This repository serves as a template for creating Dockerized *data refinement instructions* that transform raw user data into normalized (and potentially anonymized) SQLite-compatible databases, so data in Vana can be querying by Vana's Query Engine.
+
+## Overview
+
+Here is an overview of the data refinement process on Vana.
+
+![How Refinements Work](https://files.readme.io/25f8f6a4c8e785a72105d6eb012d09449f63ab5682d1f385120eaf5af871f9a2-image.png "How Refinements Work")
+
+1. DLPs upload user-contributed data through their UI, and run proof-of-contribution against it. Afterwards, they call the refinement service to refine this data point.
+1. The refinement service downloads the file from the Data Registry and decrypts it.
+1. The refinement container, containing the instructions for data refinement (this repo), is executed
+   1. The decrypted data is mounted to the container's `/input` directory
+   1. The raw data points are transformed against a normalized SQLite database schema (specifically libSQL, a modern fork of SQLite)
+   1. Optionally, PII (Personally Identifiable Information) is removed or masked
+   1. The refined data is symmetrically encrypted with a derivative of the original file encryption key
+1. The encrypted refined data is uploaded and pinned to a DLP-owned IPFS
+1. The IPFS CID is written to the refinement container's `/output` directory
+1. The CID of the file is added as a refinement under the original file in the Data Registry
+1. Vana's Query Engine indexes that data point, aggregating it with all other data points of a given refiner. This allows SQL queries to run against all data of a particular refiner (schema).
+
+## Project Structure
+
+- `refiner/`: Contains the main refinement logic
+    - `refine.py`: Core refinement implementation
+    - `config.py`: Environment variables and settings needed to run your refinement
+    - `__main__.py`: Entry point for the refinement execution
+    - `models/`: Pydantic and SQLAlchemy data models (for both unrefined and refined data)
+    - `transformer/`: Data transformation logic
+    - `utils/`: Utility functions for encryption, IPFS upload, etc.
+- `input/`: Contains raw data files to be refined
+- `output/`: Contains refined outputs:
+    - `schema.json`: Database schema definition
+    - `db.libsql`: SQLite database file
+    - `db.libsql.pgp`: Encrypted database file
+- `Dockerfile`: Defines the container image for the refinement task
+- `requirements.txt`: Python package dependencies
+
+## Getting Started
+
+1. Fork this repository
+2. Copy `.env.example` to `.env` and modify the values to match your environment
+3. Update the schemas in `refiner/models/` to define your raw and normalized data models
+4. Modify the refinement logic in `refiner/transformer/` to match your data structure
+5. If needed, modify `refiner/refiner.py` with your file(s) that need to be refined
+6. Build and test your refinement container
+
+### Environment variables
+
+Copy `.env.example` to `.env` and configure the following variables:
+
+```dotenv
+# Local directories where inputs and outputs are found
+# When running on the refinement service, files will be mounted to the /input and /output directory of the container
+INPUT_DIR=input
+OUTPUT_DIR=output
+
+# This key is derived from the user file's original encryption key, automatically injected into the container by the refinement service
+# When developing locally, any string can be used here for testing
+REFINEMENT_ENCRYPTION_KEY=0x1234
+
+# Schema configuration
+SCHEMA_NAME=Google Drive Analytics
+SCHEMA_VERSION=0.0.1
+SCHEMA_DESCRIPTION=Schema for the Google Drive DLP, representing some basic analytics of the Google user
+SCHEMA_DIALECT=sqlite
+
+# IPFS configuration
+# Required if using https://pinata.cloud (IPFS pinning service)
+PINATA_API_KEY=your_pinata_api_key_here
+PINATA_API_SECRET=your_pinata_api_secret_here
+
+# Public IPFS gateway URL for accessing uploaded files
+# Recommended to use own dedicated IPFS gateway to avoid congestion / rate limiting
+# Example: "https://ipfs.my-dao.org/ipfs" (Note: won't work for third-party files)
+IPFS_GATEWAY_URL=https://gateway.pinata.cloud/ipfs
+```
+
+## Local Development
+
+To run the refinement locally for testing:
+
+```bash
+# With Python
+pip install --no-cache-dir -r requirements.txt
+python -m refiner
+
+# Or with Docker
+docker build -t refiner .
+docker run \
+  --rm \
+  --volume $(pwd)/input:/input \
+  --volume $(pwd)/output:/output \
+  --env PINATA_API_KEY=your_key \
+  --env PINATA_API_SECRET=your_secret \
+  refiner
+```
+
+## Contributing
+
+If you have suggestions for improving this template, please open an issue or submit a pull request.
+
+## License
+
+[MIT License](LICENSE)
+

input/user.zip

@@ -0,0 +1,50 @@
+import json
+import logging
+import os
+import sys
+import traceback
+import zipfile
+
+from refiner.refine import Refiner
+from refiner.config import settings
+
+logging.basicConfig(level=logging.INFO, format='%(message)s')
+
+
+def run() -> None:
+    """Transform all input files into the database."""
+    input_files_exist = os.path.isdir(settings.INPUT_DIR) and bool(os.listdir(settings.INPUT_DIR))
+
+    if not input_files_exist:
+        raise FileNotFoundError(f"No input files found in {settings.INPUT_DIR}")
+    extract_input()
+
+    refiner = Refiner()
+    output = refiner.transform()
+    
+    output_path = os.path.join(settings.OUTPUT_DIR, "output.json")
+    with open(output_path, 'w') as f:
+        json.dump(output.model_dump(), f, indent=2)    
+    logging.info(f"Data transformation complete: {output}")
+
+
+def extract_input() -> None:
+    """
+    If the input directory contains any zip files, extract them
+    :return:
+    """
+    for input_filename in os.listdir(settings.INPUT_DIR):
+        input_file = os.path.join(settings.INPUT_DIR, input_filename)
+
+        if zipfile.is_zipfile(input_file):
+            with zipfile.ZipFile(input_file, 'r') as zip_ref:
+                zip_ref.extractall(settings.INPUT_DIR)
+
+
+if __name__ == "__main__":
+    try:
+        run()
+    except Exception as e:
+        logging.error(f"Error during data transformation: {e}")
+        traceback.print_exc()
+        sys.exit(1)