Skip to content

README.md

Latest commit

 

History

History
74 lines (49 loc) · 2.03 KB

README.md

File metadata and controls

74 lines (49 loc) · 2.03 KB

userdata-extractor

An npm package that runs in the browser, to extract contents of user's social data in a meaningful way.

Development

Local setup

# Install dependencies
yarn install

# Run tests
yarn test

Adding a new extractor

  • Copy/paste an existing extractor in the appropriate extractor/<integration>/<html|json>/ directory
  • Initialize the extractor at the bottom of the file, ex:
export const accountInformationJson = new AccountInformationJson(
  config.SERVICE_INSTAGRAM, // integration name
  "account_information/account_information.json", // File regex
  "account_information" // Table name
);
  • Add the extractor to src/extractor/<integration>/index.ts

Adding new sample data

When adding sample data (ex: tests/data/social.zip), please ensure all PII and media has been stripped out. Here are some CLI commands to speed up this process.

# Find and replace text in all json files recursively, starting from current directory
find . -name '*.json' -print0 | xargs -0 perl -pi -e 's/FIND_TEXT/REPLACE_WITH/g'

# Remove all jpg files recursively, starting from current directory
find . -type f -name '*.jpg' -delete

Development notes

  • All project file imports must end in .js, ex: import { Table } from "../models/table/index.js";

Deployment

Create a new release

When we're ready to release a new version of this package, visit Github Releases. Enter the following:

  • Create a tag (ex: v0.0.2)
  • Enter a release title: same as tag (ex: v0.0.2)
  • Enter a description in this format:
# v0.0.2 - July 14, 2022
- Added a feature
- Fixed a bug
  • Publish release. A github action will be triggered that creates the package.

General TODOs

  • Instagram exports either all HTML or all JSON. Need extractors for both
  • Check non-english exports. Perhaps the column names will be different

Current Coverage

Track the currently supported services here.