Code

Support for Argo backend#19

Code

Merged

zhanyuanucb merged 60 commits intopetuum:masterpetuum/pirlib:masterfrom Nilabhra:masterNilabhra/pirlib:masterCopy head branch name to clipboardDec 7, 2022

+902-32

Commits

Commits on Nov 10, 2022

• add: subparser for argoize

  Nilabhra

  committed
  47eda1c

  Copy full SHA for 47eda1c

• add: Nikhil's changes.

  Nilabhra

  committed
  828e6d5

  Copy full SHA for 828e6d5

• feat: generating workflow name using output file name.

  Nilabhra

  committed
  5c72277

  Copy full SHA for 5c72277

• feat: function to create Argo template from node.

  Nilabhra

  committed
  c394308

  Copy full SHA for c394308

• chore: added docstring.

  Nilabhra

  committed
  09f1c1c

  Copy full SHA for 09f1c1c

• chore: added comments.

  Nilabhra

  committed
  e42223c

  Copy full SHA for e42223c

• feat: setting the env vars before generation of the argo yaml.

  Nilabhra

  committed
  35cb1be

  Copy full SHA for 35cb1be

• feat: function to generate NFS volume specs for K8s.

  Nilabhra

  committed
  0b35916

  Copy full SHA for 0b35916

• feat: added support for mounting NFS volumes.

  Nilabhra

  committed
  c1ff95a

  Copy full SHA for c1ff95a

• fix: using absolute path for the NFS mount paths.

  Nilabhra

  committed
  d579f24

  Copy full SHA for d579f24

• fix: typo.

  Nilabhra

  committed
  64396a3

  Copy full SHA for 64396a3

• feat: function for generating Argo templates for PIRlib Graph objects.

  Nilabhra

  committed
  1a7c0a9

  Copy full SHA for 1a7c0a9

• chore: added comments.

  Nilabhra

  committed
  b901d3b

  Copy full SHA for b901d3b

• feat: enabled creation of DAG tasks.

  Nilabhra

  committed
  6052ae4

  Copy full SHA for 6052ae4

• chore: added Argo specific refactoring.

  Nilabhra

  committed
  fcc6768

  Copy full SHA for fcc6768

• chore: refactored argo task names.

  Nilabhra

  committed
  6316c87

  Copy full SHA for 6316c87

• fix: files cannot be mounted as volumes in Argo/K8s.

  Nilabhra

  committed
  8a9b5fd

  Copy full SHA for 8a9b5fd

• chore: removed redundant module.

  Nilabhra

  committed
  95fd3dd

  Copy full SHA for 95fd3dd

• chore: debugging code.

  Nilabhra

  committed
  d339da2

  Copy full SHA for d339da2

• chore: optimized docker image generation.

  Nilabhra

  committed
  1bc1cea

  Copy full SHA for 1bc1cea

• add: dockerignore file to prevent copying of unnecessary files.

  Nilabhra

  committed
  99912ef

  Copy full SHA for 99912ef

• fix: missing volume mount.

  Nilabhra

  committed
  d552227

  Copy full SHA for d552227

Commits on Nov 11, 2022

• chore: reverted to original formatting.

  Nilabhra

  committed
  a1c26c9

  Copy full SHA for a1c26c9

• chore: removed spurious code.

  Nilabhra

  committed
  46b3ca1

  Copy full SHA for 46b3ca1

• chore: increased black linelength to 100.

  Nilabhra

  committed
  ee19634

  Copy full SHA for ee19634

Commits on Nov 14, 2022

• Multistep pipeline support for Argo (#1)

  Show description for 31c542a
  Nilabhra

  authored
  31c542a

  Copy full SHA for 31c542a

Commits on Nov 16, 2022

• fix: type annotation.

  Nilabhra

  committed
  2a60ba2

  Copy full SHA for 2a60ba2

• add: WIP files for one step execution of the wiki_parse pipeline.

  Nilabhra

  committed
  49d1000

  Copy full SHA for 49d1000

Commits on Nov 17, 2022

• fix: type annotation.

  Nilabhra

  committed
  01d33ff

  Copy full SHA for 01d33ff

• add: WIP files for one step execution of the wiki_parse pipeline.

  Nilabhra

  committed
  557d82b

  Copy full SHA for 557d82b

Commits on Nov 18, 2022

• feat: wiki_parse example running as a single step.

  Nilabhra

  committed
  332f345

  Copy full SHA for 332f345

• fix: merge conflicts.

  Nilabhra

  committed
  cd51aab

  Copy full SHA for cd51aab

Commits on Nov 22, 2022

• feat: PIRlib pipeline for parsing the sample Wikipedia data.

  Nilabhra

  committed
  5045d4d

  Copy full SHA for 5045d4d

• chore: renamed input directory.

  Nilabhra

  committed
  8995742

  Copy full SHA for 8995742

• add: input files.

  Nilabhra

  committed
  2b8c8f6

  Copy full SHA for 2b8c8f6

• add: shell file to download the full dbpedia dump.

  Nilabhra

  committed
  d695521

  Copy full SHA for d695521

• add: scripts to run the PIRlib pipeline on the entire dbpedia dump.

  Nilabhra

  committed
  9ad7382

  Copy full SHA for 9ad7382

• chore: code cleanup.

  Nilabhra

  committed
  c9fce68

  Copy full SHA for c9fce68

• chore: removed unneeded imports.

  Nilabhra

  committed
  ead051c

  Copy full SHA for ead051c

Commits on Nov 23, 2022

• feat: enabled parallel downloading of Wikipedia dumps.

  Nilabhra

  committed
  367330a

  Copy full SHA for 367330a

• add: readme containing usage instructions.

  Nilabhra

  committed
  355d6c1

  Copy full SHA for 355d6c1

• Merge pull request #2 from Nilabhra/forte-wiki-parser

  Show description for 4e64fa4
  Nilabhra

  authored
  4e64fa4

  Copy full SHA for 4e64fa4

• Revert "Forte wiki_parser"

  Nilabhra

  authored
  f082adb

  Copy full SHA for f082adb

• Merge pull request #3 from Nilabhra/revert-2-forte-wiki-parser

  Show description for 7fa241f
  Nilabhra

  authored
  7fa241f

  Copy full SHA for 7fa241f

Commits on Nov 24, 2022

• Merge pull request #4 from Nilabhra/forte

  Show description for 66dea30
  Nilabhra

  authored
  66dea30

  Copy full SHA for 66dea30

• Revert "add: wiki_parser example from Forte"

  Nilabhra

  authored
  ce78767

  Copy full SHA for ce78767

• Merge pull request #5 from Nilabhra/revert-4-forte

  Show description for 793c048
  Nilabhra

  authored
  793c048

  Copy full SHA for 793c048

Commits on Nov 29, 2022

• feat: raising exception if required env vars are not defined.

  Nilabhra

  committed
  f584c95

  Copy full SHA for f584c95

• chore: refactored inp_name to inp_id.

  Nilabhra

  committed
  6dbb27a

  Copy full SHA for 6dbb27a

• chore: reformatted shell command.

  Nilabhra

  committed
  2f00467

  Copy full SHA for 2f00467

Commits on Dec 5, 2022

• chore: reverted to inclusion of setting PYTHONPATH in the image.

  Nilabhra

  committed
  333b4d2

  Copy full SHA for 333b4d2

• feat: prepending env of python to .

  Nilabhra

  committed
  c17156a

  Copy full SHA for c17156a

• feat: changed base image to continuumio/miniconda3.

  Nilabhra

  committed
  1ad74fe

  Copy full SHA for 1ad74fe

• add: generated YAML files.

  Nilabhra

  committed
  535fa13

  Copy full SHA for 535fa13

• chore: incoporated PR feedback.

  Nilabhra

  committed
  c4510c8

  Copy full SHA for c4510c8

• fix: conflicts.

  Nilabhra

  committed
  d23ca89

  Copy full SHA for d23ca89

• add: Usage instructions.

  Nilabhra

  committed
  2d65765

  Copy full SHA for 2d65765

• Merge branch 'argo-multistep'

  Nilabhra

  committed
  679bbf1

  Copy full SHA for 679bbf1

Commits on Dec 7, 2022

• chore: added Argo example to the main README.rst.

  Nilabhra

  committed
  bc9d923

  Copy full SHA for bc9d923

• chore: updated documentation.

  Nilabhra

  committed
  99386f6

  Copy full SHA for 99386f6