Code

Openai compatible gauntlet#1017

Code

Draft

bmosaicml wants to merge 167 commits intomainmosaicml/llm-foundry:mainfrom openai_compatible_gauntletmosaicml/llm-foundry:openai_compatible_gauntletCopy head branch name to clipboard

Commits

Commits on Jan 27, 2024

• start

  bmosaicml

  committed
  cd18e74

  Copy full SHA for cd18e74

Commits on Feb 2, 2024

• still need to migrate fixtures

  bmosaicml

  committed
  1fffbad

  Copy full SHA for 1fffbad

• Merge branch 'main' into migrate_subclasses_to_foundry

  bmosaicml

  committed
  5a6e81c

  Copy full SHA for 5a6e81c

• wip onboarding tests

  bmosaicml

  committed
  4aac81e

  Copy full SHA for 4aac81e

• still workin'

  bmosaicml

  committed
  946a4af

  Copy full SHA for 946a4af

• still wip

  bmosaicml

  committed
  289ca55

  Copy full SHA for 289ca55

• maybe done; test out on mcli now

  bmosaicml

  committed
  3696f8d

  Copy full SHA for 3696f8d

• mcli

  bmosaicml

  committed
  a20877d

  Copy full SHA for a20877d

• remove calibration error

  bmosaicml

  committed
  53da3ea

  Copy full SHA for 53da3ea

Commits on Feb 7, 2024

• merge

  bmosaicml

  committed
  16b8e32

  Copy full SHA for 16b8e32

• migration

  bmosaicml

  committed
  a90766e

  Copy full SHA for a90766e

Commits on Feb 8, 2024

• migration

  bmosaicml

  committed
  72ce793

  Copy full SHA for 72ce793

Commits on Feb 9, 2024

• Merge branch 'migrate_subclasses_to_foundry' of github.com:mosaicml/llm-foundry into migrate_subclasses_to_foundry

  bmosaicml

  committed
  667bdec

  Copy full SHA for 667bdec

• full migration

  bmosaicml

  committed
  ceff0c4

  Copy full SHA for ceff0c4

Commits on Feb 12, 2024

• precommit

  bmosaicml

  committed
  5bb06cc

  Copy full SHA for 5bb06cc

• fix

  bmosaicml

  committed
  fe83828

  Copy full SHA for fe83828

• fix pytests

  bmosaicml

  committed
  b54a12b

  Copy full SHA for b54a12b

Commits on Feb 16, 2024

• refactor QA

  bmosaicml

  committed
  71e8391

  Copy full SHA for 71e8391

Commits on Feb 22, 2024

• update

  bmosaicml

  committed
  414153e

  Copy full SHA for 414153e

Commits on Feb 23, 2024

• restore

  bmosaicml

  committed
  a3f5a31

  Copy full SHA for a3f5a31

• Merge branch 'main' into migrate_subclasses_to_foundry

  bmosaicml

  authored
  820069a

  Copy full SHA for 820069a

• add

  bmosaicml

  committed
  4a1cd79

  Copy full SHA for 4a1cd79

• Merge branch 'migrate_subclasses_to_foundry' of github.com:mosaicml/llm-foundry into migrate_subclasses_to_foundry

  bmosaicml

  committed
  d265979

  Copy full SHA for d265979

Commits on Feb 26, 2024

• wip

  bmosaicml

  committed
  6cbaad4

  Copy full SHA for 6cbaad4

• Merge branch 'main' into migrate_subclasses_to_foundry

  bmosaicml

  committed
  ddfd7b5

  Copy full SHA for ddfd7b5

• fix

  bmosaicml

  committed
  71f77e3

  Copy full SHA for 71f77e3

Commits on Feb 27, 2024

• wip

  bmosaicml

  committed
  cb3725b

  Copy full SHA for cb3725b

• update readme

  bmosaicml

  committed
  5135152

  Copy full SHA for 5135152

• Merge branch 'main' into migrate_subclasses_to_foundry

  bmosaicml

  authored
  18bae17

  Copy full SHA for 18bae17

• final pyright

  bmosaicml

  committed
  c6162dd

  Copy full SHA for c6162dd

• Merge branch 'migrate_subclasses_to_foundry' of github.com:mosaicml/llm-foundry into migrate_subclasses_to_foundry

  bmosaicml

  committed
  25d431e

  Copy full SHA for 25d431e

• done

  bmosaicml

  committed
  f1b334d

  Copy full SHA for f1b334d

• pass prelimiter into ALL the ICL task datasets

  eitanturok

  authored
  c4ed644

  Copy full SHA for c4ed644

Commits on Mar 4, 2024

• Merge branch 'main' into migrate_subclasses_to_foundry

  bmosaicml

  authored
  2516c24

  Copy full SHA for 2516c24

• allow QA task name stil lfor backward compatibility

  bmosaicml

  committed
  f213a40

  Copy full SHA for f213a40

• Merge branch 'migrate_subclasses_to_foundry' of github.com:mosaicml/llm-foundry into migrate_subclasses_to_foundry

  bmosaicml

  committed
  35fd2f1

  Copy full SHA for 35fd2f1

Commits on Mar 5, 2024

• fix

  bmosaicml

  committed
  d570e5d

  Copy full SHA for d570e5d

• fix test

  bmosaicml

  committed
  a5cd308

  Copy full SHA for a5cd308

• Merge branch 'main' into migrate_subclasses_to_foundry

  bmosaicml

  authored
  0fb37cd

  Copy full SHA for 0fb37cd

• add generation length

  bmosaicml

  committed
  901fc69

  Copy full SHA for 901fc69

• Merge branch 'migrate_subclasses_to_foundry' of github.com:mosaicml/llm-foundry into migrate_subclasses_to_foundry

  bmosaicml

  committed
  a313499

  Copy full SHA for a313499

• remove max_new_tokens

  bmosaicml

  committed
  df19c0d

  Copy full SHA for df19c0d

Commits on Mar 6, 2024

• fix cpu trsts

  bmosaicml

  committed
  54bb4c7

  Copy full SHA for 54bb4c7

• Merge branch 'main' into migrate_subclasses_to_foundry

  bmosaicml

  authored
  9ebeaa0

  Copy full SHA for 9ebeaa0

• Merge branch 'main' into migrate_subclasses_to_foundry

  maxisawesome

  authored
  ca9816c

  Copy full SHA for ca9816c

Commits on Mar 7, 2024

• try and fix lm eval test

  bmosaicml

  committed
  b9d6cd1

  Copy full SHA for b9d6cd1

• Merge branch 'migrate_subclasses_to_foundry' of github.com:mosaicml/llm-foundry into migrate_subclasses_to_foundry

  bmosaicml

  committed
  691ab20

  Copy full SHA for 691ab20

• temp disable lm task eval test

  bmosaicml

  committed
  c207cd9

  Copy full SHA for c207cd9

• add data

  bmosaicml

  committed
  0662d59

  Copy full SHA for 0662d59

• Merge branch 'main' into openai_compatible_gauntlet

  bmosaicml

  committed
  7b06a9f

  Copy full SHA for 7b06a9f

• test

  bmosaicml

  committed
  ee440b5

  Copy full SHA for ee440b5

• Merge branch 'main' into openai_compatible_gauntlet

  bmosaicml

  committed
  6083891

  Copy full SHA for 6083891

Commits on Mar 8, 2024

• finish

  bmosaicml

  committed
  d6c1ae7

  Copy full SHA for d6c1ae7

• fix test?

  bmosaicml

  committed
  c85813b

  Copy full SHA for c85813b

• push dataset

  bmosaicml

  committed
  110eaf4

  Copy full SHA for 110eaf4

• fix

  bmosaicml

  committed
  4642578

  Copy full SHA for 4642578

• finish

  bmosaicml

  committed
  b6beb49

  Copy full SHA for b6beb49

Commits on Mar 11, 2024

• fix bug

  bmosaicml

  committed
  c3ef3a7

  Copy full SHA for c3ef3a7

• fix tet

  bmosaicml

  committed
  08ef908

  Copy full SHA for 08ef908

• remove human eval

  bmosaicml

  committed
  f1e50dc

  Copy full SHA for f1e50dc

• finish

  bmosaicml

  committed
  aca0e63

  Copy full SHA for aca0e63

Commits on Mar 12, 2024

• fix

  bmosaicml

  committed
  30fcedd

  Copy full SHA for 30fcedd

• modify mlci

  bmosaicml

  committed
  877e5eb

  Copy full SHA for 877e5eb

Commits on Mar 13, 2024

• Merge branch 'main' into migrate_subclasses_to_foundry

  maxisawesome

  authored
  59daa26

  Copy full SHA for 59daa26

• Update scripts/eval/README.md

  Show description for 4217a78
  

  bmosaicml

  and

  dakinggg

  authored
  4217a78

  Copy full SHA for 4217a78

• fix comments

  bmosaicml

  committed
  6f597a9

  Copy full SHA for 6f597a9

• Merge branch 'migrate_subclasses_to_foundry' of github.com:mosaicml/llm-foundry into migrate_subclasses_to_foundry

  bmosaicml

  committed
  8c6e622

  Copy full SHA for 8c6e622

Commits on Mar 14, 2024

• fix bug with seq len

  bmosaicml

  committed
  f387a73

  Copy full SHA for f387a73

• Merge branch 'main' into migrate_subclasses_to_foundry

  bmosaicml

  authored
  cbfa3da

  Copy full SHA for cbfa3da

• restore mcli

  bmosaicml

  committed
  2f405d9

  Copy full SHA for 2f405d9

• Merge branch 'migrate_subclasses_to_foundry' of github.com:mosaicml/llm-foundry into migrate_subclasses_to_foundry

  bmosaicml

  committed
  76e600a

  Copy full SHA for 76e600a

Commits on Mar 15, 2024

• Merge branch 'main' into migrate_subclasses_to_foundry

  bmosaicml

  authored
  898928e

  Copy full SHA for 898928e

• fix

  bmosaicml

  committed
  d1c1b33

  Copy full SHA for d1c1b33

• add output logging

  bmosaicml

  committed
  dd95f2d

  Copy full SHA for dd95f2d

• fix generation length

  bmosaicml

  committed
  4ad2b2c

  Copy full SHA for 4ad2b2c

• fix generation length

  bmosaicml

  committed
  132a179

  Copy full SHA for 132a179

• add example dep deployment eval

  bmosaicml

  committed
  650de02

  Copy full SHA for 650de02

• finish

  bmosaicml

  committed
  8178589

  Copy full SHA for 8178589

• update dep deployment mcli

  bmosaicml

  committed
  dbfebba

  Copy full SHA for dbfebba

Commits on Mar 16, 2024

• update dep deployment mcli

  bmosaicml

  committed
  72cffea

  Copy full SHA for 72cffea

Commits on Mar 18, 2024

• Merge branch 'main' into migrate_subclasses_to_foundry

  bmosaicml

  authored
  07fb59e

  Copy full SHA for 07fb59e

• Merge branch 'main' into migrate_subclasses_to_foundry

  bmosaicml

  authored
  587971f

  Copy full SHA for 587971f

• change temp

  bmosaicml

  committed
  eed61da

  Copy full SHA for eed61da

• change mcli

  bmosaicml

  committed
  6abc07b

  Copy full SHA for 6abc07b

• dep deployment is kinda jank

  bmosaicml

  committed
  cb49cf6

  Copy full SHA for cb49cf6

• finish merge

  bmosaicml

  committed
  0dd6897

  Copy full SHA for 0dd6897

Commits on Mar 19, 2024

• add init file

  bmosaicml

  committed
  0684aef

  Copy full SHA for 0684aef

• add init file

  bmosaicml

  committed
  8278298

  Copy full SHA for 8278298

• finish

  bmosaicml

  committed
  d651ed5

  Copy full SHA for d651ed5

• finish

  bmosaicml

  committed
  b83761b

  Copy full SHA for b83761b

Commits on Mar 20, 2024

• modify formatting to match eleuther

  bmosaicml

  committed
  53b812c

  Copy full SHA for 53b812c

• modify long context

  bmosaicml

  committed
  79660c7

  Copy full SHA for 79660c7

• add custom api keys

  bmosaicml

  committed
  8904d27

  Copy full SHA for 8904d27

• add custom api keys

  bmosaicml

  committed
  8b3328b

  Copy full SHA for 8b3328b

• fix api key

  bmosaicml

  committed
  1ca727b

  Copy full SHA for 1ca727b

• fix api key

  bmosaicml

  committed
  a585bdf

  Copy full SHA for a585bdf

• fix api key

  bmosaicml

  committed
  5129dd4

  Copy full SHA for 5129dd4

• fix api key

  bmosaicml

  committed
  8e1159c

  Copy full SHA for 8e1159c

Commits on Mar 21, 2024

• fix yaml

  bmosaicml

  committed
  4d83efa

  Copy full SHA for 4d83efa

• add 16k tasks

  bmosaicml

  committed
  c425390

  Copy full SHA for c425390

• add 16k tasks

  bmosaicml

  committed
  6f580ef

  Copy full SHA for 6f580ef

• add 16k tasks

  bmosaicml

  committed
  4b2aa6a

  Copy full SHA for 4b2aa6a

• add 16k tasks

  bmosaicml

  committed
  cc7bc1e

  Copy full SHA for cc7bc1e

Commits on Mar 22, 2024

• add api key to config

  bmosaicml

  committed
  1455833

  Copy full SHA for 1455833

• fix

  bmosaicml

  committed
  b0dbbb3

  Copy full SHA for b0dbbb3

• fix ctor

  bmosaicml

  committed
  c70c7f7

  Copy full SHA for c70c7f7

Commits on Mar 25, 2024

• gemini chat

  bmosaicml

  committed
  09017ff

  Copy full SHA for 09017ff

• gemini

  bmosaicml

  committed
  c601eba

  Copy full SHA for c601eba

• restore tasks

  bmosaicml

  committed
  1f04738

  Copy full SHA for 1f04738

• fault tolerance for content filter

  bmosaicml

  committed
  47bb064

  Copy full SHA for 47bb064

• fix

  bmosaicml

  committed
  e307dd7

  Copy full SHA for e307dd7

Commits on Mar 29, 2024

• Merge branch 'main' into migrate_subclasses_to_foundry

  bmosaicml

  authored
  18efa86

  Copy full SHA for 18efa86

Commits on Apr 1, 2024

• merge

  bmosaicml

  committed
  7faeb78

  Copy full SHA for 7faeb78

• fix builder

  bmosaicml

  committed
  343e115

  Copy full SHA for 343e115

• add deprecation warning

  bmosaicml

  committed
  bf6231e

  Copy full SHA for bf6231e

• Merge branch 'main' into migrate_subclasses_to_foundry

  bmosaicml

  authored
  58859a3

  Copy full SHA for 58859a3

• add deprecation warning

  bmosaicml

  committed
  501d4cc

  Copy full SHA for 501d4cc

• Merge branch 'migrate_subclasses_to_foundry' of github.com:mosaicml/llm-foundry into migrate_subclasses_to_foundry

  bmosaicml

  committed
  22f6759

  Copy full SHA for 22f6759

• merge

  bmosaicml

  committed
  414467a

  Copy full SHA for 414467a

• finish merge

  bmosaicml

  committed
  b60066e

  Copy full SHA for b60066e

• merge

  bmosaicml

  committed
  65fbbed

  Copy full SHA for 65fbbed

• add logging necessities to nlp.py

  maxisawesome

  committed
  5696f09

  Copy full SHA for 5696f09

• add attention_mask test update

  maxisawesome

  committed
  91a2b18

  Copy full SHA for 91a2b18

• fix generation_length in tests

  maxisawesome

  committed
  79877ee

  Copy full SHA for 79877ee

• Merge branch 'main' into migrate_subclasses_to_foundry

  maxisawesome

  authored
  9c50795

  Copy full SHA for 9c50795

• fix bug

  maxisawesome

  committed
  eac919a

  Copy full SHA for eac919a

• Merge branch 'migrate_subclasses_to_foundry' of github.com:mosaicml/llm-foundry into migrate_subclasses_to_foundry

  maxisawesome

  committed
  965c20c

  Copy full SHA for 965c20c

Commits on Apr 2, 2024

• rm conflicts

  maxisawesome

  committed
  6db05bd

  Copy full SHA for 6db05bd

• rm extra arg

  maxisawesome

  committed
  ce3e19c

  Copy full SHA for ce3e19c

• Merge branch 'main' into migrate_subclasses_to_foundry

  bmosaicml

  authored
  57e902a

  Copy full SHA for 57e902a

• Merge branch 'main' into migrate_subclasses_to_foundry

  maxisawesome

  authored
  7c0996f

  Copy full SHA for 7c0996f

• Merge branch 'main' into migrate_subclasses_to_foundry

  maxisawesome

  authored
  599695c

  Copy full SHA for 599695c

Commits on Apr 3, 2024

• restore yamls

  bmosaicml

  committed
  e10086f

  Copy full SHA for e10086f

• merge

  bmosaicml

  committed
  0b5da4b

  Copy full SHA for 0b5da4b

• fix?

  bmosaicml

  committed
  c889fd2

  Copy full SHA for c889fd2

• fix yaml; testing

  bmosaicml

  committed
  1b2f6bc

  Copy full SHA for 1b2f6bc

• fix api key

  maxisawesome

  committed
  7afee25

  Copy full SHA for 7afee25

• add formatting

  bmosaicml

  committed
  5720676

  Copy full SHA for 5720676

• Merge branch 'openai_compatible_gauntlet' of github.com:mosaicml/llm-foundry into openai_compatible_gauntlet

  bmosaicml

  committed
  c303b91

  Copy full SHA for c303b91

Commits on Apr 4, 2024

• Merge branch 'main' into migrate_subclasses_to_foundry

  bmosaicml

  authored
  a60ef1d

  Copy full SHA for a60ef1d

• Merge branch 'main' into migrate_subclasses_to_foundry

  maxisawesome

  authored
  d78d783

  Copy full SHA for d78d783

Commits on Apr 9, 2024

• Merge branch 'main' into migrate_subclasses_to_foundry

  maxisawesome

  authored
  1ddf194

  Copy full SHA for 1ddf194

Commits on Apr 10, 2024

• Merge branch 'main' into openai_compatible_gauntlet

  bmosaicml

  authored
  78ac2d9

  Copy full SHA for 78ac2d9

• working

  bmosaicml

  committed
  34c967b

  Copy full SHA for 34c967b

• fix typos

  bmosaicml

  committed
  d5aebc8

  Copy full SHA for d5aebc8

• Merge branch 'migrate_subclasses_to_foundry' of github.com:mosaicml/llm-foundry into migrate_subclasses_to_foundry

  bmosaicml

  committed
  d7272b1

  Copy full SHA for d7272b1

• add deprecation warning for code

  maxisawesome

  committed
  a5082b0

  Copy full SHA for a5082b0

• Merge branch 'migrate_subclasses_to_foundry' of github.com:mosaicml/llm-foundry into migrate_subclasses_to_foundry

  maxisawesome

  committed
  3c8ac56

  Copy full SHA for 3c8ac56

• pyright wip

  maxisawesome

  committed
  642ad40

  Copy full SHA for 642ad40

Commits on Apr 11, 2024

• Merge branch 'main' into migrate_subclasses_to_foundry

  maxisawesome

  authored
  f30db14

  Copy full SHA for f30db14

• fix pyright

  bmosaicml

  committed
  de321b2

  Copy full SHA for de321b2

• fix pyright error again

  bmosaicml

  committed
  019c58a

  Copy full SHA for 019c58a

• fix pyright

  bmosaicml

  committed
  779f490

  Copy full SHA for 779f490

• fix pyright

  bmosaicml

  committed
  03f7e91

  Copy full SHA for 03f7e91

Commits on Apr 12, 2024

• Merge branch 'openai_compatible_gauntlet' of github.com:mosaicml/llm-foundry into openai_compatible_gauntlet

  bmosaicml

  committed
  bb2728b

  Copy full SHA for bb2728b

• Merge branch 'migrate_subclasses_to_foundry' into openai_compatible_gauntlet

  bmosaicml

  committed
  65f1a3e

  Copy full SHA for 65f1a3e

• Merge branch 'main' into openai_compatible_gauntlet

  maxisawesome

  committed
  f493e35

  Copy full SHA for f493e35

Commits on Apr 13, 2024

• Merge branch 'main' into openai_compatible_gauntlet

  maxisawesome

  committed
  e62f584

  Copy full SHA for e62f584

Commits on Apr 19, 2024

• add api key to test

  maxisawesome

  committed
  d23aa5f

  Copy full SHA for d23aa5f

• Merge branch 'main' into openai_compatible_gauntlet

  maxisawesome

  authored
  558538b

  Copy full SHA for 558538b

• put openai imports behind typechecking

  maxisawesome

  committed
  47a0cb9

  Copy full SHA for 47a0cb9

• correct typecheckign import

  maxisawesome

  committed
  21de07a

  Copy full SHA for 21de07a

• Merge branch 'openai_compatible_gauntlet' of github.com:mosaicml/llm-foundry into openai_compatible_gauntlet

  maxisawesome

  committed
  7c1c32c

  Copy full SHA for 7c1c32c

• linting

  maxisawesome

  committed
  5dce21d

  Copy full SHA for 5dce21d

• fix gemni import errors

  maxisawesome

  committed
  619e2ce

  Copy full SHA for 619e2ce

• fix hf_eval

  maxisawesome

  committed
  eec82a1

  Copy full SHA for eec82a1

• linting

  maxisawesome

  committed
  6fc714b

  Copy full SHA for 6fc714b