Code

Update spec infer#15

Code

Open

yukavio wants to merge 27 commits intomainyukavio/sglang:mainfrom update_spec_inferyukavio/sglang:update_spec_inferCopy head branch name to clipboard

+2,504-175

Commits

Commits on Oct 13, 2024

• temp

  

  kavioyu

  committed
  70135d6

  Copy full SHA for 70135d6

Commits on Oct 14, 2024

• migrated to new upstream, need implement evict memory

  

  kavioyu

  committed
  65fae7b

  Copy full SHA for 65fae7b

Commits on Oct 15, 2024

• prove single req

  

  kavioyu

  committed
  064cca6

  Copy full SHA for 064cca6

Commits on Oct 16, 2024

• fix bug for long generate due to eagle_verify_retrive kernel

  

  kavioyu

  committed
  cb01c64

  Copy full SHA for cb01c64

• fix bug of eagle spec verify

  

  kavioyu

  committed
  df3de9d

  Copy full SHA for df3de9d

Commits on Oct 19, 2024

• support cuda graph

  

  kavioyu

  committed
  b7628f2

  Copy full SHA for b7628f2

Commits on Oct 22, 2024

• support batch inference

  

  kavioyu

  committed
  e2634e9

  Copy full SHA for e2634e9

Commits on Oct 23, 2024

• temp

  

  kavioyu

  committed
  f557a06

  Copy full SHA for f557a06

Commits on Oct 24, 2024

• fix memeory leak

  

  kavioyu

  committed
  9987741

  Copy full SHA for 9987741

• add sampling score

  

  kavioyu

  committed
  dcbc11c

  Copy full SHA for dcbc11c

• support target model cuda graph

  

  kavioyu

  committed
  5578b18

  Copy full SHA for 5578b18

• disable target model cuda graph

  

  kavioyu

  committed
  af2e79a

  Copy full SHA for af2e79a

• fix batch bug

  

  kavioyu

  committed
  0fdd0b1

  Copy full SHA for 0fdd0b1

Commits on Oct 25, 2024

• disable cuda graph pad in eagle

  

  kavioyu

  committed
  923523b

  Copy full SHA for 923523b

• fix server args

  

  kavioyu

  committed
  0e3fea2

  Copy full SHA for 0e3fea2

• fix cuda graph and split prefill

  

  kavioyu

  committed
  4faaa31

  Copy full SHA for 4faaa31

• optimize generate attn arg

  

  kavioyu

  committed
  33d8aef

  Copy full SHA for 33d8aef

• fix parent list dtype

  

  kavioyu

  committed
  11d6e86

  Copy full SHA for 11d6e86

Commits on Oct 26, 2024

• fix draft worker memory problem

  

  kavioyu

  committed
  2b3cb22

  Copy full SHA for 2b3cb22

• need to fix decode error when request retract happend

  

  kavioyu

  committed
  7aa0aff

  Copy full SHA for 7aa0aff

• remove debug info

  

  kavioyu

  committed
  404c5ab

  Copy full SHA for 404c5ab

Commits on Oct 27, 2024

• fix bug

  

  kavioyu

  committed
  e095ec0

  Copy full SHA for e095ec0

Commits on Oct 29, 2024

• fix some bug and support target model use cuda graph

  

  kavioyu

  committed
  9f0a0c2

  Copy full SHA for 9f0a0c2

Commits on Nov 1, 2024

• fix conflict, should solve scheduler and cuda graph problem

  

  kavioyu

  committed
  35c5678

  Copy full SHA for 35c5678

• fix naive cuda graph

  

  kavioyu

  committed
  b647a70

  Copy full SHA for b647a70

Commits on Nov 2, 2024

• fix cuda graph

  

  kavioyu

  committed
  7226987

  Copy full SHA for 7226987

• support split prefill batch

  

  kavioyu

  committed
  aaf1cae

  Copy full SHA for aaf1cae