Skip to content

Commit

Permalink
fix imports
Browse files Browse the repository at this point in the history
  • Loading branch information
ShawnXuan committed Sep 20, 2024
1 parent 400dcab commit 88fcc10
Showing 1 changed file with 1 addition and 3 deletions.
4 changes: 1 addition & 3 deletions projects/Qwen/tokenizer.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,12 +20,10 @@
import unicodedata
from functools import lru_cache
from io import open
from typing import List, Optional
from typing import Optional

import oneflow as flow
import regex as re

import libai.utils.distributed as dist
from libai.tokenizer.tokenization_base import PreTrainedTokenizer

logger = logging.getLogger(__name__)
Expand Down

0 comments on commit 88fcc10

Please sign in to comment.