[tokenizer] fix llama tokenizer (slow)

[DrownFish19]

What does this PR do?

Fix the convert_tokens_to_string function of llama tokenizer.

In the convert_tokens_to_string function, when out_string != "" and the next token is a special token, the function will add [whitespace] before text.

transformers/src/transformers/models/llama/tokenization_llama.py

Lines 285 to 301 in 4286586

	current_sub_tokens = []
	out_string = ""
	prev_is_special = False
	for i, token in enumerate(tokens):
	# make sure that special tokens are not decoded using sentencepiece model
	if token in self.all_special_tokens:
	if not prev_is_special and i != 0 and self.legacy:
	out_string += " "
	out_string += self.sp_model.decode(current_sub_tokens) + token
	prev_is_special = True
	current_sub_tokens = []
	else:
	if prev_is_special and i == 1 and self.add_prefix_space and not token.startswith(SPIECE_UNDERLINE):
	out_string += " "
	current_sub_tokens.append(token)
	prev_is_special = False
	out_string += self.sp_model.decode(current_sub_tokens)

Fixes # (issue)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@ArthurZucker