Support phi 3.5

[minhthuc2502]

No description provided.

[BBC-Esq]

After converting to the int8_bfloat16 I get this error when trying to run it in a script;

  File "D:\Scripts\bench_chat\ct2_phi3.py", line 87, in main
    results_batch = generator.generate_batch(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: expected storage to be of type float32, but is of type bfloat16

I also received the same exact message when first converting the model into bfloat16:

    results_batch = generator.generate_batch(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: expected storage to be of type float32, but is of type bfloat16

The latter is really weird because the model card says that it's originally in bfloat16...

[BBC-Esq]

See this if it help:

#1792

[minhthuc2502]

I quantized phi3.5 to int8_bfloat16 and didn't have any error like you mentioned above in inference time. Please provide more in detail how to reproduce this and which model that you used.

[BBC-Esq]

Sure...

The script I used to convert it is located here:

https://github.com/BBC-Esq/Ctranslate2-Converter/blob/main/Ctranslate2-Converter/convert_ctranslate2.py

And the script I used to run it is as follows:

import os
import ctranslate2
from transformers import AutoTokenizer

model_dir = r"D:\Scripts\bench_chat\models\Phi-3.5-mini-instruct-ct2-bfloat16"

def build_prompt():
    system_message = "You are a helpful AI assistant."
    user_message = "Tell me a short joke."
    
    prompt = f"""<s><|system|>
{system_message}<|end|>
<|user|>
{user_message}<|end|>
<|assistant|>
"""
    return prompt

def main():
    print(f"Loading the model: {os.path.basename(model_dir)}...")
    
    generator = ctranslate2.Generator(
        model_dir,
        device="cuda",
        compute_type="bfloat16"
    )
    
    tokenizer = AutoTokenizer.from_pretrained(model_dir)
    
    prompt = build_prompt()
    tokens = tokenizer.convert_ids_to_tokens(tokenizer.encode(prompt))
    
    print("Generating response...")
    results = generator.generate_batch(
        [tokens],
        include_prompt_in_result=False,
        max_batch_size=4096,
        batch_type="tokens",
        beam_size=1,
        num_hypotheses=1,
        max_length=512,
        sampling_temperature=0.00,
    )
    
    output = tokenizer.decode(results[0].sequences_ids[0])
    
    print("\nGenerated response:")
    print(output)

if __name__ == "__main__":
    main()

[minhthuc2502]

It seems like you are running this script on a GPU < 8.x while it does not support compute type bfloat16. Remove this line compute_type="bfloat16" and try again. If you have any futher problem, feel free to open a new issue.

Sure...

The script I used to convert it is located here:

https://github.com/BBC-Esq/Ctranslate2-Converter/blob/main/Ctranslate2-Converter/convert_ctranslate2.py

And the script I used to run it is as follows:

import os
import ctranslate2
from transformers import AutoTokenizer

model_dir = r"D:\Scripts\bench_chat\models\Phi-3.5-mini-instruct-ct2-bfloat16"

def build_prompt():
    system_message = "You are a helpful AI assistant."
    user_message = "Tell me a short joke."
    
    prompt = f"""<s><|system|>
{system_message}<|end|>
<|user|>
{user_message}<|end|>
<|assistant|>
"""
    return prompt

def main():
    print(f"Loading the model: {os.path.basename(model_dir)}...")
    
    generator = ctranslate2.Generator(
        model_dir,
        device="cuda",
        compute_type="bfloat16"
    )
    
    tokenizer = AutoTokenizer.from_pretrained(model_dir)
    
    prompt = build_prompt()
    tokens = tokenizer.convert_ids_to_tokens(tokenizer.encode(prompt))
    
    print("Generating response...")
    results = generator.generate_batch(
        [tokens],
        include_prompt_in_result=False,
        max_batch_size=4096,
        batch_type="tokens",
        beam_size=1,
        num_hypotheses=1,
        max_length=512,
        sampling_temperature=0.00,
    )
    
    output = tokenizer.decode(results[0].sequences_ids[0])
    
    print("\nGenerated response:")
    print(output)

if __name__ == "__main__":
    main()

[BBC-Esq]

It seems like you are running this script on a GPU < 8.x while it does not support compute type bfloat16. Remove this line compute_type="bfloat16" and try again. If you have any futher problem, feel free to open a new issue.

Sure...
The script I used to convert it is located here:
https://github.com/BBC-Esq/Ctranslate2-Converter/blob/main/Ctranslate2-Converter/convert_ctranslate2.py
And the script I used to run it is as follows:

import os
import ctranslate2
from transformers import AutoTokenizer

model_dir = r"D:\Scripts\bench_chat\models\Phi-3.5-mini-instruct-ct2-bfloat16"

def build_prompt():
    system_message = "You are a helpful AI assistant."
    user_message = "Tell me a short joke."
    
    prompt = f"""<s><|system|>
{system_message}<|end|>
<|user|>
{user_message}<|end|>
<|assistant|>
"""
    return prompt

def main():
    print(f"Loading the model: {os.path.basename(model_dir)}...")
    
    generator = ctranslate2.Generator(
        model_dir,
        device="cuda",
        compute_type="bfloat16"
    )
    
    tokenizer = AutoTokenizer.from_pretrained(model_dir)
    
    prompt = build_prompt()
    tokens = tokenizer.convert_ids_to_tokens(tokenizer.encode(prompt))
    
    print("Generating response...")
    results = generator.generate_batch(
        [tokens],
        include_prompt_in_result=False,
        max_batch_size=4096,
        batch_type="tokens",
        beam_size=1,
        num_hypotheses=1,
        max_length=512,
        sampling_temperature=0.00,
    )
    
    output = tokenizer.decode(results[0].sequences_ids[0])
    
    print("\nGenerated response:")
    print(output)

if __name__ == "__main__":
    main()

Are you referring to the cuda compute level? The GPU I'm running it on is an rtx 4090 so it supports compute more than 8...

[BBC-Esq]

Also, I just commented out this and same thing...

[BBC-Esq]

Do you want me to open a separate issue still? Seems redundant>