There are problems with using inference with vllm and using inference with Transformrers

[cug-auto-zp]

I used the provided aria_ui_vllm.py and aria_ui_hf.py for inference separately and found that there are inconsistencies in the results.

1. running aria_ui_vllm.py

    llm = LLM(
        model=model_path,
        tokenizer_mode="slow",
        dtype="bfloat16",
        trust_remote_code=True,
    )
    tokenizer = AutoTokenizer.from_pretrained(
        model_path, trust_remote_code=True, use_fast=False
    )
    instruction = "Try Aria."
    messages = [
        {
            "role": "user",
            "content": [
                {"type": "image"},
                {
                    "type": "text",
                    "text": "Given a GUI image, what are the relative (0-1000) pixel point coordinates for the element corresponding to the following instruction or description: " + instruction,
                }
            ],
        }
    ]

outputs：

```(684, 786)```<|im_end|>

After running draw_coord, it can be seen that the correct coordinates for 'Try Aria' were not found.

2. running aria_ui_hf.py

instruction = "Try Aria."

image = Image.open(image_file).convert("RGB")

# NOTE: using huggingface on a single 80GB GPU, we resize the image to 1920px on the long side to prevent OOM. this is unnecessary with vllm.
image = resize_image(image, long_size=1920)

messages = [
    {
        "role": "user",
        "content": [
            {"text": None, "type": "image"},
            {"text": instruction, "type": "text"},
        ],
    }
]

outputs：

```(767, 782)```<|im_end|>

After draw_coord

3. When I changed the prompt in vllm. I removed the sentence 'Given a GUI image, what are the relative (0-1000) pixel point coordinates for the element corresponding to the following instruction or description:'.

instruction = "Try Aria."
    messages = [
        {
            "role": "user",
            "content": [
                {"type": "image"},
                {
                    "type": "text",
                    "text":  instruction,
                }
            ],
        }
    ]

outputs：

```(760, 786)```<|im_end|>

This way, the observed results are correct

[AriaUI]

Hi,

Thanks for your feedback. We personally never experienced the first case as this specific example is actually very easy to ground.

Could you try fixing the temperature to 0 and top_k to 1 for the vLLM model using SamplingParams?

[cug-auto-zp]

Hi,

Thanks for your feedback. We personally never experienced the first case as this specific example is actually very easy to ground.

Could you try fixing the temperature to 0 and top_k to 1 for the vLLM model using SamplingParams?

Hi，

Thank you very much for your reply. I tried the temperature to 0 and top_k to 1 you said, but the output still does not change.

```(684, 786)```<|im_end|>

[AriaUI]

Hi,

Sorry for late reply. In your scenario, does vllm-powered Aria-UI works well for other images and prompts?

I'm using a single A800 for inference and still has difficulty in reproducing the problem you presented. :(

[hj611]

请问问题解决了吗，我遇到了类似的情况，我用vllm在AndroidWorld上测试结果全部为0。

[AriaUI]

请问问题解决了吗，我遇到了类似的情况，我用vllm在AndroidWorld上测试结果全部为0。

Hi,
感谢反馈，请问hf结果正常吗，另外能否提供下vllm版本？

BTW，我们自己的所有测试都是基于vllm做的，从来没遇到过这个情况 :(

[hj611]

请问问题解决了吗，我遇到了类似的情况，我用vllm在AndroidWorld上测试结果全部为0。

您好， 感谢反馈，请问hf结果正常吗，另外能否提供下vllm版本？

顺便说一句，我们自己的所有测试都是基于vllm做的，从来没有遇到过这种情况:(

您好，感谢回复。我的vllm版本是“0.6.6.dev3+g866fa455” ，gpt版本是“gpt-4-turbo-2024-04-09”，使用python3 -m vllm.entrypoints.openai.api_server --served-model-name Aria-UI/Aria-UI-base方式启动的服务端，没有测试hf的结果。

[hj611]

这里是我的测试记录：
Action: {"action_type": "click", "instruction": "Click the 'Get started' button to proceed", "target": "the 'Get started' button on the welcome screen"}
Reason: To proceed with recording an audio clip, the first step is to access the main interface of the Audio Recorder app. The "Get started" button on the welcome screen needs to be clicked to move forward.
Chat completion output:(500, 439)
AAction: {"action_type": "click", "instruction": "Click the 'Get started' button again to proceed", "target": "the 'Get started' button on the welcome screen"}
Reason: The previous attempt to click the "Get started" button may not have been registered properly by the app. Since the screen has not changed and the button is still visible, it is appropriate to attempt clicking the "Get started" button again to proceed with the app's functionality.
Chat completion output:(332, 324)
Action: {"action_type": "long_press", "instruction": "Long press the 'Get started' button to check for any different response", "target": "the 'Get started' button on the welcome screen"}
Reason: The "Get started" button has been clicked twice without any response or change in the screen, suggesting a possible issue with the button or the app's responsiveness. To troubleshoot and attempt to progress, I will try a long press on the "Get started" button to see if it triggers any different response.
Chat completion output:(500, 903)

Action: {"action_type": "click", "instruction": "Tap on the 'Apply' button again to confirm settings and proceed", "target": "'Apply' button at the bottom of the screen"}
Reason: The previous attempt to click the 'Apply' button did not transition the screen to the recording interface as expected. It might be necessary to retry clicking the 'Apply' button to confirm the settings and proceed to the recording interface.
Chat completion output:(259, 294)

[AriaUI]

请问问题解决了吗，我遇到了类似的情况，我用vllm在AndroidWorld上测试结果全部为0。

您好， 感谢反馈，请问hf结果正常吗，另外能否提供下vllm版本？
顺便说一句，我们自己的所有测试都是基于vllm做的，从来没有遇到过这种情况:(

您好，感谢回复。我的vllm版本是“0.6.6.dev3+g866fa455” ，gpt版本是“gpt-4-turbo-2024-04-09”，使用python3 -m vllm.entrypoints.openai.api_server --served-model-name Aria-UI/Aria-UI-base方式启动的服务端，没有测试hf的结果。

感谢反馈，我判断可能是vllm自带的api_server对Aria系列模型的支持不够完善，我在测试时使用的是内部的api server实现，贴代码在下面，麻烦再测试下看看：

from fastapi import FastAPI, HTTPException
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel
from typing import List, Union, Optional
from PIL import Image, ImageDraw
import numpy as np
import base64
import io
from transformers import AutoTokenizer
from vllm import LLM, SamplingParams
import ast

app = FastAPI()

# Add CORS middleware
app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)

# Initialize model and tokenizer
MODEL_PATH = "Aria-UI/Aria-UI-base"
llm = LLM(
    model=MODEL_PATH,
    tokenizer_mode="slow",
    dtype="bfloat16",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(
    MODEL_PATH, 
    trust_remote_code=True, 
    use_fast=False
)

class Message(BaseModel):
    role: str
    content: List[dict]

class ChatCompletionRequest(BaseModel):
    messages: List[Message]
    model: str
    max_tokens: Optional[int] = 512
    stop: Optional[List[str]] = ["<|im_end|>"]
    extra_body: Optional[dict] = {
        "split_image": True,
        "image_max_size": 980
    }

class Choice(BaseModel):
    message: Message
    finish_reason: str
    index: int

class ChatCompletionResponse(BaseModel):
    id: str
    object: str
    created: int
    model: str
    choices: List[Choice]

def base64_to_pil(base64_string: str) -> Image.Image:
    """Convert base64 image to PIL Image."""
    # Remove data URL prefix if present
    if 'base64,' in base64_string:
        base64_string = base64_string.split('base64,')[1]
    image_data = base64.b64decode(base64_string)
    return Image.open(io.BytesIO(image_data))

@app.post("/v1/chat/completions", response_model=ChatCompletionResponse)
async def create_chat_completion(request: ChatCompletionRequest):
    try:
        # Process the last message which should contain the image and prompt
        last_message = request.messages[-1]
        
        # Extract text prompt and image from the message content
        text_content = None
        image_content = None
        for content in last_message.content:
            if content["type"] == "text":
                text_content = content["text"]
            elif content["type"] == "image_url":
                image_url = content["image_url"]["url"]
                image_content = base64_to_pil(image_url)

        if not text_content or not image_content:
            raise HTTPException(status_code=400, message="Missing text prompt or image")

        # Format messages for the model
        messages = [{
            "role": "user",
            "content": [
                {"type": "image"},
                {
                    "type": "text",
                    "text": text_content,
                }
            ],
        }]

        # Generate message using tokenizer
        message = tokenizer.apply_chat_template(messages, add_generation_prompt=True)

        # Get model output
        outputs = llm.generate(
            {
                "prompt_token_ids": message,
                "multi_modal_data": {
                    "image": [image_content],
                    "max_image_size": request.extra_body.get("image_max_size", 980),
                    "split_image": request.extra_body.get("split_image", True),
                },
            },
            sampling_params=SamplingParams(
                max_tokens=request.max_tokens,
                top_k=1,
                stop=request.stop
            ),
        )

        # Process the response
        generated_tokens = outputs[0].outputs[0].token_ids
        response = tokenizer.decode(generated_tokens, skip_special_tokens=True)

        # Create response object
        return ChatCompletionResponse(
            id="chatcmpl-" + base64.b64encode(str(hash(response)).encode()).decode()[:10],
            object="chat.completion",
            created=int(time.time()),
            model=request.model,
            choices=[
                Choice(
                    message=Message(
                        role="assistant",
                        content=[{"type": "text", "text": response}]
                    ),
                    finish_reason="stop",
                    index=0
                )
            ]
        )

    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))

if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="0.0.0.0", port=8000)

[AriaUI]

用hf space简单测试了一下，模型是完全没问题的，应该是使用vllm api server推理时产生的问题

[hj611]

使用hf space简单测试一下，模型是没问题的，应该是使用vllm api server完全推理时产生的问题

grounding依旧存在问题：
----------step 2
Action: {"action_type": "click", "instruction": "Click on the 'Get started' button to proceed", "target": "the 'Get started' button"}
Reason: To proceed with recording an audio clip using the Audio Recorder app, the next step is to access the main interface of the app where recording functionalities are available. The "Get started" button on the welcome screen likely leads to this interface.
python(8515) MallocStackLogging: can't turn off malloc stack logging because it was not enabled.
python(8516) MallocStackLogging: can't turn off malloc stack logging because it was not enabled.
Chat completion output:(34, 33)
----------step 3
Action: {"action_type": "click", "instruction": "Click on the 'Get started' button again to proceed", "target": "the 'Get started' button"}
Reason: The previous attempt to click the "Get started" button did not change the screen, suggesting that the button may not have functioned as expected. To troubleshoot, I will try clicking the "Get started" button again to see if it responds this time.
Chat completion output:(204, 972)

llm收到的prompt：
Received request chatcmpl-33c444e829874771bedd0e418def043f: prompt: '<|im_start|>user\nThe agent is performing the ultimate task: Record an audio clip using Audio Recorder app and save it..\nHistory of the agent's steps:\n\t1. Action selected: {"action_type": "open_app", "app_name": "Audio Recorder"}. Opened the Audio Recorder app successfully; the app is now on the welcome screen with a "Get started" button visible, ready for further interaction.\n\t2. Action selected: {"action_type": "click", "instruction": "Click on the 'Get started' button to proceed", "target": "the 'Get started' button"}. Clicked the 'Get started' button in the Audio Recorder app to access the main interface for recording; the screen did not change, indicating the button may not have functioned as expected or requires another action..\nStep 3. Given a GUI image, what are the relative (0-1000) pixel point coordinates for the element corresponding to the following instruction or description: description: the 'Get started' button; instruction: Click on the 'Get started' button again to proceed<fim_prefix><|img|><fim_suffix><|im_end|>\n<|im_start|>assistant\n', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=1.0, top_p=1.0, top_k=-1, min_p=0.0, seed=None, stop=['<|im_end|>'], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=512, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None), prompt_token_ids: None, lora_request: None, prompt_adapter_request: None.

[AriaUI]

<|im_start|>user\nThe agent is performing the ultimate task: Record an audio clip using Audio Recorder app and save it..\nHistory of the agent's steps:\n\t1. Action selected: {"action_type": "open_app", "app_name": "Audio Recorder"}. Opened the Audio Recorder app successfully; the app is now on the welcome screen with a "Get started" button visible, ready for further interaction.\n\t2. Action selected: {"action_type": "click", "instruction": "Click on the 'Get started' button to proceed", "target": "the 'Get started' button"}. Clicked the 'Get started' button in the Audio Recorder app to access the main interface for recording; the screen did not change, indicating the button may not have functioned as expected or requires another action..\nStep 3. Given a GUI image, what are the relative (0-1000) pixel point coordinates for the element corresponding to the following instruction or description: description: the 'Get started' button; instruction: Click on the 'Get started' button again to proceed<fim_prefix><|img|><fim_suffix><|im_end|>

这一部分Prompt看起来不太对

1. 在history部分，应该只包含instruction部分即可，不需要额外信息，我再检查下开源代码
2. 你用的模型是base模型吗，试试只输入当前步骤的instruction如下：

Given a GUI image, what are the relative (0-1000) pixel point coordinates for the element corresponding to the following instruction or description: description: the 'Get started' button; instruction: Click on the 'Get started' button again to proceed

看看表现会否正常