Clarification needed on Gemini Level1 Who evaluation protocol

[lovepope]

Hi, thanks for releasing SocialOmni.

I am trying to reproduce the Level1 Who results reported in the paper, especially
the Gemini results in Table 2 / Table 3 / Table 5. I found a large mismatch and
would like to confirm the exact evaluation protocol.

What I tested

Task: Level1 Who, 2000 samples.

I reran the benchmark with Gemini native video input:

• Input: full original video + audio
• API path: Gemini native generateContent / streamGenerateContent
• Media format: fileData.fileUri pointing to video files
• No ground-truth ASR transcript injected
• Temperature: 0
• Metrics recomputed from JSONL:
  • Accuracy
  • Macro-F1 over A/B/C/D
  • Bootstrap 95% CI with 10,000 resamples, seed=42
  • Consistent / inconsistent split accuracy

My rerun results are much higher than the paper numbers:

• Gemini 2.5 Pro:

  • My Acc: 93.30
  • My Macro-F1: 93.20
  • My Consistent Acc: 93.97
  • My Inconsistent Acc: 89.09
  • Paper Acc: 44.69
  • Paper Macro-F1: 44.53

• Gemini 3 Flash Preview:

  • My Acc: 93.25
  • My Macro-F1: 93.25
  • My Consistent Acc: 94.20
  • My Inconsistent Acc: 87.27
  • Paper Acc: 53.23
  • Paper Macro-F1: 53.36

This does not look like a metric calculation issue, since the gap is very large.

Possible protocol mismatch

The paper states that models receive raw video and audio input, and are not given
the ground-truth ASR transcript.

However, in the released repo, the hosted Gemini path appears to use
OpenAICompatTester, which sends text plus sampled image frames, rather than Gemini
native video/audio input.

Because of this, I am unsure which input protocol was actually used for the reported
Gemini numbers:

• Full original video + audio
• Video prefix up to the queried timestamp
• Cropped segment around the queried timestamp
• Sampled image frames only
• Some other internal/private evaluation runner

The full-video native Gemini result is much higher than the paper result, so I
suspect my input protocol is not equivalent to the one used in the paper.

Questions

Could you clarify the following?

1. For Table 2 / Table 3 / Table 5 Level1 Who, did Gemini receive full videos,
   cropped clips, video prefixes, or sampled frames?
2. Did Gemini receive native audio/video input, or only extracted image frames?
3. Were the released repo scripts used directly for the reported Gemini results, or
   was there a separate internal evaluation runner?
4. Is there a recommended command/configuration to reproduce the reported Gemini
   Level1 Who numbers exactly?

---

中文翻译：

您好，感谢开源 SocialOmni。

我正在尝试复现论文中的 Level1 Who 结果，尤其是 Table 2 / Table 3 / Table 5 里的
Gemini 结果。但我发现复测分数和论文报告分数差异很大，因此想确认一下具体的评测协议。

我的测试方式

任务：Level1 Who，共 2000 条样本。

我使用 Gemini 原生视频输入重新跑了一遍：

• 输入：完整原始视频 + 音频
• API 路径：Gemini 原生 generateContent / streamGenerateContent
• 媒体格式：fileData.fileUri 指向视频文件
• 没有注入 ground-truth ASR transcript
• Temperature: 0
• 从 JSONL 重新计算指标：
  • Accuracy
  • A/B/C/D 四类 Macro-F1
  • Bootstrap 95% CI，10,000 次 resampling，seed=42
  • Consistent / inconsistent split accuracy

我的复测结果明显高于论文分数：

• Gemini 2.5 Pro：

  • 我的 Acc: 93.30
  • 我的 Macro-F1: 93.20
  • 我的 Consistent Acc: 93.97
  • 我的 Inconsistent Acc: 89.09
  • 论文 Acc: 44.69
  • 论文 Macro-F1: 44.53

• Gemini 3 Flash Preview：

  • 我的 Acc: 93.25
  • 我的 Macro-F1: 93.25
  • 我的 Consistent Acc: 94.20
  • 我的 Inconsistent Acc: 87.27
  • 论文 Acc: 53.23
  • 论文 Macro-F1: 53.36

这个差距非常大，看起来不像是指标计算方式造成的。

可能的协议差异

论文中写到，模型接收 raw video 和 audio input，并且不会看到 ground-truth ASR
transcript。

但是在公开 repo 里，hosted Gemini 路径看起来使用的是 OpenAICompatTester，它发送的
是文本加抽样 image frames，而不是 Gemini 原生 video/audio 输入。

因此我不确定论文报告 Gemini 分数时实际使用的是哪种输入协议：

• 完整原始视频 + 音频
• 截止到 query timestamp 的视频前缀
• query timestamp 附近裁剪后的视频片段
• 只抽样 image frames
• 其他内部/private evaluation runner

完整视频 + 原生 Gemini 输入得到的分数远高于论文分数，所以我怀疑我的输入协议和论文实
际使用的协议并不等价。

想确认的问题

请问能否澄清以下几点？

1. 对于 Table 2 / Table 3 / Table 5 的 Level1 Who，Gemini 收到的是完整视频、裁剪片
   段、视频前缀，还是抽样帧？
2. Gemini 实际收到的是原生 audio/video 输入，还是只收到抽取出来的 image frames？
3. 论文中的 Gemini 结果是否直接由公开 repo 脚本生成？还是使用了单独的内部 evaluation
   runner？
4. 是否有推荐的命令或配置，可以精确复现论文中 Gemini 的 Level1 Who 分数？