Enhance Gemini usage tracking to collect comprehensive token data

[kiqaps]

When using Gemini, I noticed that some tokens (such as reasoning tokens and usage by modality) were not being collected. I made an adjustment so that all of these are included within the Usage details.

Since details is a dict from str to int, I couldn't simply throw them in there, so I created the dict in what I thought was the most intuitive way, but I'm not sure if it's the best approach.

I also had to add line breaks in this CLI test to make it pass on my PC (not sure why), but after pushing I saw that it broke the tests, so I removed them again :D

Now, every token documented here: https://ai.google.dev/api/generate-content#UsageMetadata is collected.

Also, when streaming, I'm retrieving usage data only from the last chunk and this should fix #1736

[amiyapatanaik]

Thanks @kiqaps, much appreciated.

[DouweM]

@kiqaps Thanks for picking this up! Just one suggestion.

[Kludex]

@kiqaps Would you mind checking the google.py as well? (after rebase)

[kiqaps]

@kiqaps Would you mind checking the google.py as well? (after rebase)

yeah, sure, no problem

[kiqaps]

ok, its done... but i have done a more intrusive implementation to keep a single "token parser" for both providers (via genai or http), will you guys maintain both or http will be deprecated?

if you prefer, I can keep the parsing in both providers

[Kludex]

The idea is to move forward with the google-genai implementation, and at some point deprecate the httpx one.

So I think it's better to have duplicated logic and not have a common file. Would you mind changing it? Sorry if this was not clear before.