parse HotS beta replays

[dsjoerg]

I can provide some if necessary

[GraylinKim]

That would be great. I have my own but experts agree, more is better.

On Fri, Oct 5, 2012 at 4:22 PM, David Joerg [email protected]:

I can provide some if necessary

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/74.

[dsjoerg]

Do you know of a reliable way to identify a replay file as a HotS beta replay from python? If so, I can trawl the replays uploaded to ggtracker since the beta and identify a ton of them that way.

[GraylinKim]

Yes, something like this will work:

>>> import sc2reader
>>> sc2reader.load_replay('beta/sc2replay/star_station.SC2Replay', load_level=0).release_string
'2.0.0.23271'

All HotS replays will be 2.x.x.build. All WoL replays will be 1.x.x.build.

[GraylinKim]

I already have HotS parsing working for my personal replays so if you could provide a few hundred extras for testing that would be great.

After confirming that the parsing works on your replays I'll need to figure out how the sc2replay-csharp project generates their data dictionaries. I have some rough instructions from @mischanix but I still need to go through them and actually do it.

[dsjoerg]

Here are the 69 HotS replays that have been uploaded to ggtracker:
http://davidjoerg.com/ggtracker/hotsreplays.tar.gz

On Sun, Oct 7, 2012 at 2:58 PM, Graylin Kim [email protected]:

I already have HotS parsing working for my personal replays so if you
could provide a few hundred extras for testing that would be great.

After confirming that the parsing works on your replays I'll need to
figure out how the sc2replay-csharphttps://github.com/ascendedguard/sc2replay-csharpproject generates their data dictionaries. I have some rough instructions
from @mischanix https://github.com/Mischanix but I still need to go
through them and actually do it.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/74#issuecomment-9211048.

[GraylinKim]

Thanks David, seems like all of these parse as well!

@mischanix helped me compile the HotS unit and ability lookup tables so if all goes well we should have HotS support in the next couple days.

[dsjoerg]

sweeeeeeeeeeeet

On Sun, Oct 7, 2012 at 6:09 PM, Graylin Kim [email protected]:

Thanks David, seems like all of these parse as well!

@mischanix https://github.com/Mischanix helped me compile the HotS unit
and ability lookup tables so if all goes well we should have HotS support
in the next couple days.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/74#issuecomment-9213080.

[GraylinKim]

Code for parsing HotS beta replays is committed in 86b85de. Replays now have an expansion attribute with values of either WoL or HotS that can be used to distinguish them in sc2reader.

Unit/Ability lookups and information still pending.

[GraylinKim]

The hots branch now has fully operational hots support with unit/ability look ups completed as well.

I'll look to do some more testing of this before merging it into master.

[Prillan]

Just want you to know that I won a beta key in the TL Giveaway this week so I can provide replays as the games keeps getting patched.

[Prillan]

All my uploaded replays can be found here:
https://www.dropbox.com/sh/pie4b4oyhb3yksw/sJbyNGMGyd/hots-beta-replays

Please tell me if you need some specific game, ability, unit, whatever :)

[GraylinKim]

Thanks @Prillan. I need to update the data dictionaries for yesterday's patch.

Then I'll see how I do on your sample set.

[dsjoerg]

Graylin, what remains to be done on this in your opinion? Lots of people are interested in Hots support, so if there are concrete remaining tasks, perhaps people could pitch in.

[GraylinKim]

@wraithan has reported that one of the recent patches broke parsing and promised to send me replays.

As such there are two main tasks:

1. Update the parser - HotS replays (from all patches) required.
2. Update the data - Try out the nice data tool @mischanix sent me.

The parser fix I think I might be best handling if I can get a sizeable replay set. If someone wants to help out with the data update I can forward more information to them. Once the updates are in place I imagine you could also help out a lot with verification @dsjoerg.

[dsjoerg]

OK @GraylinKim here are some HotS replays: http://davidjoerg.com/ggtracker/hotsreplays.zip

20 replays from version 2.0.0.23271, 86 from 2.0.0.23925 and 108 from 2.0.0.24247.

[GraylinKim]

All ladder games from HotS should be parsing now with a couple caveats:

1. In build 24247 player numbers start at 0, not at 1. I haven't decided how to resolve this yet.
2. New data packs have not been loaded. Unit/ability level information may be wrong in places.

Non-Ladder melee games should also work fine with the above caveats as long as there are no observers! At some point in beta (at least since 24247) observers were added to the replay.attributes.events file. Because of this I still need to adjust the load_players() procedure. I'll wait on that until I figure out to do with caveat 1 above.

Additionally, it seems that Blizzard can and will patch data without releasing new builds now. We're going to need to find a new way of selecting data packs. I have a couple ideas here but I'll deal with this aspect last.

[dsjoerg]

good news, getting there!

[GraylinKim]

It seems that computer player information is no longer included in the replay.attributes.events file. Indeed I can't find any information re: the AI options (difficulty, strategy) anywhere right now actually. Very confusing change. So add "games with computer players" to the list of currently non-functional beta replays. All non-beta replays still parse fine.

Computer players and observers may affect the pid distribution differently than they did before. Or maybe I just never understood it that well in the first place. So now the standing question is:

given a set of players, observers, and computers. How are pids distributed?

To work this out, I require replays with both human and computer players as well at least 1 observer/referee. Specifically, I'd like replays where the order in which the observers/computers/players are added to the lobby and teams changes. You don't need to play the game out, just make a few actions so there is something in the game.event queue for me and quit.

I currently only have 1 such game to go off of from @wraithan. It suggests that pids might be assigned purely on lobby entry order with computer players tacked onto the end. I would like to confirm this but I need help.

Thanks!

[GraylinKim]

I implemented what I think fixes the player load issues in commit 5c67131. I don't have many replays on hand with observers and/or computers for testing/confirmation though.

As a note to self, it seems like the message.events format has changed and we're no longer properly breaking them up. Investigate.

[GraylinKim]

The message.events parsing has now been fixed. All but two of my 800+ replay set are parsing now. The two non-parsing replays are very strange and not representative of the whole. I'll probably ignore these for now.

Computer support is completely broken since I don't know where to get their attributes (like team, difficulty, strategy) without there being entries for them in the attributes.events file. It must be somewhere but I have no idea where. I think computer support need not be a blocker though.

In unrelated news, resource sharing has now been fixed in what appears to have been a bug in all versions. I guess that shows how much people really look at resource sharing events.

Next up: making sure the data lookups are correct as reasonable for the more recent patches.

[dsjoerg]

Awwww yeah!

On Sat, Jan 19, 2013 at 2:11 AM, Graylin Kim [email protected]:

The message.events parsing has now been fixed. All but two of my 800+
replay set are parsing now. The two non-parsing replays are very strange
and not representative of the whole. I'll probably ignore these for now.

Computer support is completely broken since I don't know where to get
their attributes (like team, difficulty, strategy) without there being
entries for them in the attributes.events file. It must be _somewhere_but I have no idea where. I think computer support need not be a blocker
though.

In unrelated news, resource sharing has now been fixed in what appears to
have been a bug in all versions. I guess that shows how much people really
look at resource sharing events.

Next up: making sure the data lookups are correct as reasonable for the
more recent patches.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/74#issuecomment-12451585.

[Prillan]

Perhaps this is related to the problems with AI players?
http://us.battle.net/sc2/en/forum/topic/7708262879

[GraylinKim]

Hmm, it doesn't seem directly related @Prillan as it deals with spectators and referees. But it does indicate that they've been screwing around with handling the player data behind the scenes. And clearly AI handling in replays was affected.

Thanks for the link.

[GraylinKim]

All HotS replays, including the new patch should be parsing successfully now I think. That doesn't mean that the information is verified 100% right and complete but it is a good milestone. As always I'm accepting new replays from people as confirmation.

If people on this ticket could do some data verification on their games and report bugs that would help me focus on other HotS stuff.

[GraylinKim]

HotS replays should be functional in the HotS branch now with the exception of "Resume from Replay" replays. We'll track those separately in #91.