handleSelectionEvent throws KeyError

[eqy]

I'm currently working on a project that attempts to handle batches of replays at time. One of my test packs of replay is the Dreamhack Summer 2013 replay pack:
http://www.teamliquid.net/forum/viewmessage.php?topic_id=418887

Please confirm if Group stage 2/Group B/Suppy vs Nightend/Suppy vs Nightend ZvP Game 2 Akilon Wastes.SC2Replay
produces this error:

Traceback (most recent call last):
  File "/usr/lib/python3.3/site-packages/sc2reader/engine/engine.py", line 174, in run
    for new_event in (event_handler(event, replay) or []):
  File "/usr/lib/python3.3/site-packages/sc2reader/engine/plugins/context.py", line 89, in handleSelectionEvent
    unit = replay.objects[unit_id]
KeyError: 41418753

I believe this is caused by a tracker event unit somehow not already existing in replay.objects and not existing in the dictionary.

[GraylinKim]

What version of sc2reader do you have installed. This was a known issue with 0.6.0 and was patched the next day in 0.6.1.

Current release is 0.6.3 and available on pypi.

[eqy]

I believe I am using 0.6.3:

$ cat *.py | grep 0.6.3
__version__ = "0.6.3"

(in /usr/lib/python3.3/site-packages/sc2reader)

[GraylinKim]

Okay thanks, I'll take a look at this then.

[GraylinKim]

Wow, this is kind of interesting. It seems that the replay.tracker.events file is actually corrupted/truncated here. The first event in the file is a UnitBornEvent for a probe over 4 minutes into the game! It is possible for us to work around this but I'm not sure if we should.

I will have to take this up with Blizzard.

[GraylinKim]

Posted about this on the Battle.net forums in this thread. Putting this on hold until Blizzard (hopefully) has a response for this.

[GraylinKim]

It looks like this one is pretty unexplainable. I'm not sure if I should apply a work around or not. It would be good to know when/if this happens again.

I can at least post a patch for the work around if people want to apply it.

[danielyule]

Hi,
I'm doing a similar project to @eqy, but I'm using the WCS Season 1 & 2 replay pack. I've got the same issue. Obviously, I can just skip the file, but it's worth pointing out that the replay is for a game that is resumed. Starcraft plays the replay just fine.

I'm on v 0.6.4. The replay in question is available at https://dl.dropboxusercontent.com/u/2337534/2013-05-21%2023-59-46%20TheStC%2C%20mOOnGLaDe%20-%20TvZ%20-%20WCS%20America%20Whirlwind.SC2Replay

[GraylinKim]

Yes, the issue is in the tracker.events file which Starcraft doesn't use when playing replays. None of the replays are corrupt in that sense and all will replay just fine. If someone can identify a common thread between replays found to have this problem it would go a long way towards getting it fixed.

[GraylinKim]

Until then, I guess we'll just leave this open for people to keep adding to.

[eqy]

I just checked the replay I initially pointed out as having this problem, and it is also a replay for a game that was resumed from replay.

Would it be a good idea to omit loading tracker events if the replay is found to be resumed? It seems this issue may be related to #91.

[GraylinKim]

This issue is almost certainly caused by resume from replay games and probably affects all of them. I was able to reproduce quite easily. Just take a normal functioning replay, resume from replay on it, and observe that all tracker events prior to the hijack point are missing from the second replay.

I don't think it is a good idea to hack around this though. Tracker events bring a higher level of data quality and analysis that just won't be there even if I hack around the issue. Data will be partially missing or inaccurate with no straight forward way to know which data is good and which data is bad.

Most people using sc2reader should retry with a lower level of processing before completely giving up on a replay. Especially if your service needs to report something to the user after uploading. If you only want basic replay data you can retry parsing with most of the plugins disabled:

small_engine = sc2reader.engine.GameEngine(plugins=[sc2reader.engine.plugins.GameHeartNormalizer()])
replay = sc2reader.load_replay('broken.SC2Replay', engine=small_engine)

If you know you don't need to parse GameHeart games and don't need event information you can go even lower:

replay = sc2reader.load_replay(replay_path, load_level=2)

I'll leave this ticket open for reference and feedback until Blizzard patches the client.

[GraylinKim]

Just as an update for watchers, I have confirmation from Blizzard that this bug will be fixed in the next Starcraft patch.

Any hacks around the original issue will create invisible issues with data quality for affected replays. Since this bug is confirmed fixed in future sc2 versions I am closing the issue as a won't fix. The post above details how projects can deal with this issue in existing replays.

[GraylinKim]

As a final update here, 6fb12e8 adds the functionality to throw a CorruptTrackerFileError when this situation is encountered. A project can catch this and recover as they see fit.