Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

"2207 bytes left!" on newer replays. #190

Open
MLLeKander opened this issue Feb 8, 2016 · 10 comments
Open

"2207 bytes left!" on newer replays. #190

MLLeKander opened this issue Feb 8, 2016 · 10 comments

Comments

@MLLeKander
Copy link

I get an error when trying to parse recent replay files, such as this one.

The output of sc2parse using the lotv branch:

$ sc2parse IEM10-Taipei_Byun-sOs_G5.SC2Replay
dealing with IEM10-Taipei_Byun-sOs_G5.SC2Replay

IEM10-Taipei_Byun-sOs_G5.SC2Replay
Total failure parsing 3.1.1.39948
[ERROR] 2207 bytes left!
[ERROR] 2207 bytes left!
Traceback (most recent call last):
  File "/home/michael/Documents/sc2reader/sc2reader/scripts/sc2parse.py", line 78, in main
    replay = sc2reader.load_replay(path, debug=True, load_level=2)
  File "/home/michael/Documents/sc2reader/sc2reader/factories/sc2factory.py", line 85, in load_replay
    return self.load(Replay, source, options, **new_options)
  File "/home/michael/Documents/sc2reader/sc2reader/factories/sc2factory.py", line 137, in load
    return self._load(cls, resource, filename=filename, options=options)
  File "/home/michael/Documents/sc2reader/sc2reader/factories/sc2factory.py", line 146, in _load
    obj = cls(resource, filename=filename, factory=self, **options)
  File "/home/michael/Documents/sc2reader/sc2reader/resources.py", line 271, in __init__
    self._read_data(data_file, self._get_reader(data_file))
  File "/home/michael/Documents/sc2reader/sc2reader/resources.py", line 618, in _read_data
    self.raw_data[data_file] = reader(data, self)
  File "/home/michael/Documents/sc2reader/sc2reader/readers.py", line 128, in __call__
    raise ValueError("{0} bytes left!".format(data.length-data.tell()))
ValueError: 2207 bytes left!

If it's relevant, this is the output from the master branch:

$ sc2parse IEM10-Taipei_Byun-sOs_G5.SC2Replay
dealing with IEM10-Taipei_Byun-sOs_G5.SC2Replay

IEM10-Taipei_Byun-sOs_G5.SC2Replay
Total failure parsing 3.1.1.39948
[ERROR] 'utf8' codec can't decode byte 0xfc in position 87: invalid start byte
[ERROR] 'utf8' codec can't decode byte 0xfc in position 87: invalid start byte
Traceback (most recent call last):
  File "/home/michael/Documents/SC_RT/sc2reader/sc2reader/scripts/sc2parse.py", line 78, in main
    replay = sc2reader.load_replay(path, debug=True, load_level=2)
  File "/home/michael/Documents/SC_RT/sc2reader/sc2reader/factories/sc2factory.py", line 85, in load_replay
    return self.load(Replay, source, options, **new_options)
  File "/home/michael/Documents/SC_RT/sc2reader/sc2reader/factories/sc2factory.py", line 137, in load
    return self._load(cls, resource, filename=filename, options=options)
  File "/home/michael/Documents/SC_RT/sc2reader/sc2reader/factories/sc2factory.py", line 146, in _load
    obj = cls(resource, filename=filename, factory=self, **options)
  File "/home/michael/Documents/SC_RT/sc2reader/sc2reader/resources.py", line 271, in __init__
    self._read_data(data_file, self._get_reader(data_file))
  File "/home/michael/Documents/SC_RT/sc2reader/sc2reader/resources.py", line 601, in _read_data
    self.raw_data[data_file] = reader(data, self)
  File "/home/michael/Documents/SC_RT/sc2reader/sc2reader/readers.py", line 33, in __call__
    ) for i in range(data.read_bits(5))],
  File "/home/michael/Documents/SC_RT/sc2reader/sc2reader/decoders.py", line 252, in read_aligned_string
    return self._buffer.read_string(count, encoding)
  File "/home/michael/Documents/SC_RT/sc2reader/sc2reader/decoders.py", line 108, in read_string
    return self.read_bytes(count).decode(encoding)
  File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xfc in position 87: invalid start byte
@dsjoerg
Copy link
Collaborator

dsjoerg commented Feb 8, 2016

I think the lotv branch is behind the times unfortunately.

My branch https://github.com/dsjoerg/sc2reader/tree/upstream is more
up-to-date.

On Mon, Feb 8, 2016 at 3:36 AM, OMGTallMonster [email protected]
wrote:

I get an error when trying to parse recent replay files, such as this one
http://lotv.spawningtool.com/10851/.

The output of sc2parse using the lotv branch:

$ sc2parse IEM10-Taipei_Byun-sOs_G5.SC2Replay
dealing with IEM10-Taipei_Byun-sOs_G5.SC2Replay

IEM10-Taipei_Byun-sOs_G5.SC2Replay
Total failure parsing 3.1.1.39948
[ERROR] 2207 bytes left!
[ERROR] 2207 bytes left!
Traceback (most recent call last):
File "/home/michael/Documents/sc2reader/sc2reader/scripts/sc2parse.py", line 78, in main
replay = sc2reader.load_replay(path, debug=True, load_level=2)
File "/home/michael/Documents/sc2reader/sc2reader/factories/sc2factory.py", line 85, in load_replay
return self.load(Replay, source, options, *_new_options)
File "/home/michael/Documents/sc2reader/sc2reader/factories/sc2factory.py", line 137, in load
return self._load(cls, resource, filename=filename, options=options)
File "/home/michael/Documents/sc2reader/sc2reader/factories/sc2factory.py", line 146, in _load
obj = cls(resource, filename=filename, factory=self, *_options)
File "/home/michael/Documents/sc2reader/sc2reader/resources.py", line 271, in init
self._read_data(data_file, self._get_reader(data_file))
File "/home/michael/Documents/sc2reader/sc2reader/resources.py", line 618, in _read_data
self.raw_data[data_file] = reader(data, self)
File "/home/michael/Documents/sc2reader/sc2reader/readers.py", line 128, in call
raise ValueError("{0} bytes left!".format(data.length-data.tell()))
ValueError: 2207 bytes left!

If it's relevant, this is the output from the master branch:

$ sc2parse IEM10-Taipei_Byun-sOs_G5.SC2Replay
dealing with IEM10-Taipei_Byun-sOs_G5.SC2Replay

IEM10-Taipei_Byun-sOs_G5.SC2Replay
Total failure parsing 3.1.1.39948
[ERROR] 'utf8' codec can't decode byte 0xfc in position 87: invalid start byte
[ERROR] 'utf8' codec can't decode byte 0xfc in position 87: invalid start byte
Traceback (most recent call last):
File "/home/michael/Documents/SC_RT/sc2reader/sc2reader/scripts/sc2parse.py", line 78, in main
replay = sc2reader.load_replay(path, debug=True, load_level=2)
File "/home/michael/Documents/SC_RT/sc2reader/sc2reader/factories/sc2factory.py", line 85, in load_replay
return self.load(Replay, source, options, *_new_options)
File "/home/michael/Documents/SC_RT/sc2reader/sc2reader/factories/sc2factory.py", line 137, in load
return self._load(cls, resource, filename=filename, options=options)
File "/home/michael/Documents/SC_RT/sc2reader/sc2reader/factories/sc2factory.py", line 146, in _load
obj = cls(resource, filename=filename, factory=self, *_options)
File "/home/michael/Documents/SC_RT/sc2reader/sc2reader/resources.py", line 271, in init
self._read_data(data_file, self._get_reader(data_file))
File "/home/michael/Documents/SC_RT/sc2reader/sc2reader/resources.py", line 601, in _read_data
self.raw_data[data_file] = reader(data, self)
File "/home/michael/Documents/SC_RT/sc2reader/sc2reader/readers.py", line 33, in call
) for i in range(data.read_bits(5))],
File "/home/michael/Documents/SC_RT/sc2reader/sc2reader/decoders.py", line 252, in read_aligned_string
return self._buffer.read_string(count, encoding)
File "/home/michael/Documents/SC_RT/sc2reader/sc2reader/decoders.py", line 108, in read_string
return self.read_bytes(count).decode(encoding)
File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xfc in position 87: invalid start byte

—
Reply to this email directly or view it on GitHub
#190.

@MLLeKander
Copy link
Author

Thanks for the heads up. Your branch works!

@stevemao
Copy link

stevemao commented Sep 1, 2017

I got the similar error when trying your branch and the branches here:

sc2parse MyReplay.SC2Replay
dealing with MyReplay.SC2Replay

MyReplay.SC2Replay
Total failure parsing 3.16.0.55505
[ERROR] 'utf-8' codec can't decode byte 0xfc in position 74: invalid start byte
[ERROR] 'utf-8' codec can't decode byte 0xfc in position 74: invalid start byte
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/sc2reader-0.7.0rc0-py3.6.egg/sc2reader/scripts/sc2parse.py", line 44, in main
    replay = sc2reader.load_replay(path, debug=True, load_level=1)
  File "/usr/local/lib/python3.6/site-packages/sc2reader-0.7.0rc0-py3.6.egg/sc2reader/factories/sc2factory.py", line 85, in load_replay
    return self.load(Replay, source, options, **new_options)
  File "/usr/local/lib/python3.6/site-packages/sc2reader-0.7.0rc0-py3.6.egg/sc2reader/factories/sc2factory.py", line 137, in load
    return self._load(cls, resource, filename=filename, options=options)
  File "/usr/local/lib/python3.6/site-packages/sc2reader-0.7.0rc0-py3.6.egg/sc2reader/factories/sc2factory.py", line 146, in _load
    obj = cls(resource, filename=filename, factory=self, **options)
  File "/usr/local/lib/python3.6/site-packages/sc2reader-0.7.0rc0-py3.6.egg/sc2reader/resources.py", line 271, in __init__
    self._read_data(data_file, self._get_reader(data_file))
  File "/usr/local/lib/python3.6/site-packages/sc2reader-0.7.0rc0-py3.6.egg/sc2reader/resources.py", line 618, in _read_data
    self.raw_data[data_file] = reader(data, self)
  File "/usr/local/lib/python3.6/site-packages/sc2reader-0.7.0rc0-py3.6.egg/sc2reader/readers.py", line 36, in __call__
    ) for i in range(data.read_bits(5))],
  File "/usr/local/lib/python3.6/site-packages/sc2reader-0.7.0rc0-py3.6.egg/sc2reader/readers.py", line 36, in <listcomp>
    ) for i in range(data.read_bits(5))],
  File "/usr/local/lib/python3.6/site-packages/sc2reader-0.7.0rc0-py3.6.egg/sc2reader/decoders.py", line 252, in read_aligned_string
    return self._buffer.read_string(count, encoding)
  File "/usr/local/lib/python3.6/site-packages/sc2reader-0.7.0rc0-py3.6.egg/sc2reader/decoders.py", line 108, in read_string
    return self.read_bytes(count).decode(encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfc in position 74: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/sc2reader-0.7.0rc0-py3.6.egg/sc2reader/scripts/sc2parse.py", line 78, in main
    replay = sc2reader.load_replay(path, debug=True, load_level=2)
  File "/usr/local/lib/python3.6/site-packages/sc2reader-0.7.0rc0-py3.6.egg/sc2reader/factories/sc2factory.py", line 85, in load_replay
    return self.load(Replay, source, options, **new_options)
  File "/usr/local/lib/python3.6/site-packages/sc2reader-0.7.0rc0-py3.6.egg/sc2reader/factories/sc2factory.py", line 137, in load
    return self._load(cls, resource, filename=filename, options=options)
  File "/usr/local/lib/python3.6/site-packages/sc2reader-0.7.0rc0-py3.6.egg/sc2reader/factories/sc2factory.py", line 146, in _load
    obj = cls(resource, filename=filename, factory=self, **options)
  File "/usr/local/lib/python3.6/site-packages/sc2reader-0.7.0rc0-py3.6.egg/sc2reader/resources.py", line 271, in __init__
    self._read_data(data_file, self._get_reader(data_file))
  File "/usr/local/lib/python3.6/site-packages/sc2reader-0.7.0rc0-py3.6.egg/sc2reader/resources.py", line 618, in _read_data
    self.raw_data[data_file] = reader(data, self)
  File "/usr/local/lib/python3.6/site-packages/sc2reader-0.7.0rc0-py3.6.egg/sc2reader/readers.py", line 36, in __call__
    ) for i in range(data.read_bits(5))],
  File "/usr/local/lib/python3.6/site-packages/sc2reader-0.7.0rc0-py3.6.egg/sc2reader/readers.py", line 36, in <listcomp>
    ) for i in range(data.read_bits(5))],
  File "/usr/local/lib/python3.6/site-packages/sc2reader-0.7.0rc0-py3.6.egg/sc2reader/decoders.py", line 252, in read_aligned_string
    return self._buffer.read_string(count, encoding)
  File "/usr/local/lib/python3.6/site-packages/sc2reader-0.7.0rc0-py3.6.egg/sc2reader/decoders.py", line 108, in read_string
    return self.read_bytes(count).decode(encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfc in position 74: invalid start byte

@dsjoerg
Copy link
Collaborator

dsjoerg commented Sep 1, 2017

@stevemao if you post the replay somewhere I'll give it a try on my branch and see if there's something easy to fix. By the way, what kind of replay was this? (SC2 LotV? Ladder? 1v1?)

@stevemao
Copy link

stevemao commented Sep 4, 2017

Try this one:
http://lotv.spawningtool.com/29785/

@dsjoerg
Copy link
Collaborator

dsjoerg commented Sep 4, 2017

Hi @stevemao I just uploaded that replay to GGTracker, which indicates that my branch https://github.com/ggtracker/sc2reader/tree/upstream can parse the replay OK. http://ggtracker.com/matches/7166787

@Oneoeigh
Copy link

Hi @dsjoerg I just tried your branch and got the similar error:
"UnicodeDecodeError: 'utf8' codec can't decode byte 0xfc in position 27: invalid start byte"
The platform is OSX 10.10.2, and the python version is 2.7.6.
Besides I also tried upload the replay to ggtracker.com and it worked well, so I guess maybe there are some problems about compatibility.

BTW: In the "creeptracker.py" there is some code that do these kind of things: "from Image import xxx" that may not work on many computers, which can issue an error "No module named Image". To me I think this may be better for compatibility:

try:
    from Image import xxx
except:
    from PIL.Image import xxx

dsjoerg referenced this issue in ggtracker/sc2reader Sep 18, 2017
@dsjoerg
test demonstrating that we can parse the replay mentioned in #190
a15a5f5
@dsjoerg
Copy link
Collaborator

dsjoerg commented Sep 18, 2017

OK this is crazy/interesting. I committed a test to the suite (ggtracker@a15a5f5) that demonstrates that sc2reader can parse the replay. And yet when I run sc2parse on the replay I get the same failure that @stevemao shows. Tomorrow I'll look closer and figure out what's going on, unless someone beats me to it.

@dsjoerg
Copy link
Collaborator

dsjoerg commented Sep 19, 2017

OK @stevemao @MLLeKander everything is working.

In my environment, it turned out to be a PYTHONPATH problem that made it so that I was sometimes accidentally running the wrong version of sc2reader!

Once I fixed things so that I was running the correct version, all replays are parsed successfully. As I mentioned yesterday, I committed a test the demonstrates that sc2reader can parse the replay provided by @stevemao (http://lotv.spawningtool.com/29785/).

I also just tried the original replay provided by @MLLeKander (http://lotv.spawningtool.com/10851/) and it also parsed fine for me, and I uploaded it to GGTracker (http://ggtracker.com/matches/6447606) as well.

I'm leaving this issue open, because this issue is about GraylinKim/sc2reader, not ggtracker/sc2reader.

@dsjoerg
Copy link
Collaborator

dsjoerg commented Sep 19, 2017

And by the way everyone, please use https://github.com/ggtracker/sc2reader/tree/upstream and not https://github.com/dsjoerg/sc2reader/tree/upstream !!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@MLLeKander @dsjoerg @stevemao @Oneoeigh