Redump Serial vs Internal Serial

[kikmon]

Hi, I see this topic has been discussed some time ago, but I had a few questions.
(Trying to fix some PSX games not scanning properly :) )
If I understand properly, Redump has 2 kind of serials.
The Internal, that is really in the disk data (and I think this is the one Retroach will find), and the Serial that appears on the disk labels / box.
If the internal has the same value as the regular one, internal is just omitted by Redump.
So to fix the serial scanning of these games that have an internal serial, we need the information in the RDB, and for, that we need to get from Redump's dat.
Is there a way to have the internal serial in the Redump dat ?
I see the tool calls http://redump.org/datfile/psx/serial,version to get the serial. Is there a URL to also get the internal one ?
If we can get this info in the rdb, then the scanning logic of RA could be altered to take the 'internal serial' into account if present.

[RobLoach]

All data that's grabbed from redump is build into .dat files over at:
https://github.com/libretro/libretro-database/tree/master/metadat/redump

This includes the serial number. For example...

game (
	name "Crazy Taxi (USA)"
	description "Crazy Taxi (USA)"
	region "USA"
	serial "51035"
	rom ( name "Crazy Taxi (USA) (Track 1).bin" size 1058400 crc F8BB5B3C md5 BE87B59616171C57D9FD252ACDDA806B sha1 CF38F6C262F304729870F443FCE69FFD182214E1 serial "51035" )
)

[kikmon]

Thanks for the followup.
Yes, but there's only one serial in this example, and the 'serial vs internal serial' plagues Multi Disc games mostly.

Let's take an example that RA is unable to scan: PS1 game Alive (Japan) (Disk 2)
On Redump, (http://redump.org/disc/22867/) the serial is 'SLPS 01528' (printed on Disk Label) and Internal Serial is 'SLPS-01527' (the one found in the track data)
The dat file you linked shows 2 entries for Alive (Japan) (Disk 2), featuring Serial SLPS 01528 and SLPS 01528-1, both derived from the regular Serial, but there is no trace of the internal serial (SLPS-01527)
When RA extracts the serial from the disk, it finds ... SLPS-01527, and since it's a multidisc game, as a good soldier, it adds -1, and then performs a lookup in the database for SLPS-01527-1, which is unknown in the DB

For the scanning to work, we'd need some knowledge of this 'Internal Serial'. It's not present in the Redump dat we're downloading, hence my question about the existence of a Redump Download URL that can return both serials. :)

[RobLoach]

Ah, I understand. That makes sense, thanks a lot. Curious about this as well. RetroArch requires one-rom to one-entry, so we would somehow need to split the entries.

[kikmon]

Yep, Once we know SLPS-01527 is the internal one, we can maybe build an entry for SLPS-01527-1
But in order to have the right data, with need Redump to cooperate.
I've looked at my other scan failures, and here's another funny one
http://redump.org/disc/32630/ (SLPM 86306 vs SLPS-01487)
It's a single disc game, but the serial on the label doesn't match the internal serial :)

[RobLoach]

@pkos may be interested in some of this too. He had been working on this repo: https://github.com/pkos/CRoSG which brings in a few additional serials.

Tested out building it into the .dat files over at #53 . Could use some thoughts and testing.

[kikmon]

Interesting.
If I read this well, it's mainly to add chd crc scanning ? (I'm not seeing the serials in the dat)
Do we have a privileged communication channel with redump to get the missing info ?

[kikmon]

Ok, got in touch with the kind people at Redump.
Internal Serials are not part of the database, only comments, so we can't get them in the dat. There's a chance to have them in the future, but not anytime soon, so we must find another way to recognize these games.

[pkos]

In most system disc media there are enough clues to find the serials, however others system media simply never recorded a serial in the data, so the hook is missing.  CRoSG is an alternative for such media, in fact we've already implemented internal CHD SHA in the data files. Sent from Yahoo Mail on Android On Thu, Aug 11, 2022 at 10:40 AM, ***@***.***> wrote: Ok, got in touch with the kind people at Redump. Internal Serials are not part of the database, only comments, so we can't get them in the dat. There's a chance to have them in the future, but not anytime soon, so we must find another way to recognize these games. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[pkos]

Rob can also confirm that when no serial is detected, the next step is CRC, even large ISO and BIN/CUE.  These data files Rob loads cover both serials and CRC, correct Rob?  Next step could be rated internal CHD SHA which is precalculated and stored in CHD creation from such original disc media. Sent from Yahoo Mail on Android On Thu, Aug 11, 2022 at 10:50 AM, Peter ***@***.***> wrote: In most system disc media there are enough clues to find the serials, however others system media simply never recorded a serial in the data, so the hook is missing.  CRoSG is an alternative for such media, in fact we've already implemented internal CHD SHA in the data files. Sent from Yahoo Mail on Android On Thu, Aug 11, 2022 at 10:40 AM, ***@***.***> wrote: Ok, got in touch with the kind people at Redump. Internal Serials are not part of the database, only comments, so we can't get them in the dat. There's a chance to have them in the future, but not anytime soon, so we must find another way to recognize these games. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[pkos]

Cranking up the speed it goes CRC, Serial, but internal SHA is extremely faster. Sent from Yahoo Mail on Android On Thu, Aug 11, 2022 at 11:00 AM, Peter ***@***.***> wrote: Rob can also confirm that when no serial is detected, the next step is CRC, even large ISO and BIN/CUE.  These data files Rob loads cover both serials and CRC, correct Rob?  Next step could be rated internal CHD SHA which is precalculated and stored in CHD creation from such original disc media. Sent from Yahoo Mail on Android On Thu, Aug 11, 2022 at 10:50 AM, Peter ***@***.***> wrote: In most system disc media there are enough clues to find the serials, however others system media simply never recorded a serial in the data, so the hook is missing.  CRoSG is an alternative for such media, in fact we've already implemented internal CHD SHA in the data files. Sent from Yahoo Mail on Android On Thu, Aug 11, 2022 at 10:40 AM, ***@***.***> wrote: Ok, got in touch with the kind people at Redump. Internal Serials are not part of the database, only comments, so we can't get them in the dat. There's a chance to have them in the future, but not anytime soon, so we must find another way to recognize these games. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[kikmon]

Thanks for joining :)
As far as I see, there are 2 issues mixing up here:

• I've played a bit in the RA scanner recently to improve PSX scanning, and RA will only fallback on crc if a serial cannot be extracted. So if a serial is found (like in the case of internal serials different from the one on the box), RA will continue doing the DB lookup and fail. Maybe we should do the fallback on the crc if the whole lookup fails.
• Then, we have an issue with the RDB, as they are built using Serial as the key, several entries are missing, and crc lookup will fail (A Ressha de Ikou 4 - Evolution (Japan) is a good example). if we build the rdb with crc key instead, I think it would work better, as some serial would be missed on the first lookup, but then crc will find them.

[pkos]

Perhaps let us know how many PSX games are matching to serials and which are going to CRC, that can be retested. Sent from Yahoo Mail on Android On Thu, Aug 11, 2022 at 11:38 PM, ***@***.***> wrote: Thanks for joining :) As far as I see, there are 2 issues mixing up here: - I've played a bit in the RA scanner recently to improve PSX scanning, and RA will only fallback on crc if a serial cannot be extracted. So if a serial is found (like in the case of internal serials different from the one on the box), RA will continue doing the DB lookup and fail. Maybe we should do the fallback on the crc if the whole lookup fails. - Then, we have an issue with the RDB, as they are built using Serial as the key, several entries are missing, and crc lookup will fail (A Ressha de Ikou 4 - Evolution (Japan) is a good example). if we build the rdb with crc key instead, I think it would work better, as some serial would be missed on the first lookup, but then crc will find them. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[pkos]

The log files can be enabled for data on your complete scan if you enable logging in RA. Sent from Yahoo Mail on Android On Fri, Aug 12, 2022 at 10:33 AM, Peter ***@***.***> wrote: Perhaps let us know how many PSX games are matching to serials and which are going to CRC, that can be retested. Sent from Yahoo Mail on Android On Thu, Aug 11, 2022 at 11:38 PM, ***@***.***> wrote: Thanks for joining :) As far as I see, there are 2 issues mixing up here: - I've played a bit in the RA scanner recently to improve PSX scanning, and RA will only fallback on crc if a serial cannot be extracted. So if a serial is found (like in the case of internal serials different from the one on the box), RA will continue doing the DB lookup and fail. Maybe we should do the fallback on the crc if the whole lookup fails. - Then, we have an issue with the RDB, as they are built using Serial as the key, several entries are missing, and crc lookup will fail (A Ressha de Ikou 4 - Evolution (Japan) is a good example). if we build the rdb with crc key instead, I think it would work better, as some serial would be missed on the first lookup, but then crc will find them. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[pkos]

Rob could we get a quick patch to make sure the overall RDBs have a latest update date stamp.  Redump is much more active about dats. Sent from Yahoo Mail on Android On Fri, Aug 12, 2022 at 10:42 AM, Peter ***@***.***> wrote: The log files can be enabled for data on your complete scan if you enable logging in RA. Sent from Yahoo Mail on Android On Fri, Aug 12, 2022 at 10:33 AM, Peter ***@***.***> wrote: Perhaps let us know how many PSX games are matching to serials and which are going to CRC, that can be retested. Sent from Yahoo Mail on Android On Thu, Aug 11, 2022 at 11:38 PM, ***@***.***> wrote: Thanks for joining :) As far as I see, there are 2 issues mixing up here: - I've played a bit in the RA scanner recently to improve PSX scanning, and RA will only fallback on crc if a serial cannot be extracted. So if a serial is found (like in the case of internal serials different from the one on the box), RA will continue doing the DB lookup and fail. Maybe we should do the fallback on the crc if the whole lookup fails. - Then, we have an issue with the RDB, as they are built using Serial as the key, several entries are missing, and crc lookup will fail (A Ressha de Ikou 4 - Evolution (Japan) is a good example). if we build the rdb with crc key instead, I think it would work better, as some serial would be missed on the first lookup, but then crc will find them. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>