Convert GDI to ISO

Discussion in 'Sega Dreamcast Development and Research' started by gorgyrip, Dec 5, 2016.

  1. SiZiOUS

    SiZiOUS Spirited Member

    Joined:
    Mar 27, 2009
    Messages:
    111
    Likes Received:
    140
    I'm using Windows 7 x64.
    OK, let me check this particular ISO again. I'll post back the results here.
     
  2. SiZiOUS

    SiZiOUS Spirited Member

    Joined:
    Mar 27, 2009
    Messages:
    111
    Likes Received:
    140
    Still not OK. WARNING.DA files are different after shrinking, for the Demolition Racer - No Exit Demo Disc v0.800 (2000)(Infogrames)(NTSC)(US)[!] release:
    • Original: f08119a0
    • Shrinked: 5b2e42e5
     
  3. -=FamilyGuy=-

    -=FamilyGuy=- Site Supporter 2049

    Joined:
    Mar 3, 2007
    Messages:
    2,925
    Likes Received:
    753
    I mean it's normal that they are different. It actually points to before track05, but after track03, so the data that's being read isn't clear to me, probably the end of track03.

    In other words, .da files are just pointers to the cdda tracks. GDItools doesn't consider the section with audio, so it can have weird, unaccounted for, behavior that is different for the shrinked track03 or the full one.

    Those files aren't used by the game anyways.

    I should detect those kind of files and extract them as empty files I think.
     
    Last edited: Oct 21, 2017
    SiZiOUS likes this.
  4. SiZiOUS

    SiZiOUS Spirited Member

    Joined:
    Mar 27, 2009
    Messages:
    111
    Likes Received:
    140
    Sounds clear to me, thank you :)
     
    -=FamilyGuy=- likes this.
  5. Mrneo240

    Mrneo240 Peppy Member

    Joined:
    Sep 15, 2017
    Messages:
    363
    Likes Received:
    343
    Wait what? so is that true of most .DA files in the root directory? a file that just references the LBA of a GDDA track?
     
  6. -=FamilyGuy=-

    -=FamilyGuy=- Site Supporter 2049

    Joined:
    Mar 3, 2007
    Messages:
    2,925
    Likes Received:
    753
    Afaik yes. You can test this with GDIShrink by looking at its LBA, if it's an audio track, it should correspond to the lba of a track in the .GDI file.

    Edit: See my later post for clarifications on this, I might have been wrong.
    Thanks for the testing, I'll try and do something for the DA files and then include the option to shrink in the main!

    Do you have a GDEmu? If not I'll have to test some games on mine.
     
    Last edited: Oct 21, 2017
  7. -=FamilyGuy=-

    -=FamilyGuy=- Site Supporter 2049

    Joined:
    Mar 3, 2007
    Messages:
    2,925
    Likes Received:
    753
    Ok so I did some tests on a similar 5-tracks disc with a "WARNING.DA" track. It seems it's not as I though a simple reference to track04, it's rather an audio file in the data-track filesystem. In my case it's the 4s audio clip "This disc is for use only on a SEGA Dreamcast" track. My track04 is 6s long, but it's this same audio track +2s of silence.

    The warning.da extracted from the original GDI is the proper one, you can even convert it to wav using raw2wav if you feel like listening to it.

    I think that some developers always put at least that file as padding in track03, as I think the SDK said there should be at least one file per track, and a track should always be 4s or more on a cd. Now we don't need this track, and it'd be complicated to keep it in a shrinked dump as it's at the end of track03, but after the zero-padding. I think that for the purpose of shrinking, it's good enough to remove it.

    Could you confirm this for me @SiZiOUS?
    Code:
    from gditools import GDIfile
    with GDIfile('/home/user/folder/disc.gdi') as gdi:
        print gdi.get_record('warning.da')
        print '\n',gdi._gdi
    
    The ouput of that code would be useful.

    Also, if you guys know of other games with DA tracks in the filesystem, beside warning.da, I'd like to know!

    Thanks again for your help @SiZiOUS!


    [EDIT]
    I just tested with 5 games totalling 7.1 GB, and it shrinked to 3.6 GB. I'm gonna test in GDEMU soon, but it's promising!

    [EDIT2]
    All games confirmed working in GDEMU flawlessly!
     
    Last edited: Oct 21, 2017
  8. Mrneo240

    Mrneo240 Peppy Member

    Joined:
    Sep 15, 2017
    Messages:
    363
    Likes Received:
    343
    ill have more for you soon, still wrapped up with other stuff but Oooga Booga has CRAP.DA.

    i've been trying to crack this game for a while now and im still losing :(
     
  9. Karnako

    Karnako Newly Registered

    Joined:
    Oct 30, 2017
    Messages:
    4
    Likes Received:
    1
    First of all I would like to thank @-=FamilyGuy=- for the amazing job on gditools.

    Long time lurker but joined to share one error I found today. I've been using GDIshrink on all Tosec images - using the latest version committed on sourceforge. For the first time I'm receiving an error message after successfully shrinking dozens of images. When trying GDIshrink on Q-Bert I get this error:
    Code:
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "gditools.py", line 752, in gdishrink
        first_file_sector = gdi.get_first_file_sector()
      File "gditools.py", line 359, in get_first_file_sector
        lba = self._sorted_records()[i]['ex_loc']
    IndexError: list index out of range
    Hope this can be of help. Let me know if I should try/test something.

    Thanks!
     
  10. -=FamilyGuy=-

    -=FamilyGuy=- Site Supporter 2049

    Joined:
    Mar 3, 2007
    Messages:
    2,925
    Likes Received:
    753
    Hi and thanks!

    Could you please try and extract this GDI using GDItools, I mean extract all the files from it, and report any error? I'll look into it in the next weeks.
     
  11. Karnako

    Karnako Newly Registered

    Joined:
    Oct 30, 2017
    Messages:
    4
    Likes Received:
    1
    Used -i --extract-all option. "All files were dumped successfully."
     
  12. -=FamilyGuy=-

    -=FamilyGuy=- Site Supporter 2049

    Joined:
    Mar 3, 2007
    Messages:
    2,925
    Likes Received:
    753
    Could you please replace those lines, starting at line 357 of gditools.py:
    Code:
            while lba < self._gdi[-1]['lba']:
                i -= 1
                lba = self._sorted_records()[i]['ex_loc']
    
    for thoses
    Code:
            while lba < self._gdi[-1]['lba']:
                i -= 1
                print i,' \t',self._sorted_records()
                lba = self._sorted_records()[i]['ex_loc']
    
    and post the output? The extracted file list would also be helpful (sorttxt.txt is fine).

    I have a feeling you know python, but if not, be careful about the indentation, it is important.

    I'll try and find some time to debug it the weekend after the next if the results of this test don't help me figure it out.

    Thanks for the help!
     
    fafadou likes this.
  13. Karnako

    Karnako Newly Registered

    Joined:
    Oct 30, 2017
    Messages:
    4
    Likes Received:
    1
    I'm no expert in python but I can follow instructions! :D

    I hope I did it right - there are 3 files there: the output from --extract-all; output from gdishrink (I think that's the one you're looking for) and the output of a list of extracted files under the folder data.

    File was too big to upload here so had to use an external site - https://ufile.io/xs6p9
     
  14. darcagn

    darcagn Site Supporter 2013, Site Supporter 2014

    Joined:
    May 13, 2007
    Messages:
    459
    Likes Received:
    87
    Very cool job FamilyGuy on this utility. I was thinking the other day about how a utility could shrink GDIs like this, but figured the 4-track approach could cause compatibility problems and I would be too lazy to test a bunch of games for it. It sucks that the GDEmu doesn't support even the simplest of compression algorithms that are good on long blocks of zeros, because that would eliminate this whole shrinking business completely.

    Have you tried any later games that have a bunch of copy protection? What happens if a game were to read the disc at an LBA with missing data?

    You and Sizious are awesome; between the two of you I have very little reason to need my WinXP-based Dreamcast tools VM. I am nearly all native macOS. My dream is for these utilities to be so robust that any game can be converted from GDI to MIL-CD with a script that works on Windows, macOS, and Linux/BSD.
     
  15. -=FamilyGuy=-

    -=FamilyGuy=- Site Supporter 2049

    Joined:
    Mar 3, 2007
    Messages:
    2,925
    Likes Received:
    753
    I can't check it out properly right now, but I'll try to do it the weekend after the next.

    If I had to bet, I'd say it's probably an issue with the handling of the files that are outside of the last track, similar to what sizious saw, or the TOC of the game being obfuscated with weird values in the little endian version of the TOC. Let's consider gdishrink to still be in beta.

    Could you tell me how many games you successfully shrinked and how much space you saved vs total original size?

    Thanks again!

    I'll try to address those points one by one. I'm on mobile so it's hard to break it in parts manually.

    Thanks! My initial idea was to put a dummy track04 to mimic the normal format, but someone (can't remember who, maybe @S4pph4rad?) somehow told me 4 tracks was working fine.

    I don't know what the behaviour would be if a game was to request data from a location that was pruned. It'd depend on mnemo's/deunan's implementation details. I think the cleanest option would be to return zeros.

    About compression, even GZ might be taxing of the FPGA if done continuously in real time, and it might use a lot of precious logical gates.

    If you have GDIs with those copy protections, it'd be interesting to test. I don't have many GDIs at hand myself, and I have very little free time these days, but if anyone want to take the time to test images, I'm very interested in the results. The idea is to improve it until it's bullet proof!

    If you can get me the Padus SDK source, or a clear reverse engineered specification of the CDI format, we would have all the tools to do it. I do all in Python, except building the CDI and downsampling, nowadays. Downsampling could be automated quite easily, using wine of need be.
     
    Last edited: Nov 10, 2017
  16. darcagn

    darcagn Site Supporter 2013, Site Supporter 2014

    Joined:
    May 13, 2007
    Messages:
    459
    Likes Received:
    87
    Had a little bit more time to play with this...

    I took the full TOSEC set (U.S.A. region, games only/no demo discs or utilities discs) and deleted any dumps with different audio offsets and deleted any dumps that were old revisions of games that had multiple revisions.

    Zipped filesize: 147,914,502,726 bytes -- 147.915 gigabytes
    Unzipped filesize: 309,677,740,465 bytes -- 309.7 gigabytes
    Shrunk filesize*: 168,030,156,095 bytes -- 168.03 gigabytes

    With regular ZIP compression, we get 48% filesize, and with gdishrink we get 54% filesize. Not too bad!

    * Two games were unable to be gdishrink'd. The gdishrink function crashed. These are Q-bert and Conflict Zone: Modern War Strategy.

    I took a look at the structure of these games:
    Q-bert is 13 tracks and Conflict Zone is 5 tracks.
    Both have extremely small final tracks (Q-bert's is only 689KB and Conflict Zone's is 3.03MB -- these are the raw bin filesizes)

    Conflict Zone appears to have 1ST_READ.BIN in track05.bin, but all of the rest of the game's data is in track03.bin! track04.raw is just a dummy audio track (the game doesn't use CDDA), so the data is still pretty much at the end of the disc regardless.

    However, Q-bert appears to have NO FILES in track13.bin. ALL of the game's data is in track03.bin. And since this game has a bunch of CDDA tracks, it's definitely not padded properly.

    So neither of these games can be shrunk using this method without some more creativity.

    When I get some time to test I'll mess around with later games that implemented more rigorous copy protection checks. If you need any GDIs you know where to find me ;)

    Personally I'd be happy with it if the output were just ISO/WAV files and the python tool printed the necessary cdrecord commands to the terminal.

    But in the interest of the idea of a "full tool" that takes you from GDI to CDI, there are some references you could use for CDI files implementation:
    - CDIrip source code available here https://sourceforge.net/projects/cdimagetools/files/CDIRip/0.6.3/
    - iceGDROM CDI-to-GIO conversion utility source code https://github.com/zeldin/iceGDROM/tree/master/tools/source
    - img4dc source code https://sourceforge.net/projects/img4dc/
     
    Karnako, fafadou and -=FamilyGuy=- like this.
  17. Karnako

    Karnako Newly Registered

    Joined:
    Oct 30, 2017
    Messages:
    4
    Likes Received:
    1
    I shrinked all the US TOSEC set plus the European exclusive games (also from TOSEC). I'm short on disk space so I took the risk of deleting original files...so I can't compare it properly. :rolleyes: I have a 189Gb as a result which seems to be in line to what @darcagn posted given the difference on the game list.

    Now I'm to test the results on my Dreamcast. I know the only way to be 100% sure the shrinked copy works is by playing the full game to the end. But what would be a reasonable test? Just loading the games or do I need to play for some time? I'm not familiar on how the games are loaded (if Dreamcast do some sort of check when loading, etc.)

    Interesting...my Conflict Zone shrinked without errors. However it's down to just 5Mb...seems I'll have to test this one.
     
    -=FamilyGuy=- likes this.
  18. Blockhead

    Blockhead Member

    Joined:
    Mar 5, 2018
    Messages:
    17
    Likes Received:
    5
    Hi @-=FamilyGuy=-,
    I tried this code and ended up with an .iso file amongst the others after shrinking, is this correct?

    The reason I’m asking is that if i try to add the resulting folder to an SD card using the GDEMU_SD app, it shows up blank for the name, making it not very practical as once you have added many files you will never know which game is which (any renaming is removed after each sync.)

    GDI Rebuild works fine for this purpose, but I have encountered some incompatible titles and was hoping your solution would be a bit more robust.

    Thanks for any help you can provide here!
     
  19. -=FamilyGuy=-

    -=FamilyGuy=- Site Supporter 2049

    Joined:
    Mar 3, 2007
    Messages:
    2,925
    Likes Received:
    753
    Some games don't include the proper details in the bootsector, in which case you can edit the title manually in GDEmu_SD (I used to keep a copy of the list it creates and reuse it for further sync).

    GDIShrink won't change those metadata, it should never change any game-accessible data, but once it's in iso format you can change it manually in the 3rd track and the change will stick. I don't remember the exact offset of the game title. You can use another GDIShrinked game to figure out the offset of each of the metadata entry, it'll be at the very beginning of the first sector of track 03. Edit it with an Hex editor.

    At the output, you should get a GDI file with ISOs for the data tracks and raw files for the audio tracks.

    Please test if the game plays properly and report here for further support if it doesn't.
     
    Last edited: Mar 20, 2018
    Mrneo240 likes this.
  20. Blockhead

    Blockhead Member

    Joined:
    Mar 5, 2018
    Messages:
    17
    Likes Received:
    5
    Thanks for the clarifications.

    Just to follow up, I tried GDIshrink on one one the 3 games not working for me with RebuildGDI script, and it ran just fine.
    I’m now wondering if there is reason to redo all my .gdi files from scratch using GDIShrink instead, if they end up closer to the original .gdi files (meaning less potential issues with GDEMU)?
     

Share This Page