PS2 HDD APA Format specs

Discussion in 'Sony Programming and Development' started by rs1n, Oct 9, 2019 at 11:06 PM.

  1. rs1n

    rs1n Active Member

    Joined:
    Sep 5, 2016
    Messages:
    26
    Likes Received:
    4
    Hi,

    I was wondering if anyone has any documentation on the APA format for the PS2 HDD. One thing I keep running into when transferring my ISO rips onto an HDD is that partitions are not sequential. This often breaks games, in particular during FMVs. I am thinking about getting started on PS2 development, and would like to know if there is any documentation available on the APA format.
     
  2. wisi

    wisi Rising Member

    Joined:
    Apr 16, 2016
    Messages:
    56
    Likes Received:
    76
  3. sp193

    sp193 Site Soldier

    Joined:
    Mar 29, 2012
    Messages:
    2,235
    Likes Received:
    1,077
    It is a normal thing to have a "partition" made up of multiple parts. You need to handle that, if you are writing your own filesystem driver.

    On a side note, the HDLoader format breaks when you optimize the disk because the LBAs of each partition are hardcoded.
     
    pool7 likes this.
  4. rs1n

    rs1n Active Member

    Joined:
    Sep 5, 2016
    Messages:
    26
    Likes Received:
    4
    I understood as much. Please correct me if I'm wrong, but generally speaking a partition is made of X number of parts of equal size plus possibly one "ending" part of smaller size (like a "remainder" portion), correct? Ideally I would want these parts to be located in sequential physical order. When I scan my PS2 HDD, it appears that a single partition's parts are sometimes scattered about (in terms of physical address), and several partitions might have their parts intermingled. This was from using WinHiip. I would like to add that I did not "optimize" or repair anything; it seems WinHiip does this on its own as all I did was install my partitions. Does HDLGameInstaller do this?

    Is there software that does not do this optimization? It's this very reason that I am even looking at writing my own partition installer. (Or perhaps even more likely there is something I still do not understand about the APA format that I need to learn.) Even having small gaps between two physically neighboring partitions (as long as their parts are in the right order) is more ideal to me than trying to recover that space at the cost of all the parts of that partition being spread around.
     
  5. rs1n

    rs1n Active Member

    Joined:
    Sep 5, 2016
    Messages:
    26
    Likes Received:
    4
    I guess I could start here for now. Reverse engineering the APA and PFS specs from someone else's code might be more than I bargained for :)
     
  6. fanhais

    fanhais Member

    Joined:
    Mar 10, 2018
    Messages:
    14
    Likes Received:
    2
    would be nice if someone genius could patch the limitation of APA drive on getting HDDOSD load more content and faster but can be very hard to happen
     
  7. Masamune3210

    Masamune3210 Rising Member

    Joined:
    Jul 3, 2016
    Messages:
    60
    Likes Received:
    1
    Pretty sure the slow loading times in HDDOSD is caused by the fact that apa doesn't have any master partition table or anything like that, so the osd has to scan the drive from the first partition to the last, following the chain of partitions until it finds the last one every time it wants to display all of the games installed

    HDDOSD wasnt really ever meant to deal with anything bigger than 8gb or so of potential storage
     
  8. sp193

    sp193 Site Soldier

    Joined:
    Mar 29, 2012
    Messages:
    2,235
    Likes Received:
    1,077
    Yes. Each part is also limited to a maximum of 1/32th of the disk's capacity, and must be a multiple of 128MB. Each main partition may only have up to 64 sub-partitions associated with it.

    Sony has a document about the APA format and how to allocate space, in the PS2SDK. You may want to locate a copy and read it.

    Even if it is not something that is normally done, it is something that is permissible under APA. Technically the APA driver needs to just find sufficient space and to allocate it. Fragmentation is normal.

    At some point, it probably will.

    There exists the Sony HDD Utility Disc, or HDDChecker (the homebrew equivalent). However, this function was designed more for reducing external fragmentation that would otherwise prevent the remaining space on a nearly-full disk from becoming unusable due to the maximum sub-partition number.

    Although it would seem to literally defragment HDLoader-format games, remember that PFS was the official filesystem for the Sony PlayStation 2 (fragmentation of files within PFS partitions would not be reduced).

    Okay, but why would this be a problem?

    This is true, but there is a little more to it. When you have a large disk, you will naturally have many partitions. Due to the design of APA and the IOP only having so much memory, the APA driver's cache will fill out and then it will become ineffective; since it cannot possibly cache the locations of all partitions on the HDD, it will be (inconveniently) evicting required entries as the HDD browser scans the disk and partitions are being opened (the HDD browser will access each partition to load its icon).

    The official HDD was 40GB in capacity and it was originally designed before 48-bit LBA became a thing. This design flaw of the standard APA driver starts happening with perhaps around 160GB of space, also subjective to the parameters specified to the HDD.IRX driver.
     
    pool7 likes this.
  9. AKuHAK

    AKuHAK Spirited Member

    Joined:
    Jul 25, 2012
    Messages:
    179
    Likes Received:
    53
    You also should deal with HDLoader format which is unofficial and is not documented in ps2sdk. As @sp193 said this format has more limitations than normal apa specs. You faced with fragmentation problem, which is common in one or another manner for other filesystems as well. As I know there is no developed any program for defragmenting hdd disc with games in hdloader format.
    You can use HDD Checker developed by @sp193, it has the feature called disk optimization (which in fact optimizes disk in the manner you need). But this feature does not work for installed games due to HD Loader format restrictions. Correction: this feature works for installed games as well. You can take hdd checker source code if you really wish to dig out.

    HD Loader format is not documented much though. You can find some documentation here. In fact, it will be useful to provide some sort of patch for opl to calculate most of the fields from apa header and ignore HD Loader header as much as possible.

    Also for saving space (save space also will reduce fragmentation) there is an idea to use some sort of compressing (CSO compressed iso for example). Here.
     
    Last edited: Oct 14, 2019 at 12:59 AM
    pool7 likes this.
  10. sp193

    sp193 Site Soldier

    Joined:
    Mar 29, 2012
    Messages:
    2,235
    Likes Received:
    1,077
    HDDChecker can defragment these partitions properly, as it will update the recorded LBAs. Only the HDD Utility Disc will not, since the HDLoader game format broke APA.

    But what is the purpose of reducing fragmentation here? It's not like fragmentation is like fragmentation on your PC filesystems like FAT32 etc, whereby the clusters can only be in 32 or 64KB and defragmentation will really improve performance. Here, the minimum partition size is 128MB, but your game will probably be made up of much larger partitions (unless your disk is really small or the game was installed when the disk is nearly full).
     
    pool7 and AKuHAK like this.
  11. fanhais

    fanhais Member

    Joined:
    Mar 10, 2018
    Messages:
    14
    Likes Received:
    2
    Nice to know these things and a very noob question from someone is an normal user one thing I learn and badly was pascal and some visual basic in school.
    It is possible do some trick for HDD-OSD have or build a master partition or like gamecache list in opl but built for HDDOSD for read the content without hddosd scan all partitions and show only a master partition of game or app ?
     
  12. Masamune3210

    Masamune3210 Rising Member

    Joined:
    Jul 3, 2016
    Messages:
    60
    Likes Received:
    1
    Likely not considering the massive amount of modding it would take to hddosd
     
  13. fanhais

    fanhais Member

    Joined:
    Mar 10, 2018
    Messages:
    14
    Likes Received:
    2
    yeah another question, is possible port the XMB to osdsyss ? like psx dvr has i read out there in another forums
     
  14. AKuHAK

    AKuHAK Spirited Member

    Joined:
    Jul 25, 2012
    Messages:
    179
    Likes Received:
    53
    My bad, didn't realize that. I guess HDD Checker should be the solution for OP.

    @rs1n I don't think that FMV's can stutter because of HDD fragmentation. Internal HDDs have much faster access speed than DVD or CD disks. Games normally are divided not more than in 5 parts, and it can be no more than 3 times when such division trap in movies. If you expected stuttering it is more likely something else.
     
    Last edited: Oct 14, 2019 at 5:47 AM
    AlGollan84 likes this.

Share This Page