bdm MBR add support for VBR

[DKWDRV]

Latest bdm exFAT drivers fail on some FAT32 drivers. Looks like bdm has forgot to add support for VBR.

On invalid MBR records it is trying to mount invalid partitions. This restores that functionality. Many of the such devices are formatted this way to bypass FAT32 max limit of 32GB in Windows.

Alternatively someone needs to add better support for VBR. Sometimes a VBR can be valid and not even have MBR signature in the end of first sector. For now this will do.

The original authors of bdm are welcomed to give their insight.

resolves DKWDRV/DKWDRV#82

Possible impacts OPL since a long time already?

resolves ps2homebrew/Open-PS2-Loader#1311
resolves ps2homebrew/Open-PS2-Loader#1295
resolves ps2homebrew/Open-PS2-Loader#1271
resolves ps2homebrew/Open-PS2-Loader#1493

[DKWDRV]

@fjtrujy @uyjulian @rickgaiser
Can we review this and fix what is needed to support all VBR (is impacting a lot of devices).

[PC-maniak]

bump! this will help with a lot of things and solve a lot of problems

[rickgaiser]

The only job of the MBR driver is to split 1 single block device (a USB stick for instance), into 1 or more smaller block devices (partitions). This can also be done recursively, for instance with extended boot records (EBR).

What exactly is the role of the VBR, and by do we need it in our case?
https://wiki.osdev.org/Volume_Boot_Record
"A Volume Boot Record (VBR) is the first sector of a partition (unlike MBR which is the first sector of a hard disk). A VBR (just like a MBR) also contains some code and data, but it's far less standard."

I would expect a USB stick to be formatted as:

• no partitioning, just FAT32/exFat directly
• MBR (4 entries, possibly recursively linking to more entries)
• GPT
  These 3 cases should be supported.

Then inside each partition (with or without VBR ?) I would expect the FAT32/exFat driver to be able to process the contents.

Does the MBR driver fail to properly find all available partitions?
Or does the exFat driver fail to process the partition?
Or am I missing something?

[DKWDRV]

The only job of the MBR driver is to split 1 single block device (a USB stick for instance), into 1 or more smaller block devices (partitions). This can also be done recursively, for instance with extended boot records (EBR).

MBR driver is doing some partitions checks in part_driver_mbr.c which are not correct and trying to mount invalid MBR.

What exactly is the role of the VBR, and by do we need it in our case? https://wiki.osdev.org/Volume_Boot_Record "A Volume Boot Record (VBR) is the first sector of a partition (unlike MBR which is the first sector of a hard disk). A VBR (just like a MBR) also contains some code and data, but it's far less standard."

Think of it as just one single partition right at start and with as much sectors as the device.

I would expect a USB stick to be formatted as:

* no partitioning, just FAT32/exFat directly

This the case of VBR here pretty much, which is making a lot of devices invalid.

* MBR (4 entries, possibly recursively linking to more entries)

Correct but not everything that looks like MBR is actually MBR.

* GPT

This is not main problem right now.

  These 3 cases should be supported.

Then inside each partition (with or without VBR ?) I would expect the FAT32/exFat driver to be able to process the contents.
In theory we might call "VBR" so to speak each of the valid partitions.

Does the MBR driver fail to properly find all available partitions?

It's even worse! MBR driver is parsing bad and invalid MBR and even trying to mount them! This is because it thinks all MBR are correct as long as they have MBR signature (two bytes at the end of the sector) but it so happens ironically that a VDR might have it too.

Or does the exFat driver fail to process the partition?

The exfat fails to process the partition because it's invalid and it will fail somewhere. Or worse it will make it work sometimes but with very bad max sector count etc (whatever was random garbage in there).
The first thing to do is to try to determine if the MBR is valid but bdm is not doing this. This just restores the same behaviour as it was in old driver. Ironically currect vfat driver is correct on this so you can check it. Really you can check older drivers and you will understand it.

Or am I missing something?

MBR code had a few bugs which make it process invalid or random partitions sometimes.

I have over 8 devices which were formatted by Windows 10/11 all of them having VDR and the bdm drivers acting funny. Depending on what garbage is in partition data they sometimes randomly mount stuff and will corrupt everything.

[DKWDRV]

@rickgaiser here is an example of a perfectly VALID usb device working on older drivers, on Windows and everywhere else. Guess what partitions is MBR driver trying to mount which the exFat drivers later refuse them.

[DKWDRV]

An even better example. Signature is ok, all 4 parts are valid but at minimum an active part is needed. Since it missing the MBR must be considered invalid and default back to VBR which is valid!

[DKWDRV]

Maybe another case here?

[rickgaiser]

Assuming a block device with no partitioning, just FAT32/exFat directly, the intended parsing of block devices is:

• A new block device is registered with bdm_connect_bd (the entire USB stick, usb0p0)
• All registered filesystems are requested if they can handle this block device, 1-by-1, untill 1 of them reports success:
  -- MBR, should return -1
  -- GPT, should return -1
  -- exFat, should mount usb0p0 and return 0

The root cause I think is the MBR driver mounting the exFat partition, becouse of the VBR that looks like an MBR, but isn't.

Instead of failing, your proposed solution lets the MBR driver mounts the exfat partition, and then creates another block device, with the exact same properties as the original block device:

ps2sdk/iop/fs/bdm/src/part_driver_mbr.c

Lines 103 to 112 in 1c6f598

	// Create the pseudo block device for the partition.
	g_part[partIndex].bd = bd;
	g_part_bd[partIndex].name = bd->name;
	g_part_bd[partIndex].devNr = bd->devNr;
	g_part_bd[partIndex].parNr = 1;
	g_part_bd[partIndex].parId = 0; //Can be any type of (ex)FATxx, is parsed later.
	g_part_bd[partIndex].sectorSize = bd->sectorSize;
	g_part_bd[partIndex].sectorOffset = bd->sectorOffset;
	g_part_bd[partIndex].sectorCount = bd->sectorCount;
	bdm_connect_bd(&g_part_bd[partIndex]);

Resulting in:

• MBR mounts usb0p0, and creates usb0p1 with the same dimensions as usb0p0
• exFat mounts usb0p1

In theory this can cause an endless recursive loop, becouse what happens if the MBR driver is requested again if it can handle the newly registered block device? But I guess by luck/accident this psuedo block device will be handled by the exFat driver first.

If the parsing of partitions was done properly, the entire partitions_sanity_check_mbr would not be needed, becouse:

ps2sdk/iop/fs/bdm/src/part_driver_mbr.c

Lines 152 to 153 in 1c6f598

	// If one or more partitions were mounted then return success.
	rval = mountCount > 0 ? 0 : -1;

Would cause the MBR mount to fail.

I think the only missing thing in the current MBR driver are the missing sanity checks for the partition table entries. Can you try with only the following changes?:

        // Ignore partitions that are not active. 0 is empty partition_type.
        if (pMbrBlock->primary_partitions[i].partition_type == 0)
            continue;

        // Sanity check on dimensions
        if((pMbrBlock->primary_partitions[i].first_lba == 0) || (pMbrBlock->primary_partitions[i].first_lba >= bd->sectorCount))
        {
            FreeSysMemory(pMbrBlock);
            return -1;
        }

The first check you already added, but the second check was only present in partitions_sanity_check_mbr

edit: Code above changed so that all 4 entries are valid. So instead of continue; for the sanity check it should return -1;.

[DKWDRV]

Assuming a block device with no partitioning, just FAT32/exFat directly, the intended parsing of block devices is:

This the case we are trying to fix. They seem to be all failing.

* A new block device is registered with `bdm_connect_bd` (the entire USB stick, `usb0p0`)

* All registered filesystems are requested if they can handle this block device, 1-by-1, untill 1 of them reports success:
  -- MBR, should return -1

MBR is trying to parse invalid parts and it's not even checking if they are valid or even if they were actually mounted at all.

  -- GPT, should return -1
  -- exFat, should mount `usb0p0` and return 0

where is the part which checks for FAT/exFAT? part_driver_mbr.c and part_driver_gpt.c do the check for MBR and GPT, where is FAT/exFAT done?
Where is that code that does the three of them in such order?

The root cause I think is the MBR driver mounting the exFat partition, becouse of the VBR that looks like an MBR, but isn't.

Instead of failing, your proposed solution lets the MBR driver mounts the exfat partition, and then creates another block device, with the exact same properties as the original block device:
MBR driver does not fail at all. It still thinks it has managed to mount them and return valid. But all mounts have failed.

ps2sdk/iop/fs/bdm/src/part_driver_mbr.c

Lines 103 to 112 in 1c6f598

	// Create the pseudo block device for the partition.
	g_part[partIndex].bd = bd;
	g_part_bd[partIndex].name = bd->name;
	g_part_bd[partIndex].devNr = bd->devNr;
	g_part_bd[partIndex].parNr = 1;
	g_part_bd[partIndex].parId = 0; //Can be any type of (ex)FATxx, is parsed later.
	g_part_bd[partIndex].sectorSize = bd->sectorSize;
	g_part_bd[partIndex].sectorOffset = bd->sectorOffset;
	g_part_bd[partIndex].sectorCount = bd->sectorCount;
	bdm_connect_bd(&g_part_bd[partIndex]);

Resulting in:

* MBR mounts `usb0p0`, and creates `usb0p1` with the same dimensions as `usb0p0`

* exFat mounts `usb0p1`

In theory this can cause an endless recursive loop, becouse what happens if the MBR driver is requested again if it can handle the newly registered block device? But I guess by luck/accident this psuedo block device will be handled by the exFat driver first.
Why would MBR "be requested again?" Where is the exFAT driver handling it "first"? Is it not bdm that takes each devices and then tries to mount everything?

If the parsing of partitions was done properly, the entire partitions_sanity_check_mbr would not be needed, becouse:

ps2sdk/iop/fs/bdm/src/part_driver_mbr.c

Lines 152 to 153 in 1c6f598

	// If one or more partitions were mounted then return success.
	rval = mountCount > 0 ? 0 : -1;

Would cause the MBR mount to fail.

No, because mountCount is not a safe way to tell. It does not reflect if the mount actually worked. It just increases on every loop but the parts fail to register and then return valid(meaning bdm validated the driver as MBR which is incorrect because bdm is not even checking if the MBR is correct other than the signature bytes) which fails everything.

I think the only missing thing in the current MBR driver are the missing sanity checks for the partition table entries. Can you try with only the following changes?:

        // Ignore partitions that are not active. 0 is empty partition_type.
        if (pMbrBlock->primary_partitions[i].partition_type == 0)
            continue;

        // Sanity check on dimensions
        if((pMbrBlock->primary_partitions[i].first_lba == 0) || (pMbrBlock->primary_partitions[i].first_lba >= bd->sectorCount))
        {
            FreeSysMemory(pMbrBlock);
            return -1;
        }

The first check you already added, but the second check was only present in partitions_sanity_check_mbr

No, we need partitions checks at start because first we need to validate if this is real and legit MBR or not. This must be done before trying to do anything at all with your partitions.
If we check only in the loop then bad things can and will happen depending on garbage data. What if first partition is correct by chance passing all checks? And the rest if failing? You will be going though many incorrect bdm_connect_bd and random garbage data can produce all sort of issues.

If there is exFAT driver than it defaults too after trying MBR and GPT, then the solution is to simply return -1 right after partitions_sanity_check_mbr.

[AKuHAK]

• These 3 cases should be supported.

no partitioning, just FAT32/exFat directly

@rickgaiser
this case is not supported, it was confirmed multiple times in opl issues that fat32/exfat only devices does not mount. After MBR mounting is failing in BDM driver then it does not properly fall back into using all bdm device as partition 0 for some reason.

[rickgaiser]

MBR is trying to parse invalid parts and it's not even checking if they are valid

That's wrong, lets fix that first. I think my proposed solution will do just that. Did you try?

where is the part which checks for FAT/exFAT? part_driver_mbr.c and part_driver_gpt.c do the check for MBR and GPT, where is FAT/exFAT done?
Where is that code that does the three of them in such order?

That's actually what "BDM" should do. MBR and GPT are drivers, they could have been implemented in a separate irx module, but they are integrated into bdm for ease of use. It's done in bdm here:

ps2sdk/iop/fs/bdm/src/bdm.c

Lines 154 to 157 in 608331b

	for (i = 0; i < MAX_CONNECTIONS; ++i) {
	if (g_fs[i] != NULL) {
	if (g_fs[i]->connect_bd(mount->cbd != NULL ? mount->cbd : mount->bd) == 0) {
	M_PRINTF("%s%dp%d mounted to %s\n", mount->bd->name, mount->bd->devNr, mount->bd->parNr, g_fs[i]->name);

All registered driver are asked 1-by-1 if they can handle the a specific block device. The sequence depends on what driver is registered first, but in general this will result in MBR -> GPT -> exFat (if loaded). So from this loop, the sequence to try and mount a USB stick without MBR should be:

• MBR, should return -1
• GPT, should return -1
• exFat, should mount the usb stick and return 0

Why would MBR "be requested again?" Where is the exFAT driver handling it "first"? Is it not bdm that takes each devices and then tries to mount everything?

See the loop above. Each time a new block device is registered (a partition is also a block device), all filesystems (MBR, GPT and exFat) will be asked by the above loop if they can handle that block device. Again with the sequence MBR -> GPT -> exFat.

No, we need partitions checks at start because first we need to validate if this is real and legit MBR or not. This must be done before trying to do anything at all with your partitions.
If we check only in the loop then bad things can and will happen depending on garbage data. What if first partition is correct by chance passing all checks? And the rest if failing? You will be going though many incorrect bdm_connect_bd and random garbage data can produce all sort of issues.

If there is exFAT driver than it defaults too after trying MBR and GPT, then the solution is to simply return -1 right after partitions_sanity_check_mbr.

I think you're right and that's the best solution. Can you make and test the change?

[DKWDRV]

@rickgaiser where is the registered exFAT driver then that will handle return -1 for partitions_sanity_check_mbr?

[rickgaiser]

I don't think I understand the question...

where is the registered exFAT driver

The exFat driver is here:
https://github.com/ps2dev/ps2sdk/tree/master/iop/fs/bdmfs_fatfs
It will register itself using bdm_connect_fs

that will handle return -1 for partitions_sanity_check_mbr?

Only the MBR driver will do MBR sanity checking. The exfat driver will only do exfat sanity checking.

The VBR should just be some ignored piece of reserved data at the beginning of the exfat partition, right?

[DKWDRV]

I don't think I understand the question...

where is the registered exFAT driver

The exFat driver is here: https://github.com/ps2dev/ps2sdk/tree/master/iop/fs/bdmfs_fatfs It will register itself using bdm_connect_fs

that will handle return -1 for partitions_sanity_check_mbr?

Only the MBR driver will do MBR sanity checking. The exfat driver will only do exfat sanity checking.

The VBR should just be some ignored piece of reserved data at the beginning of the exfat partition, right?

What you call exFAT driver is FAT and exFAT driver support too?
If the way this works is correct and it loops though all three of them then just checking the valid partitions returning -1 should do it.
Where do I get compiled IRX for this @israpps ? Eitherway we can merge it, since it's most likely the correct way.

[rickgaiser]

What you call exFAT driver is FAT and exFAT driver support too?

Yes, that's the one and only BDM exFat driver. I assumed you knew since you're trying to fix: "Latest bdm exFAT drivers fail". It's a port of fatfs: http://elm-chan.org/fsw/ff/00index_e.html
Is supports fat12/16/32/exFat.

Where do I get compiled IRX for this @israpps ?

You're making a PR to ps2sdk so I assume you can compile ps2sdk and have it already.

[DKWDRV]

@rickgaiser think it's better now? Can't see why build is failing.