Allow for custom image partition scheme

[BrendanSimon]

Recoverable Partition Scheme

Currently the image creation is fixed in the brickstrap.sh script. It would be better if this was factored out user/target to configuration, so that each build type can specify it's own partition scheme, etc.

My use case is for devices deployed in remote locations, that need to be remotely upgraded, and cannot be bricked if something goes wrong in the upgrade process. e.g. loss of network, power drop out, etc. In general this is solved by have multiple rootfs partitions.

My use case

Partition	Purpose	File System	Size
1	BOOT	VFAT	32MB, 64MB
2	ROOTFS1	ext4	1GB
3	ROOTFS2	ext4	1GB
4	DATA	VFAT, ext4	14GB - 30GB

More general use case

Partition	Purpose	File System	Size
1	BOOT	VFAT, FAT32, FAT16, FAT12	32MB, 48MB, 64MB
2	ROOTFS1	ext2, ext3, ext4	512MB, 1GB, 2GB
3	ROOTFS2	ext2, ext3, ext4	512MB, 1GB, 2GB
4	DATA	VFAT, ext2, ext3, ext4	1GB - 30GB

Other partitioning schemes can be considered, including those with logical volumes, extended volumes, lvm, etc.

Ping-Pong Rootfs

---

When a reboot occurs, the boot loader should have the capability to fallback to the last working rootfs if the latest rootfs fails integrity checks, does not load properly or does not run to a certain state where it would set something to notify the bootloader of it's success (e.g. a u-boot env var). This is sometimes referred to as a ping-pong rootfs system.

Golden Rootfs

---

A similar approach is to have the first partition the golden partition that never gets touched once programmed in the factory, and the second rootfs is the only one that gets upgraded. This can have issues of upgrading an upgraded system. i.e. upgrading a running system. It can be solved by booting into the golden partition before doing the upgrade of the 2nd partition. This can mean the system is down while doing the upgrade, which may not be acceptable.

An extension of the Golden Rootfs is to have a 3 rootfs partions -- one golden that never ever gets touched, and two other rootfs partitions for ping-pong. The Golden (or Recovery/Fallback) partition is usually small and simple whose main purpose is to be able to upgrade the main rootfs partitions with valid information.

[dlech]

I'm all for this. The Ping-Pong Rootfs and Golden Rootfs are outside of the scope of brickstrap (other than being implemented in a board definition directory), but we can certainly create images that can be used by these.

I'm thinking that we modify the create-image function to check for a file named create-image in the board definition. If the file exists, then it is used - if not, the existing function is called.

I also have a use case for BeagleBone Black where I need to offset the initial partition a bit to leave room for dding the bootloader at a specific address. I started working on it here.

[BrendanSimon]

Sure. The upgrade process is user specific, but it relies on an underlying partition scheme having been setup. I wouldn't want to repartition on the fly. That's just asking for trouble ;-)

System can use their own update scripts or other software.

Currently I'm trialling swupdate. It seems to be authored by at least one person at DENX Software Engineering so it has reasonable U-Boot support :) It's not fully mature yet, but has a lot of flexible infrastructure.

https://github.com/sbabic/swupdate

[BrendanSimon]

The guestfish utility does not seem to have prepared disk images for anything other than two partitions. e.g. -N bootroot

I think for my usecase, -N bootroot, followed by part-add would probably work, but the hard part is determining the start and stop sectors (probably not that hard). Would be nice to specify order of partitions, size and let the utility work it out. Similar to parted

But for total generality, one can not assume the number of partitions or filesystem types, so I'm guessing that we need to use:

part-init
part-add x N
mkfs x N

[dlech]

You can see a more general form of using guestfish that I started for the BBB here. You are on the right track.

[dlech]

I have hard-coded the sectors, but you could make bash do the math.

[BrendanSimon]

Also the partition to resize to max available space needs to be configurable. In my use case this would be partition 4, the DATA partition, used for user data such as application log files, etc.

[BrendanSimon]

Actually, I think resizing is done on the running system. It should be possible to do this on an actual SD card mounted in the CHROOT environment, yes? Though I guess that wont work for devices using onboard flash (eMMC, etc).

[dlech]

Actually, I think resizing is done on the running system.

Yes, that is how it is currently done and is the easiest way in my opinion.

It should be possible to do this on an actual SD card mounted in the CHROOT environment, yes?

If you want to resize the partitions on an SD card "offline", the best way would be to use a partitioning tool like parted or gparted. It would be dangerous to do this in a chroot because /dev is going to be bind mounted from your host computer and it would be easy to think /dev/sda is your SD card when really it is your hard drive.

[BrendanSimon]

I notice that wikiepedia says the partition type for FAT32 should be 0Ch for capacity > 7.8GB, and 0Bh for capacity < 7.8GB

For SDHC cards:
    Capacity of less than 16,450,560 logical sectors (smaller than 7.8 GB): FAT32 with partition type 0Bh and EBPB 7.1
    Capacity of at least 16,450,560 logical sectors (larger than 7.8 GB): FAT32 with partition type 0Ch and EBPB 7.1

My linux kernel isn't booting properly (it hangs) when I boot off Sandisk Ultra 16GB & 32GB Micro SD cards, but works perfectly well with Sandisk Ultra 8GB card. I'm wondering if the MBR type of 0Bh could be an issue? Or I could have just got lucky with the 8GB card and/or unlucky with the 16GB/32GB card?

[dlech]

Your FAT partition is < 7.8GB in all cases though, right? So partition type is always 0x0b.

[BrendanSimon]

Yes (64MB) but even if there was a need to increase it, it would definitely be < 7.8GB.
The wikipedia page wasn't clear if it was total disk capacity or the partition capacity, but you are probably right that it is partition capacity.

[cmacq2]

The Wikipedia article isn't very clear on the 'why' of it, and what the issue actually is. It's nothing to do with FAT16 versus FAT32 per se because that distinction is about limits on file systems and not disk partitions (which is what this limit is about).

The specific limit here is the size of the partition in number of sectors. (Which is still different from capacity limits in bytes, as sectors can in theory vary in size as well.) Specifically the limit is due to the fact only 24 bits are used to describe the size of the partition (leaving the first/last 8 out of 32 for a tag, i.e. the boot flag or partition type).

See: http://www.win.tue.nl/~aeb/partitions/partition_tables-2.html

[cmacq2]

Once PR #28 is merged, the new images output directory means it becomes feasible to address this issue thoroughly. My idea is as follows:

1. Have the create-image function source any create-image.sh scripts from the project dir. These scripts may register driver functions for creating custom image types. (basically setting a variable with the name of the function).
2. Have create-image dispatch to a driver script based on the selected image type(s). More than one type could be selected at the same time, all create-image would have to is loop over the list, similar to br_iterate_components. (Basically a case block inside a for loop.)
3. Implement default driver(s), which is basically copying the guts of current create-image to another function. Default drivers would take precedence in create-image over any drivers registered by the project with the same name. (Basically 'named' cases, and only the * case actually considers variables set in step 1).
4. Wrap this in a nice interface for selecting image types using a commandline switch, e.g. -F, and an environment variable (for use by project config, to set defaults).