md: document how to use software RAID with CoreOS

[philips]

People may want to software RAID CoreOS. Can we do this today? Can we help with the coreos-install script?

/cc @marineam

[marineam]

No, I would rather not do raid in the current installer script. If we want to start working on a full featured Linux installer we should do that in a language that isn't shell.

[robszumski]

What about software raid outside of the installation script?

[jsierles]

At least it would be useful to document what needs to be done, if anything, besides building the raid and running the installer against it. I tried this but couldn't get my machine to boot.

[marineam]

@robszumski software raid is what we are talking about.

@jsierles sounds like we have bugs to fix because my intent is to make that work.

[nekinie]

Any news on the software raid documentation?

Would be rather useful

Also would see this as very useful functionality.

[pierreozoux]

Yes, it would be great :)
@philips I saw this commit But yeah... Does anybody can tell me where to start if I want raid software? emerge mdadm ??

[marineam]

@pierreozoux mdadm is included base images but we haven't played with it at all. Setting up non-root raid volumes should work just the same as any other distro. Same ol' mdadm command for creating and assembling volumes. You may need to enable mdadm.service if you want to assemble volumes on boot via /etc/mdadm.conf as opposed to using the raid-autodetect partition type and letting the kernel do it. It might be possible to move the root filesystem as long as the raid-autodetect partition type is used but for that you are almost certainly better off with using multi-device support in btrfs.

[marineam]

What certainly won't work right now is installing all of coreos on top of software raid, the update and boot processes both assume the ESP and /usr partitions are plain disk partitions.

[brejoc]

@marineam Would this constraint of CoreOS also apply to btrfs-raids?

[marineam]

@brejoc multi-device btrfs for the root filesystem should work

[warwickchapman]

What about migrating after install. Eg to RAID 1 from installed /dev/sda (one partition sda1 for demonstration) should be something like this from a Rescue CD or similar:

sfdisk -d /dev/sda | sfdisk /dev/sdb
sfdisk --id /dev/sdb 1 fd
mdadm --zero-superblock /dev/sdb1
mdadm --create /dev/md0 --level 1 --raid-devices=2 missing /dev/sdb1
mkfs.btrfs /dev/md0
mkdir /mnt/source;mount /dev/sda /mnt/source
mkdir /mnt/target;mount /dev/md0 /target
cp -a /mnt/source/* /mnt/target

Thereafter the disk mount configuration needs to be changed and the kernel root device in the bootloader, as well as the bootloader installed to both disks.

modify /mnt/target/etc/fstab Replace /dev/sda1 with /dev/md0 - but this is non-existent on CoreOS
bootloader since 435 seems to be GRUB which helps but I cannot find a grub binary only config in /usr/boot

Thoughts?

[seeekr]

@warwickchapman just in case you finished your exploration into this topic and came up with a complete solution - or if someone else has - I'd appreciate if you shared it. I know too little about setting up and messing with RAID / mounts / boot in order to complete this myself. It's not a hard requirement for my use case but it would help being able to have RAID to be able to use both/all disks in a system. I understand it's also possible to set up a distributed file system like Ceph and let it manage the disks without RAID, and that would work for the use cases I have in mind, but for now I'm happy about any additional complexity I can avoid!

[marineam]

As noted on IRC, for btrfs if raid0 or raid1 is all you need then it is easiest to just add devices to btrfs and rebalance: https://btrfs.wiki.kernel.org/index.php/Using_Btrfs_with_Multiple_Devices

As for md raid if the partition types are the raid-autodetect type then the raid volume will be assembled automatically. But you can only put the ROOT filesystem on raid, we don't currently support putting the other partitions on anything other than plain disk devices.