Restart mdadm and lvm on usb device after disconnect

February 03, 2024

Merlin is no more. Merlin was my N40L-Microserver which performed its duty for twelve years. Yet suddenly it stopped working. Just would not boot up anymore.

I decided not to buy a new one, but I have to do someething to access the data, so I decided to buy a USB case for the four discs inside Merlin. Here is the disc-setup:

Two 1 GB discs in a soft-raid level 1, with an lvm installed on the soft-raid. It was used to store the data.
On 250 GB disc for operating system and so on. I never migrated to this to boot directly from the raid.
On 2 GB disc for backups.

After installing discs in the case and connecting it to my laptop via USB, I ran

sudo mdadm --assemble --scan --verbose --force

to initialize the drives, so I could mount them and access the data.

Disconnect

The problem occured after disconnecting the discs (or better said: Disconnecting the Thunderbolt Dock that the discs are connected to.)

I got I/O errors after reconnecting.

I tried to unount the discs and shut down the raid, but a small piece was missing. I got the following error after doing

➜  ~ sudo mdadm --stop -f /dev/md0
mdadm: Cannot get exclusive access to /dev/md0:Perhaps a running process, mounted filesystem or active volume group?

Here is what worked, and this answer on StackExchange helped me with it.

# 1. Unmount and stop everything
sudo umount /media/merlin
sudo dmsetup remove main-root
sudo mdadm --stop /dev/md0

# 2. Restart
sudo mdadm --assemble --scan --verbose

If the does not work, there might be processes around using this device:

sudo lsof | grep md0

should only report something like this

md0_raid1   583                             root  cwd       DIR              252,2      4096          2 /
md0_raid1   583                             root  rtd       DIR              252,2      4096          2 /
md0_raid1   583                             root  txt   unknown                                         /proc/583/exe