Restart mdadm and lvm on usb device after disconnect

A note to my future self, when it has forgotten the commands to do this. With a multi-disc usb case, a soft-raid with lvm, how do I fix the connection after disconnecting the cable?

Photo by Sergei Starostin on Pexels

Merlin is no more. Merlin was my N40L micro-server which performed its duty for twelve years. Yet suddenly it stopped working. Just would not boot up anymore.

I decided not to buy a new one, but I have to do something to access the data, so I decided to buy a USB case for the four discs inside Merlin. Here is the disc-setup:

  • Two 1 GB discs in a soft-raid level 1, with LVM installed on the soft-raid. It was used to store the data.
  • On 250 GB disc for operating system and so on. I never migrated to this to boot directly from the raid.
  • On 2 GB disc for backups.

After installing discs in the case and connecting it to my laptop via USB, I ran

sudo mdadm --assemble --scan --verbose --force

to initialize the drives, so I could mount them and access the data.

Disconnect

The problem occurred after disconnecting the discs (or better said: Disconnecting the Thunderbolt Dock that the discs are connected to.)

I got I/O errors after reconnecting.

I tried to unmount the discs and shut down the raid, but a small piece was missing. I got the following error after doing

➜  ~ sudo mdadm --stop -f /dev/md0
mdadm: Cannot get exclusive access to /dev/md0:Perhaps a running process, mounted filesystem or active volume group?

Here is what worked, and this answer on StackExchange helped me with it.

# 1. Unmount and stop everything
sudo umount /media/merlin
sudo dmsetup remove main-root
sudo mdadm --stop /dev/md0

# 2. Restart
sudo mdadm --assemble --scan --verbose

If the does not work, there might be processes around using this device:

sudo lsof | grep md0

should only report something like this

md0_raid1   583                             root  cwd       DIR              252,2      4096          2 /
md0_raid1   583                             root  rtd       DIR              252,2      4096          2 /
md0_raid1   583                             root  txt   unknown                                         /proc/583/exe