We had someone do a rm -Rf /var instead of rm -Rf var inside a project folder. If your reading this, you probably know the pain.
The admin ctrl+c ed the rm after a few seconds/minutes when the command didn’t finish as quickly as he expected it to.
Lucky for us, we were able to run yum, and rpm -qa still worked.
If your in the same boat and
yum info
works, we have a shot of rebuilding the box.
The first step is to run
rpm -Va > /root/missing.txt
Once this completes you will want to print out just the missing stuff:
cat /root/missing.txt |grep missing
Next you will want to use a multi line select tool or sed or something to take each line that starts with missing and then populate rpm -q --whatprovides around it. For example:
rpm -q --whatprovides /var/account
rpm -q --whatprovides /var/lib/mailman
rpm -q --whatprovides /var/lib/mailman/archives
rpm -q --whatprovides /var/lib/mailman/archives/private
rpm -q --whatprovides /var/lib/mailman/archives/public
rpm -q --whatprovides /var/lib/mailman/data
rpm -q --whatprovides /var/lib/mailman/data/sitelist.cfg
rpm -q --whatprovides /var/lib/mailman/lists
rpm -q --whatprovides /var/lib/mailman/spam
You will now want to run these commands and take note of the rpm's returned.
For example:
[bessig@after ~]$ rpm -q --whatprovides /var/lib/mailman
mailman-2.1.12-26.el6_9.3.x86_64
[bessig@after ~]$ rpm -q --whatprovides /var/lib/mailman/archives
mailman-2.1.12-26.el6_9.3.x86_64
[bessig@after ~]$ rpm -q --whatprovides /var/lib/mailman/archives/private
mailman-2.1.12-26.el6_9.3.x86_64
[bessig@after ~]$ rpm -q --whatprovides /var/lib/mailman/archives/public
mailman-2.1.12-26.el6_9.3.x86_64
[bessig@after ~]$ rpm -q --whatprovides /var/lib/mailman/data
mailman-2.1.12-26.el6_9.3.x86_64
[bessig@after ~]$ rpm -q --whatprovides /var/lib/mailman/data/sitelist.cfg
mailman-2.1.12-26.el6_9.3.x86_64
[bessig@after ~]$ rpm -q --whatprovides /var/lib/mailman/lists
mailman-2.1.12-26.el6_9.3.x86_64
[bessig@after ~]$ rpm -q --whatprovides /var/lib/mailman/spam
mailman-2.1.12-26.el6_9.3.x86_64
We can see we need to reinstall mailman-2.1.12-26.el6_9.3.x86_64.
So fire off:
yum reinstall mailman-2.1.12-26.el6_9.3.x86_64
Sometimes you may get a message from yum telling you it can no longer find that package. In this case you may need to run:
yum update mailman
In rare cases, you may need to find the rpm and manually reinstall it.
In our case, mysql was completely messed up, so we had to reinstall mysql, and then bring over the backup dumps and reimport them. Luckily for us, our most recent backup completed 4 minutes before this happened.
Once you think your done, rerun
rpm -Va > /root/missing.txt
and make sure this returns nothing:
cat /root/missing.txt |grep missing
Hope this helps. There's also a way to recover from a fully deleted /var directory (where the admin doesn't cancel the command).
I have done that before and will hope to do a write up in the future.