Many years back when I was first starting to get savvy with Linux, a co-worker approached me with a problem. He had yanked the harddrive out of his old Linux router and placed it in a different machine. The old machine had on-board ethernet and used a different driver than the new machine's PCI ethernet card. His only other computer was a laptop, which was currently offline due to the router issue. Thinking it would be an easy job, I downloaded a binary copy of the PCI driver and copied it onto a floppy at work. When I got to his house, I realized there was a bigger issue. My friend didn't have a monitor of any sort for the machine we'd be working on. We could have just given up and borrowed one from work the next day, but we decided to see how far we could get without one. We hooked up a keyboard and booted the machine waiting a few extra minutes to be sure we had a login prompt. From there, we very carefully logged in as root and mounted the floppy. The light on the floppy drive gave us a good indication that we were good so far. Not being able to remember the name of the driver forced me to do a glob copy based on the file extension to the /tmp directory, but again, the light on the floppy drive showed things were working. From there, we had another problem. We weren't sure if modprobe, the command to load a kernel module, would accept wildcards, so the globbing was no good this time. Still totally blind, I typed the following into the command line:
for I in *.ko; do modprobe ./$I; done
Fingers crossed, I hoped for the best and typed:
ifconfig eth0 192.168.0.1 netmask 255.255.255.0
From there, we plugged in his laptop, and tried to ping the machine. Much to our surprise, it was up and responding. Feeling accomplished, we did an ssh login from there, and got everything setup as it should be. Now, I'm not saying this was totally elite or anything, but it does go to show that a little creative thinking can get you pretty far. Feel free to share your own survival hacks in the comments if you have a similar story.
Subscribe to:
Post Comments (Atom)
Posts
7 comments:
Have you ever left your terminal logged in, only to find when you came back to it that a (supposed) friend had typed "rm -rf ~/*" and was hovering over the keyboard with threats along the lines of "lend me a fiver 'til Thursday, or I hit return"? Undoubtedly the person in question would not have had the nerve to inflict such a trauma upon you, and was doing it in jest. So you've probably never experienced the worst of such disasters....
http://www.cavecanen.org/linux/humour/horrorstories.txt
This is in the Windows world, but a similar thing would probably happen with X so I'm posting it anyway. My uncle brought over a computer that was set to a res that the monitor wouldn't display, so I had to fire up another machine and write down the keypress sequence, including right-clicking on the desktop, to change the resolution of the display to one that the monitor would recognize. It took a little bit of time, but it worked. For whatever reason my uncle wasn't as impressed.
Installing Ubuntu from a Live CD at one point, the GUI failed to come up... totally blank screen. CTRL-ALT-F5 took me to a terminal console. From there I downloaded a better driver and edited xorg.conf. Mind you, that just installed the driver for the Live CD instance. CTRL-ALT-F7 back to the desktop and CTRL-ALT-BACKSPACE to restart X and I could proceed with the installation. Had to repeat the acrobatic driver installation once the install was complete, but worked like a charm.
Another one I wanted to mention... I used to work at a lab, and I received an email notifying me that a critical server was having hardware problems. Upon physical inspection, I realized the keyboard wasn't working, but the console mouse was still active. Using the mouse cursor to selectively copy and paste characters from the stack trace onto the command line, I was able to spell out r-e-b-o-o-t and append a carriage return to safely bring down the machine.
Travis,
Not even the magic SysRq worked? http://en.wikipedia.org/wiki/Magic_SysRq_key
At some point in the past few years we had had a string of instances of amanda (the backup utility) failing on client machines to be backed up. For some reason, the backup clients would sometimes just fail to respond to connection requests from the amanda backup server and everything would just timeout. Just randomly, I tried '/etc/init.d/inetd restart' on the failing client machines and things worked normally again. Never really investigated the *reason* why they were failing, but restarting inetd did the trick.
My favorite survival hack though has to be booting a livecd for the purposes of fixing an ailing machine. Sometimes this involves the fun chroot trick to just poke around, or sometimes you're actually trying to extract data from a dying system and ship it over ssh to a safe place.
Once I was logged in to my Linux router remotely and thought I was in /etc/ppp and typed rm * but I was in /etc instead. As you can imagine everything ground to a halt including all editors, new login sessions (ssh + telnet), etc. I had to cat "text" >> file to rebuild several critical /etc/ files so I could at least log in again. I rebuilt /etc/user and /etc/shadow with working passwords by copying the encrypted password lines for root and my user from another machine. After a week of running and routing with a duct-tape etc directory I finally got physical access to re-build it.
Post a Comment