My Background as a System Administrator
I was thinking today how much I am beginning to dread working with the new generation system administration, programmers, and other experts that come loaded with more paper certifications and more self confidence than any human should have. Its not that this type of individual wasn’t present in an earlier time but our true computer wizards are completely invisible because of the sheer volume of professionals that make up our industry these days. The wizards of yesteryear could look at a hex dump and see the op codes in that machine dump and patch it in place. They understood the kernel in detail and loved to discuss and talk about all aspects of a system. When they spoke you listened so you could learn something and you gave them the benefit of doubt if they said something that was not quite right because you knew they knew. They knew hardware and they knew software. They are still out there but its becoming very difficult to spot them and managers who should be able to spot them like the CTO,CIO, and CSO are becoming distracted by the noise to signal ratio.
I was incredibly fortunate to learn my trade as a programmer during a time when the wizards could be found on usenet, usenix, and sometimes even the odd visit to fix an obscure driver bug in the kernel. I wanted to be as good as the old mainframe system programmers that I watched when I first got involved with computers in 1980. As a result of my occational meetings with these IBM mainframe programmers, I hacked our little kernels and tools to try and get a deep and better understanding of why things worked as they did under different conditions. It was the way that knowledge was obtained back then and the harder it took to learn the more greatful you were afterwords. I still find this true but often wonder if the web has given us a false sense of security that finding answers is always trivial. It certainly has made somethings easier but its becoming more difficult to find the complete answer with all the pretenders. A recent youtube video from a retired electronic repair man now in his 80’s stated: “I know what I know but not much more”. The phrase from my generation is: “I am self aware and know what I don’t know”. Anyway I am getting off track already so back to my story.
My attempts to emulate these computer wizards pushed me to try some neat hacks my first few years as a system programmer. It was a position I worked hard at in graduate school and I was eager to show those I worked with that I deserved that title when I got my first job as a system programmer. My first boss had a PhD in Physics and we got along great probably because my BS was in Chemistry so we saw the technical details a little different than others in our organization. I look back on this time now with much affection but they were difficult times because I lacked the foundation and knowledge at the time which fortunately pushed me to work even harder. I would try to arrive first in the morning (around 6:30am) and wanted to be the last to leave around (8:30pm). It was something I had started toward the end of graduate school and I learned that I could push myself hard and not burn out. I felt that only through long hours and hard work could you set yourself apart from others and if I worked the hardest that good things would eventually happen to me. If not it still didn’t matter because I was learning things that I had a passion about. While the hours were long, it never felt like work and I had a hard time undertanding why someone would pay me so much money doing somethiing I loved so much. I still have that passion for computers but have had to balance this with family over the past decade as I focused on becoming the best dad possible but it is perhaps the most difficult job to master since the problem space is always evolving. 🙂 My wife worked those long hours with me since graduate school and we fed off each other as we pushed ourselves to the goal of becoming as knowledgeable as we could be. It was truly a fun time to be around computers.
One of the first notable hacks that comes to mind was somewhere between 1985-1987. While others where at home, I would hack and create software that I loved or found interesting on my own time. I have always loved the kernel (OS) and networking best. In graduate school, I modified kermit which was an early serial communication program and written in C to work over tcp/ip and studied code from FTP, inetd, and followed the sockets calls in the kernel so often that I had much of it memorized. I could see in my mind the lines of code executing and any bug was an exciting time because I could replay the code line by line in my head until I saw where the problem might be located. I always loved that about programming and in later years when I had a decent mastery of debuggers, I loved this aspect of programming the best. Creating new software is always fun but finding that bug that others had missed made me feel that I might one day be as good the wizards I tried to emulate. There is a thrill to bug hunting. It is like a detective story as your gather facts in the hopes of knowing who done it. What makes it even better however is that once you know who did it, you can fix it and the story is complete.
I remember a particulary intesting time (1988) when a subcontractor to a subcontractor for the US federal government was developing a complicated device driver for a 100 platter worm juke box commissioned by the government got into a nasty dispute with SONY the developer of the hardware and the customer was an innocent bystander having paid millions for something that didn’t work. We had decided we were going to take over the maintenance and fix the driver ourself because they were slow to find solutions. I had been creating workarounds like patching the binary entry points into the driver to lock critical sections but this was inefficient and we were fed up with the kernel panics. The contractor was obligated by law to provide us the source listings and they choose to obfuscate it so that it wasn’t readable by humans but claimed it would compile and thus met their contractual obligations with Sony.. Sony had seen enough and then asked me to take a look to see if anything could be done and I briefly looked at the obfuscated source and began to write a translater to see if it would compile when the vendor arrived to patch the kernel with some recommended changes I had requested. During my late nights of hacking, I had created a remote login client and server software (over XNS and not tcp/ip) that would allow me to log me into our systems in stealth mode… not show my presence in utmp/wtmp and had modifed a program that when running would appear as -csh in the program listing to remain stealthy. I had also written a program that allowed me to read the kernels keyboard clists directly from kernel memory. In other words, I could spy remotely and capture all keystrokes as they were being typed. I fired up my program and within seconds I had the password to the vendors source tree including other propritory code that the company was selling based on our device driver. I ran down to the computer room and casually asked what XXXX was. I had his password and this vendors best kernel hacker’s face went white. I explained the hack and he gave me the source if I would give him my keyboard logger. From that day forward we were to have the source code but unofficial and I patched the driver myself. The story didn’t end there however as a few months later when things were getting bad again, the vendors kernel hacker arrived but this time he unplugged the network cable and his promise to provide the source code to me was not kept. I went down and asked him and he said that this is how it was going to be but I showed him that I had the source code again. We had a modified c-processor because of some unique coding standards by another vendor which would create a copy of the files before runing cpp so I had the original source code anyway. The next time he arrived he brought binaries only and linked them into the kernel; but this time the negotiations were going very bad so they gave us binaries where the code would expire and thus eventually stopped running. This was my last and best hack as I ran adb and then patched the opcode branch instructions so that the condition was opposite that garded the code from executing. That was check-mate and the vendor stopped screwing around and I took over all maintenance of the driver and made my changes and the kernel stopped panic-ing for good. It was an important lesson for me because I was in my first year as a system programmer and I had gained confidence that I could solve even the most difficult of problems presented to me. Although I didn’t know it at the time I had taken ownership of a problem and did whatever it took to find resolution.
When I look back, I have had many great experiences like my first year as a system programmer. All the tools I had written were done on my own time during those late and extra hours. As my managers learned about them, they gave me more and more freedom to do those on company time because they saw the value to the organization. Eventually, they wanted me to create software and fix things that I thought were broken because even a single computer that failed back then would cost productivity to 100’s of programmers. This experience taught me to be effective in this position that I needed to know what and how the computers were being used. To anticipate problems before the computers programmers would run into them. That is still true today with managing servers and services. I think the ability to understand the system in enough detail and do whatever it takes is what makes a good system administrator (todays terminology).
I have a few questions for myself however after all these years… Why did I share with others about my hacks? I was completely shocked that software that I had written to gain a deeper level of understanding had value. For example, to pull characters from the kernel as the user was typing them on his terminal was said to be impossible at the time because there was always going to be a race condition. I think that was a defining moment in my career because I wanted to learn and share. If somthing was difficult to learn, you want to make it easier for others to learn and experience that deeper understanding that was obtained from long and hard hours of study. I am still frustrated by experts who refuse to share and guard their knowledge like it is the only thing that separates them from the next guy. No. The only thing that separates them is their complete lack of understanding that an organization is much bigger than themselves and as soon as they are gone it will continue to thrive.
I wish it was as easy today to share knowledge in person but I am on a bad streak often being dismissed by people that know so little and think they have mastered the universe that I am finding that they are not interested. How many times does one have to be interuppted by a person answering their cell phone and then after explaining some very cool feature you come to the realization that they really don’t know or have mastered what they believe. I think the system administrators today might be operators of the mainframe days. I don’t mean that as a slight to them either just they have different things that are more important to them.
I recently applied for a job at the University of Victoria as a System Administrator. My wife and I had thought we would like to get back to a University environment and Victoria looked promising. I didn’t even make the cut. Not even a call. Wow. I have never been turned down for a job so this was new territory for me. I didn’t have the certifications and my experience with the commercial systems they were using was from hand assembling the pieces instead of from the 3rd party vendor. I suspect, my resume never made it past the human resources group or perhaps some in house software to match against keywords. I am only speculating or perhaps they thought my statement about running continuous email service since 1995 was a boast. Running a fedora 2 server with an uptime of 2072 days at 1and1 as an email relay – Now that is a boast. 🙂
