ramblings of a {sys,net}admin…

What do you do with a .edu that’s spread across a state, with 14 regions, 24 campuses, and perhaps 45 sites that are all independently managed and maintained and contain a mix of Windows NT, Windows 2000, Windows 2003, 98/ME/2K/XP, Novell, and Linux?

You set up a single Active Directory forest and move ‘em all into that, of course. This is the newest project I’m getting involved in… upwards of 100,000 users, probably 50,000 computers, and $deity knows how many printers, file servers and shares, and all that crap.

This one’s going to be fun… I think I’ll have a beer now. =)

So many months ago, I wrote “Web Server Migration: IIS/Win2K -> Apache2/RHEL4″ where I mentioned that I got the “ok” to begin working towards migrating our existing web server, a single Win2K box running IIS5 over to a pair of RHEL4 boxes running Apache2 set up in a failover configuration (using heartbeat). The new machines are now in production and the whole project is about 75% complete.

You may remember me bitching about the “embedded SATA RAID” these HP DL320’s come with in “RHEL4 and DL320 Embedded SATA RAID”. Well, it’s a big piece of shit, avoid it at all costs. Seriously.

I spent hours upon hours trying to figure out how to get the HP “drivers” to play well with it, having to revert back to installing RHEL4 Update 1 (after Update 3 had been released). Then, of course, once I upgraded the kernel, it broke. I finally said the hell with trying to use the hardware RAID and switched to just using Linux’s built-in software RAID and it’s working perfectly. If you run into this as well, just skip the bullshit and go for the software RAID.

So this is, officially, I suppose, the follow-up to “High-Availability Failover w/ Apache and Red Hat Enterprise Linux”. The new Linux “cluster” is up and running and is in production. A number of our web sites have been moved to this cluster — all of them, actually, except for the main one. The primary site depends so much on FrontPage Server Extensions that it’s not even funny. There are forms galore on that site, so everything had to be converted to use something else — in this case, a custom in-house application for creating web-based forms. Fortunately, that was a job for our webmaster and not me (oh, thanks again for the new book, Daniel).

This past Monday, new firewall policies were pushed that opened up 80/TCP and 443/TCP to the cluster and the DNS records were changed that evening. With only a four hour TTL value in DNS, it didn’t take long for all the incoming requests to start hitting the Linux cluster instead of the old Win2K box. Unfortunately, that server is also housing a few applications based on .NET, so we can’t completely get rid of the old server. The good news is, however, that it will soon be served up by Apache2 on Red Hat Enterprise Linux 4, and the FrontPage crap will be gone!

I’m gonna give it a week or so with the current sites to make sure we don’t run into any issues, then we’ll move our main web site over as well. The other sites are “specialty” sites, as I call them, and they don’t really get much traffic anyways (probably <100 hits/day altogether), while our “primary” site is averaging about 1.5 million/month. And since I have l33t Perl and PHP skills, I get to do some cool shit with our new site (dynamic content — finally! — and RSS feeds and podcasts, oh my!).

Should be cool…

(On a side note, I should mention that when I started working here three years and a few days ago, there was exactly one Linux box on our whole campus. It was being used by the guy who had the position that I now have. In the time that I’ve been working here, we now have a number of applications and services that RELY on Linux: our wireless networks, our web cache/proxy server, numerous in-house applications, scripts that automate our jobs and make things easier, employee directories, and a number of other things. $boss was kinda leery at first when I wanted to start moving “production” apps to Linux, but I think he’s gotten over that initially “nervousness” quite well!). Linux 1, Microsoft 0. =)

One other thing that I’m going to do now that I’m back in Mitchell is to “relaunch” my consulting business, LinuxWiz Consulting. I stopped taking new clients and finished up my active projects months ago because a certain somebody complained that it took up too much of my time. That won’t be an issue anymore, so I’m going to start it back up. There’s some legal paperwork I’ll have to take care to operate in this county again, but nothing major. Hopefully within the next 30 days or so, LinuxWiz Consulting will be back in full-swing!

A few days ago, I wrote about the beginning of a web site migration from IIS5 on Windows 2000 Server to Apache2 on Red Hat Enterprise Linux 4. For those who didn’t read the original article (above), the machine that will serve as the “new” web server is a dual Pentium 3 600MHz box — quite old, but it should serve us well.

Because the machine is so old and I value uptime and availability of our various web sites, I wanted to find a nice, easy, cheap way to provide redundancy in the event that something happens to the machine. I found this in “heartbeat”.

In my case, I’m using Red Hat Enterprise Linux. We have a site license for this, so the cost isn’t a factor. For others, the cost may very well be a factor. In that case, I’d encourage you to look into CentOS. CentOS is about as close as you can get to RHEL, without being Red Hat. Everything here should work perfectly on CentOS, as well.

I started out with two completely identical PCs. Both are dual P3 650MHz boxes with 512MB of RAM and 20GB IDE hard drives — exact same model of RAM, exact same model of hard drive. I then installed RHEL AS 4 on the first one and configured it how I wanted it — network settings, hostname, software packages, etc. When everything was set up on what I’ll call the “primary” server (”zeus”), I shut it down and installed the 2nd of the two hard drives in the primary server as well. I then booted up the primary with a Knoppix CD. Once I got in to a shell, I simply ran

dd if=/dev/hda of=/dev/hdb

This gave me a 2nd hard drive that was an exact, bit-for-bit copy of the first. I reinstalled the 2nd drive in the “secondary” server (”hera”). I booted up hera, disconnected from the network, so that I could change the hostname, IP address, etc. Once that was done, here’s what I had:

zeus - 192.168.43.51 hera - 192.168.43.52

I’ve obfuscated the actual IPs, because the real servers have public IP addresses, but that’s irrelevant at this point. Using heartbeat, the servers will “share” a “virtual IP address”, 192.168.43.53. This is the IP address that will published in DNS as “www.example.com”. When a client surfs to “www.example.com”, they will actually be connecting to 192.168.43.53, the virtual IP address. This virtual IP address will be “passed back and forth” between the two servers if or when one goes down.

We need a way for the servers to monitor each other, to detect when the other goes down. In this case, I’m done so via two different methods. First off, I’ve installed an extra network interface card (NIC) in each server and a crossover cable connects them. zeus is configured as 10.10.10.1 and hera is configured as 10.10.10.2 on these secondary NICs. In addition, a null modem cable connects /dev/ttyS1 on zeus to /dev/ttyS1 on hera. This provides us a secondary way to monitor, in case the crossover cable were to go bad, for example.

To get started, download the following files:

• heartbeat-1.2.3.cvs.20050927-1.rh.el.um.1.i386.rpm
• heartbeat-pils-1.2.3.cvs.20050927-1.rh.el.um.1.i386.rpm
• heartbeat-stonith-1.2.3.cvs.20050927-1.rh.el.um.i386.rpm
• libnet-1.1.2.1-1.rh.el.um.1.i386.rpm

I obtained these files from the Ultra Monkey web site. I’ve mirrored them locally for your convenience, but feel free to grab them from the link above if you don’t trust me.

Download each of the above files and when you’re done, run “rpm -Uvh filename.rpm” to install each one. Helpful hint: place them all in one directory, run “rpm -Uvh *.rpm” and rpm will figure out which order they need to be installed in and install them in order.

First off, let’s set up a software watchdog. A “watchdog” runs in the background and monitors the server. If the server becomes “sick”, it reboots it after one minute. This can be very handy. Feel free to skip this part if you’re not interested — it’s optional.

Load the software watchdog module:

modprobe softdog

See if the device node is already there:

ls -l /dev/watchdog

If it’s already there (and looks like “crw——- 1 root root 10, 130 Apr 19 13:05 /dev/watchdog “), then you’re good to go. If not, then:

mknod /dev/watchdog c 10 130

Create the /etc/ha.d/ directory if it doesn’t already exist (”mkdir /etc/ha.d/”). Let’s start by creating /etc/ha.d/ha.cf using your favorite text editor. Here’s what my complete ha.cf file looks like:

serial /dev/ttyS1 watchdog /dev/watchdog bcast eth1 keepalive 2 warntime 6 deadtime 12 initdead 150 baud 9600 udpport 694 autofailback on node zeus.example.com node hera.example.com respawn hacluster /usr/lib/heartbeat/ccm

Let’s go through it line by line:

serial /dev/ttyS1

This tells heartbeat that we’ll be using a serial port (/dev/ttyS1, in this case) to monitor the heartbeat.

bcast eth1

In addition, we’ll also be using broadcasts on the eth1 network device to monitor the heartbeat.

keepalive 2

We want heartbeats being sent every 2 seconds.

warntime 6

This is the time value (in seconds) before a “late heartbeat” will be logged.

deadtime 12

A node will be considered “dead” after 12 seconds.

initdead 150

On some systems, the network takes some time to come up after a reboot. This should be at least twice the normal deadtime.

baud 9600

This controls the speed we’ll be communicating at over the serial port.

udpport 694

This defines the UDP port we’ll be sending heartbeats over (on “eth1″, see above).

autofailback on

With auto_failback set to on, the primary server will resume control if/when it comes back up (after “dying”). If this is off, the secondary server will stay “in charge” after the primary goes away.

node zeus.example.com node hera.example.com

This defines the nodes in our cluster. This should be the exact names as reported by “uname -n”.

respawn hacluster /usr/lib/heartbeat/ccm

This tells heartbeat to run the command “/usr/lib/heartbeat/ccm” as the user “hacluster”, to monitor it, and restart it (”respawn”) if it dies.

Make sure to create this file on both nodes. It is exactly identical in my case, though it could potentially be different (different ethX devices, for example).

Next up is /etc/ha.d/haresources. This file defines what services we’re running and who the “owner” is of those services. In our case, the only service is “httpd”.

zeus.example.com 192.168.43.53 httpd

This file must be identical on all nodes! “zeus.example.com” refers to the primary server, 192.168.43.53 is the virtual IP address (the one that’s shared) and “httpd” refers to the service we’re monitoring.

Last up is /etc/ha.d/authkeys. This provides a bit of security to the cluster and controls the “secret keys” needed to join the cluster. There are a few different options here, but since we’re operating over a crossover cable (about as secure as we can get), we’re going to use “crc”.

This file must also be the same on all nodes, and looks something like this:

auth 42 42 crc

This file must also be readable and writeable by root. You can ensure this by running

chmod 600 /etc/ha.d/authkeys

Next up, let’s set up heartbeat to actually start when the system boots up. We want to make sure that the “softdog” module loads during system startup:

echo “/sbin/modprobe softdog” >> /etc/init.d/rc.sysinit

And activate the startup scripts:

/sbin/chkconfig –levels 345 heartbeat on

These commands must be run on all nodes, by the way. Now, let’s start up heartbeat:

/sbin/service heartbeat start

You can now monitor the logs in /var/log/ha-log to verify proper startup. I had a small issue here that took me a took to figure out. The nodes couldn’t communicate with each other. This turned out to be due to my fairly tight firewall, which wasn’t allowing the UDP packets in on the eth1 interfaces. :/

If everything starts up okay, the primary server should start up the Apache webserver, while it won’t be running on the secondary. As soon as the primary goes away, however, the secondary server will take over the virtual IP address and start up Apache.

In my case, I can ping the virtual IP address, 192.168.43.53, from another PC. While this ping is running, I can run “/sbin/shutdown now -r” on the primary, and watch it shut down and the secondary take over. Monitoring the output from ping shows that the failover is so quick that not even a single ping packet is lost. SUCCESS!

Later articles will show how I keep the data on the servers (authentication files — /etc/{group,passwd,shadow} — and the web files — under /var/www) in sync using rsync.

So I finally got the “ok” from the powers that be to begin the process of migrating our web servers from IIS on Windows 2000 to Apache on Red Hat Enterprise Linux. I sent out what I thought was a pretty convincing e-mail to all of those who are somewhat involved in the whole deal voicing my opinions and thoughts. I was going to include it here, but it’s reallllllly long, so I won’t.

Long story short, it worked. I was actually off work at the time I composed the message, due to a surgery I was having. After I got back, six or seven of us got together in a meeting to discuss, among other things, the pros and cons of migrating the site over to Linux. I got the okay.

Now, the last week or two I’ve been looking into various issues. I mentioned I could do this whole project on existing hardware with minimal financial costs. Now I’ve got to prove it. We had just ordered some new PCs to replace some aging ones, which proved to be good timing. I’ve now come into possession of a number of those “aging ones” and am going to be utilizing them for this project. Right now, I have two PCs that have dual 650MHz CPUs, 512MB of RAM, and plenty of drive space to host our website. Thus far this year, we’re averaging around 1.15 million hits/month, so we’re not outrageously high traffic or anything like that and these PCs should be plenty sufficient to do the job. I also have plenty of spare parts on hand in case any should die, and I can always beef up the RAM and such a bit, if necessary (I don’t think it will be).

I’m also going to be doing a bit of “rearranging” of hardware. I currently have two unused 73GB SCSI drives and a couple of servers that I’m going to “swap around”:

• a dual 866MHz server currently doing nothing, and
• a 933 MHz server running Squid and a few in-house web apps.

The plan, which involves a lot of swapping of hardware, will end up like this:

• a 933MHz server running either MySQL or PostgreSQL, and
• a dual 866MHz server running Squid.

The server running Squid is by far the one that’s the “busiest”, as we have users (students, mostly) surfing the web all day long (duh!). It has the highest load average of any of them and will benefit the most (I think) from the extra CPU power. Then, the 933MHz server will be a dedicated database server to support the databases used by our website (which is currently completely static — more on that later) and other databases used internally (most of our database stuff is on Microsoft SQL Server).

There are a few things that I was still kinda unsure about, but that was pretty much taken care of today. The two PCs that will be running the web site will be using heartbeat in an active/passive failover mode. That way, if the primary server goes down or otherwise becomes unavailable, the other will step up and take its spot. This should help me sleep a bit better as I don’t have to worry so much about when one of them dies. Now, if they both at the same time, then we have issues. =)

Today I got the two PCs set up identically with most things set up before I “imaged” the second one (Apache, postfix, etc.) so those are ready to go. I just installed heartbeat tonight and I’m actually thinking about going back in tonight to install an extra NIC in each one (for monitoring with heartbeat — crossover cable, baby!) and hookup a modem cable between ‘em. Of course, I’ll have each of their NICs plugged into separate switches and such to help provide a bit more redundancy.

So, I still have some work ahead of me, but I have faith that everything is going to work out well. Oh, I mentioned earlier that our web site is completely static. If anything on it changes, it’s because someone changed it by hand. This wouldn’t be so bad if we didn’t have “dynamic content” on our site. Take our little “events calendar”, for example. Anytime a new event is announced or coming up or what have you, $webmaster has to go in and manually update the site. “Wouldn’t it be nice if he didn’t have to?” was one of the questions I asked in my initial persuasive e-mail. The thought being that we can set up a front-end web app so that authorized users can add their own events to the database backend. Then, whenever the web page is accessed, it will automatically pull the latest event info from the database. Common sense, yes, but evidentally it’s not so common. Oh, and have I mentioned that I hate ASP? Well, I do, and I won’t write any if I can avoid it. At least if it’s on Linux, I have my choice of C, Perl, PHP, etc. I shudder when I think about having to do this in ASP. Oh, and the same thing for podcasts. We just “launched” our first official podcast, and will have more coming up. It’d be nice if the right people could access a front-end to upload their MP3 files, provide the info (title, description, etc.) of the podcast and have it automatically made available via our XML feed. Once again, common sense.

So, long story short (have I said that already?), this is going to be a pretty fun project to get going. I’m a huge FOSS freak, so I view it as a victory for the open-source world. =) Stay tuned, and I’ll keep you updated on how it’s going.

A week or so ago I came across an article entitled “A Student-Hacker Showdown at the Collegiate Cyber Defense Competition”. I heard never heard of this particular event before, but it definitely sounds cool. I would love to get a team together at the school where I work and try to compete in this next time around. I’d also love to hear from anyone who’s been involved in it in any fashion. For those who don’t know, I work at a post-secondary institution in Bloomington, Indiana, and do various sysadmin/netadmin/infosec chores there. I’ve thought about trying to organize some sort of “capture the flag” game, but it’s never moved past the “hey, that’s a cool idea” phase in my head. =)

For anyone out there running a combination of Exchange 2003 and Outlook 2003 and who aren’t doing it already, you should really do your users a favor and configure Outlook 2003 for RPC over HTTP, especially if you already have Outlook Web Access enabled from outside your networks (so that your users can check e-mail from home, for example).

I set up RPC over HTTPS maybe a week or so ago at $work, and I have to say it is GREAT! Before, I would just leave a Firefox tab open to my organization’s OWA server and manually refresh it all the time. Now, I don’t have to!

For those of you who don’t know, enabling RPC over HTTPS (don’t use HTTP) allows your users to use the full, “thick” Outlook 2003 client from anywhere that your Outlook Web Access site is available. It works by encapsulating RPC packets (which Outlook uses to connect to the Exchange server) inside of HTTPS packets. HTTPS is a secure protocol for web communications. It is also referred to as SSL, and is the same technology that gives you the little padlock symbol when you’re logging on to your bank.

I am not aware of any outstanding security issues with using RPC over HTTPS, but would be happy to know if there are any. For now, just members of my department (the I.T. department) are using this “feature”, but we about to start opening it up to our “general” employees that often need to stay in touch from home, off-campus meetings, vacation, etc.

Anyways, to sum it up, it’s just excellent. I know leave the “thick” Outlook 2003 client running on my laptop wherever I’m at so that I can stay caught up on e-mail without having to constantly refresh my e-mail tab in the browser. I’d be interested to hear about other’s experiences with using RPC over HTTPS.

Secunia Research has publicly reported another 0-day vulnerability in Internet Explorer. Microsoft apparently confirmed (with Secunia) the vulnerability on February 21st, but no patch during the March release. A “confirmation” of the vulnerability has been posted on the Microsoft Security Response Center Blog as well. They didn’t come right out and say it, but if you read between the lines…

The Internet Storm Center has raised the Infocon level to yellow, as they report seeing at least one proof-of-concept exploit (which fires up calc.exe).

The workaround is to disable Active Scripting in IE, but I’m wondering what all this is going to break. I’d love to be able to do it in my environment, but I can’t just blindly do that without understanding the repercussions. Any ideas? Thanks.

A month and a half or so ago, I wrote about some security podcasts I listen to on a regular basis. I was specifically looking for some good technical podcasts that I might be missing out on. Network security pro Martin McKeay (who hosts a security podcast himself) directed me to Security Catalyst and Sploitcast.

I’ve managed to find some others “on my own”, mainly thanks to the fact that I somehow stumbled across the Yahoo! Music Engine Podcast Plugin (thanks to Jeremy Zawodny). I’ve been using the Yahoo! Music Engine (YME) and subscribing to their service since just after Christmas. I had bought Lindsey a Dell DJ MP3 player for Christmas, so we subscribed to the Yahoo! Unlimited service so that she could keep it full of the latest music. Since Yahoo! lets you share the music between three PCs and two mobile devices, I also installed it on my laptop which usually travels back and forth to work with me. I started putting a few songs on my PDA as well, and the YME worked flawlessly transferring the files back and forth.

So, anyways, as I was saying… The YME is incredibly easy to use and sync’ing a mobile device with it is as easy as plugging your device in. I’ve been listening to podcasts for a while, but it’s been a manual job of downloading them using Juice then dragging them over to the PDA in Windows Explorer.

Now that I’ve got the Podcasts plugin installed, I’ve dumped Juice and subscribed to a bunch more podcasts inside of the YME application. It’s set to check for updates on a daily basis (it stays resident and active in the system tray) and will automatically download them. YME is configured to automatically sync with my device whenever it’s plugged in, so all I will have to do to get the latest podcasts is drop the PDA in its cradle and wait for it to sync. Kickass!

So, here’s a list of what I’m subscribed to now (in no particular order):

• -Technorama
• -Gillmor Gang
• -SABAGsecurity
• -Geek Muse
• -PodcastStudio.net Show
• -Geek News Central
• -Security Now! ¹
• -NerdTV
• -A Day in the Life of an Information Security Investigator
• -PaulDotCom Security Weekly
• -InfoWorld Daily Podcast
• -The Security Catalyst
• -We Hate Tech
• -KFI Tech Guy
• -Friends in Tech
• -Silicon Valley, Technology, Media InfoTalk
• -Z100 Phone Taps
• -Network Security Blog
• -Diggnation
• -In The Trenches
• -CyberSpeak
• -this WEEK in TECH

That’s all of ‘em, I think. Let me know if I’m missing out on anything worth listening to. I like the shorter ones as it’s only about a 10-minute commute to work (if I stop for gas or smokes). I usually let the longer ones build up until either I have to travel to Indianapolis or somewhere fairly far away or until I just finally decide to delete ‘em. Since I’ll probably be spending a day or two in the hospital, maybe I can catch up on some of ‘em then. The only thing there to do is watch TV, which I don’t do anyways, so at least I can listen to some podcasts.

Hmm, I do know two I.T. guys that work there at Bloomington Hospital, though — one of the Microsoft guys and one of the Information Security guys — and I did notice Cisco wireless AP’s hanging outta the ceiling on my way to X-ray. Wonder if I could get access to the wireless network while I’m there? =)

¹though Gibson is a f**kin’ idiot - “I WROTE MY OWN WEBSERVER IN x86 ASM!!!” –siglite, making fun of Gibson

Well, it’s been 11 days since I last posted and I have numerous things to post about… just not the time.

I did want to mention, however, that I passed the 70-290 exam today and am now officially a Microsoft Certified Professional. I took it with about 24 hours notice, so while I didn’t score the best in the world, I still did much better than I expected, especially with zero studying involved.

Now I’m thinking about taking 70-291 tomorrow… I could probably knock out CompTIA’s Security+ without much studying, I think. With those two under my belt, I’d be three-fourths of the way to Microsoft’s MCSA certification. Hmm, we’ll see…