ramblings of a {sys,net}admin…

Anyone know if it’s possible to pair up a Blackberry Pearl with a MacBook over Bluetooth in order to get online?

I can do this pretty easy with XP and have managed to get it to work under Ubuntu Linux after casting some ancient African spells, but once my MacBook gets here that’ll be irrelevant. I don’t have a need to do that very often, but occasionally it comes up so I thought I’d ask while I’m thinking of it.

Have any of you been able to link up your Pearl with OS X over Bluetooth in order to get online? Thanks!

I recently moved, back to the city where I work. I use the term “moved” somewhat loosely, however. 90% of my belongings are still at my home, but for the last two and a half months, I’ve spent the night there perhaps five times. The rest of the time I’ve been staying at my girlfriend’s. At the beginning of this month, we made it “official” that I’m “living here” (again).

As an I.T. professional, I need a high-speed broadband connection wherever I’m at. At my home, I had the best connection available, which was only 768/128 DSL. Prior to a few weeks ago, I would just move to a certain corner of my girlfriend’s house and connect up to an open access point if I needed to get online for a moment.

When it became “official” that I was going to be living here, I called up the cable company and placed an order for their high-speed Internet service. I did so at just the perfect time, right as they’re upgrading to 10 Mbit/s downstream (and 1 Mbit/s upstream). Anyways, I’m one of the first here to have an Internet connection at 10 Mbit/s (the fastest currently available within a 50-mile or so radius, I might add).

I don’t have all of my “geek gear” here as of yet, but I do have the new Gentoo box and my (XP) laptop. I also dug out some of my Cisco gear, which I haven’t really used in a year or two, and brought it up here as well.

Anyways, I now have a 10 Mbit/s Internet connection, with all of my packets travelling through my 100% “Cisco-powered network”. Not a big deal, I know, but I feel like a bit more of a geek when I can say that. =)

What do you do with a .edu that’s spread across a state, with 14 regions, 24 campuses, and perhaps 45 sites that are all independently managed and maintained and contain a mix of Windows NT, Windows 2000, Windows 2003, 98/ME/2K/XP, Novell, and Linux?

You set up a single Active Directory forest and move ‘em all into that, of course. This is the newest project I’m getting involved in… upwards of 100,000 users, probably 50,000 computers, and $deity knows how many printers, file servers and shares, and all that crap.

This one’s going to be fun… I think I’ll have a beer now. =)

So many months ago, I wrote “Web Server Migration: IIS/Win2K -> Apache2/RHEL4″ where I mentioned that I got the “ok” to begin working towards migrating our existing web server, a single Win2K box running IIS5 over to a pair of RHEL4 boxes running Apache2 set up in a failover configuration (using heartbeat). The new machines are now in production and the whole project is about 75% complete.

You may remember me bitching about the “embedded SATA RAID” these HP DL320’s come with in “RHEL4 and DL320 Embedded SATA RAID”. Well, it’s a big piece of shit, avoid it at all costs. Seriously.

I spent hours upon hours trying to figure out how to get the HP “drivers” to play well with it, having to revert back to installing RHEL4 Update 1 (after Update 3 had been released). Then, of course, once I upgraded the kernel, it broke. I finally said the hell with trying to use the hardware RAID and switched to just using Linux’s built-in software RAID and it’s working perfectly. If you run into this as well, just skip the bullshit and go for the software RAID.

So this is, officially, I suppose, the follow-up to “High-Availability Failover w/ Apache and Red Hat Enterprise Linux”. The new Linux “cluster” is up and running and is in production. A number of our web sites have been moved to this cluster — all of them, actually, except for the main one. The primary site depends so much on FrontPage Server Extensions that it’s not even funny. There are forms galore on that site, so everything had to be converted to use something else — in this case, a custom in-house application for creating web-based forms. Fortunately, that was a job for our webmaster and not me (oh, thanks again for the new book, Daniel).

This past Monday, new firewall policies were pushed that opened up 80/TCP and 443/TCP to the cluster and the DNS records were changed that evening. With only a four hour TTL value in DNS, it didn’t take long for all the incoming requests to start hitting the Linux cluster instead of the old Win2K box. Unfortunately, that server is also housing a few applications based on .NET, so we can’t completely get rid of the old server. The good news is, however, that it will soon be served up by Apache2 on Red Hat Enterprise Linux 4, and the FrontPage crap will be gone!

I’m gonna give it a week or so with the current sites to make sure we don’t run into any issues, then we’ll move our main web site over as well. The other sites are “specialty” sites, as I call them, and they don’t really get much traffic anyways (probably <100 hits/day altogether), while our “primary” site is averaging about 1.5 million/month. And since I have l33t Perl and PHP skills, I get to do some cool shit with our new site (dynamic content — finally! — and RSS feeds and podcasts, oh my!).

Should be cool…

(On a side note, I should mention that when I started working here three years and a few days ago, there was exactly one Linux box on our whole campus. It was being used by the guy who had the position that I now have. In the time that I’ve been working here, we now have a number of applications and services that RELY on Linux: our wireless networks, our web cache/proxy server, numerous in-house applications, scripts that automate our jobs and make things easier, employee directories, and a number of other things. $boss was kinda leery at first when I wanted to start moving “production” apps to Linux, but I think he’s gotten over that initially “nervousness” quite well!). Linux 1, Microsoft 0. =)

Why, after all this time, do people STILL rely on MAC-based authentication? Can someone tell me that?

First, let me say that we employ MAC-based authentication on the wireless network I set up at $work. We do not rely on it, however.

In order to get a valid IP address from our DHCP server, the access points must “authenticate” your MAC address. You could spoof or change your MAC in order to get onto our wireless network. It wouldn’t do you much good, however.

The access points are configured such that one client cannot talk to another client. In addition, once you’re associated with the A.P. and have a valid IP address from the DHCP server, you still can’t do anything! You are, in effect, on an isolated subnet. ACLs are in place that prevent your device from communicating with anything else in the world except for a VPN server on another subnet. In order to “get out”, you first have to establish a VPN connection. This, of course, requires valid credentials.

Even though, you’re somewhat limited. ACLs in place there allow outgoing traffic on 22/TCP, 80/TCP, and 443/TCP. That’s it. Yes, I know that one could set up proxies outside of our network or otherwise bypass these restrictions, but I’m not too concerned with that, to be honest.

Maybe that’s it. Maybe people just aren’t concerned if others “spoof” their MAC addresses and gain access to their wireless networks?

Case in point: Netsurf USA. They provide Internet access to this small town over 802.11 wireless. We have two large water towers, one at each end of town. They’ve got large 802.11 antennas on top of these two water towers, and they put a directional antenna at each customer’s site in order to get them connected. I noticed this tonight when I was sitting on the front porch with the laptop. (Remember, Blueriver hasn’t gotten my DSL up and running as of yet).

NetStumbler showed me a few networks and I remembered these particular SSIDs. I fired up aircrack-ng and it immediately spit out the MAC addresses of a few clients that were communicating with the access point.

Guess how long it took me to get connected (hypothetically, of course). =)

One other thing that I’m going to do now that I’m back in Mitchell is to “relaunch” my consulting business, LinuxWiz Consulting. I stopped taking new clients and finished up my active projects months ago because a certain somebody complained that it took up too much of my time. That won’t be an issue anymore, so I’m going to start it back up. There’s some legal paperwork I’ll have to take care to operate in this county again, but nothing major. Hopefully within the next 30 days or so, LinuxWiz Consulting will be back in full-swing!

Choose no life. Choose no career. Choose no family. Choose fucking big routers. Choose switches big as trucks, modem racks, bridges and repeaters, coffee makers, all networked. Choose no sleep, lots of caffeine and no time off. Choose no friends. Choose black clothing and long hair. Choose tons of mail and wondering why the fuck you’re logged on on a Sunday afternoon. Choose sitting in a swivel chair looking at mind-numbing network maps and galloping counters, stuffing junk food into your mouth while trying to solve someone else’s problem. Choose having ‘IOS’ tattooed on your wrist.

Choose your future.

Choose internetworking.

so cracking wep really is as easy as they say…

i was sure of this since i don’t really have a reason to not believe anyone who has says it, but i’m one of those people who like to see things before i believe them. i also use the metasploit framework to run exploits against my own networks, just to verify that they are real.

anyways, i wanted to crack wep. my laptop has the intel pro/2200 wireless (centrino) built-in but apparently it can’t do packet injection, which is kinda a must.. reports that i read indicated that most of the atheros-based cards worked wonderfully, so i set out to find one (the netgear wg511t was specifically mentioned). i ran to office depot and managed to find one, and it’s even on sale for $20 off (instant rebate, not a stupid mail-in rebate). anyways, i bought it and came home.

after downloading the madwifi code/modules from livna for fedora core 5 on the laptop, it just worked(TM). i got to work with aircrack and started looking to see what kinda activity was going on. anyways, there was no traffic on the network i wanted to crack. aireplay-ng worked perfectly, associating to the access point so that i could capture the association traffic and replay it. it started injecting/replaying almost immediately and i watched the initialization vector (IV) packet count start increasing pretty quickly. i left it running and went to bed.

when i woke up this morning, there were 1,020,384 IVs that had been captured. since most docs i’ve read say that 128-bit wep can be cracked with around 200,000, i was sure this was plenty. i was right.

i fired up aircrack-ng, pointed it at the file containing the captured IVs and let it go to work. i didn’t have to wait long… four seconds later, it informed me that the key had been found. it was done. i was then able to use the encryption key to then connect to the network (using “iwconfig”) and post this blog entry. =)

here’s a png image of aircrack-ng finding the key.

A few days ago, I wrote about the beginning of a web site migration from IIS5 on Windows 2000 Server to Apache2 on Red Hat Enterprise Linux 4. For those who didn’t read the original article (above), the machine that will serve as the “new” web server is a dual Pentium 3 600MHz box — quite old, but it should serve us well.

Because the machine is so old and I value uptime and availability of our various web sites, I wanted to find a nice, easy, cheap way to provide redundancy in the event that something happens to the machine. I found this in “heartbeat”.

In my case, I’m using Red Hat Enterprise Linux. We have a site license for this, so the cost isn’t a factor. For others, the cost may very well be a factor. In that case, I’d encourage you to look into CentOS. CentOS is about as close as you can get to RHEL, without being Red Hat. Everything here should work perfectly on CentOS, as well.

I started out with two completely identical PCs. Both are dual P3 650MHz boxes with 512MB of RAM and 20GB IDE hard drives — exact same model of RAM, exact same model of hard drive. I then installed RHEL AS 4 on the first one and configured it how I wanted it — network settings, hostname, software packages, etc. When everything was set up on what I’ll call the “primary” server (”zeus”), I shut it down and installed the 2nd of the two hard drives in the primary server as well. I then booted up the primary with a Knoppix CD. Once I got in to a shell, I simply ran

dd if=/dev/hda of=/dev/hdb

This gave me a 2nd hard drive that was an exact, bit-for-bit copy of the first. I reinstalled the 2nd drive in the “secondary” server (”hera”). I booted up hera, disconnected from the network, so that I could change the hostname, IP address, etc. Once that was done, here’s what I had:

zeus - 192.168.43.51 hera - 192.168.43.52

I’ve obfuscated the actual IPs, because the real servers have public IP addresses, but that’s irrelevant at this point. Using heartbeat, the servers will “share” a “virtual IP address”, 192.168.43.53. This is the IP address that will published in DNS as “www.example.com”. When a client surfs to “www.example.com”, they will actually be connecting to 192.168.43.53, the virtual IP address. This virtual IP address will be “passed back and forth” between the two servers if or when one goes down.

We need a way for the servers to monitor each other, to detect when the other goes down. In this case, I’m done so via two different methods. First off, I’ve installed an extra network interface card (NIC) in each server and a crossover cable connects them. zeus is configured as 10.10.10.1 and hera is configured as 10.10.10.2 on these secondary NICs. In addition, a null modem cable connects /dev/ttyS1 on zeus to /dev/ttyS1 on hera. This provides us a secondary way to monitor, in case the crossover cable were to go bad, for example.

To get started, download the following files:

• heartbeat-1.2.3.cvs.20050927-1.rh.el.um.1.i386.rpm
• heartbeat-pils-1.2.3.cvs.20050927-1.rh.el.um.1.i386.rpm
• heartbeat-stonith-1.2.3.cvs.20050927-1.rh.el.um.i386.rpm
• libnet-1.1.2.1-1.rh.el.um.1.i386.rpm

I obtained these files from the Ultra Monkey web site. I’ve mirrored them locally for your convenience, but feel free to grab them from the link above if you don’t trust me.

Download each of the above files and when you’re done, run “rpm -Uvh filename.rpm” to install each one. Helpful hint: place them all in one directory, run “rpm -Uvh *.rpm” and rpm will figure out which order they need to be installed in and install them in order.

First off, let’s set up a software watchdog. A “watchdog” runs in the background and monitors the server. If the server becomes “sick”, it reboots it after one minute. This can be very handy. Feel free to skip this part if you’re not interested — it’s optional.

Load the software watchdog module:

modprobe softdog

See if the device node is already there:

ls -l /dev/watchdog

If it’s already there (and looks like “crw——- 1 root root 10, 130 Apr 19 13:05 /dev/watchdog “), then you’re good to go. If not, then:

mknod /dev/watchdog c 10 130

Create the /etc/ha.d/ directory if it doesn’t already exist (”mkdir /etc/ha.d/”). Let’s start by creating /etc/ha.d/ha.cf using your favorite text editor. Here’s what my complete ha.cf file looks like:

serial /dev/ttyS1 watchdog /dev/watchdog bcast eth1 keepalive 2 warntime 6 deadtime 12 initdead 150 baud 9600 udpport 694 autofailback on node zeus.example.com node hera.example.com respawn hacluster /usr/lib/heartbeat/ccm

Let’s go through it line by line:

serial /dev/ttyS1

This tells heartbeat that we’ll be using a serial port (/dev/ttyS1, in this case) to monitor the heartbeat.

bcast eth1

In addition, we’ll also be using broadcasts on the eth1 network device to monitor the heartbeat.

keepalive 2

We want heartbeats being sent every 2 seconds.

warntime 6

This is the time value (in seconds) before a “late heartbeat” will be logged.

deadtime 12

A node will be considered “dead” after 12 seconds.

initdead 150

On some systems, the network takes some time to come up after a reboot. This should be at least twice the normal deadtime.

baud 9600

This controls the speed we’ll be communicating at over the serial port.

udpport 694

This defines the UDP port we’ll be sending heartbeats over (on “eth1″, see above).

autofailback on

With auto_failback set to on, the primary server will resume control if/when it comes back up (after “dying”). If this is off, the secondary server will stay “in charge” after the primary goes away.

node zeus.example.com node hera.example.com

This defines the nodes in our cluster. This should be the exact names as reported by “uname -n”.

respawn hacluster /usr/lib/heartbeat/ccm

This tells heartbeat to run the command “/usr/lib/heartbeat/ccm” as the user “hacluster”, to monitor it, and restart it (”respawn”) if it dies.

Make sure to create this file on both nodes. It is exactly identical in my case, though it could potentially be different (different ethX devices, for example).

Next up is /etc/ha.d/haresources. This file defines what services we’re running and who the “owner” is of those services. In our case, the only service is “httpd”.

zeus.example.com 192.168.43.53 httpd

This file must be identical on all nodes! “zeus.example.com” refers to the primary server, 192.168.43.53 is the virtual IP address (the one that’s shared) and “httpd” refers to the service we’re monitoring.

Last up is /etc/ha.d/authkeys. This provides a bit of security to the cluster and controls the “secret keys” needed to join the cluster. There are a few different options here, but since we’re operating over a crossover cable (about as secure as we can get), we’re going to use “crc”.

This file must also be the same on all nodes, and looks something like this:

auth 42 42 crc

This file must also be readable and writeable by root. You can ensure this by running

chmod 600 /etc/ha.d/authkeys

Next up, let’s set up heartbeat to actually start when the system boots up. We want to make sure that the “softdog” module loads during system startup:

echo “/sbin/modprobe softdog” >> /etc/init.d/rc.sysinit

And activate the startup scripts:

/sbin/chkconfig –levels 345 heartbeat on

These commands must be run on all nodes, by the way. Now, let’s start up heartbeat:

/sbin/service heartbeat start

You can now monitor the logs in /var/log/ha-log to verify proper startup. I had a small issue here that took me a took to figure out. The nodes couldn’t communicate with each other. This turned out to be due to my fairly tight firewall, which wasn’t allowing the UDP packets in on the eth1 interfaces. :/

If everything starts up okay, the primary server should start up the Apache webserver, while it won’t be running on the secondary. As soon as the primary goes away, however, the secondary server will take over the virtual IP address and start up Apache.

In my case, I can ping the virtual IP address, 192.168.43.53, from another PC. While this ping is running, I can run “/sbin/shutdown now -r” on the primary, and watch it shut down and the secondary take over. Monitoring the output from ping shows that the failover is so quick that not even a single ping packet is lost. SUCCESS!

Later articles will show how I keep the data on the servers (authentication files — /etc/{group,passwd,shadow} — and the web files — under /var/www) in sync using rsync.

So I finally got the “ok” from the powers that be to begin the process of migrating our web servers from IIS on Windows 2000 to Apache on Red Hat Enterprise Linux. I sent out what I thought was a pretty convincing e-mail to all of those who are somewhat involved in the whole deal voicing my opinions and thoughts. I was going to include it here, but it’s reallllllly long, so I won’t.

Long story short, it worked. I was actually off work at the time I composed the message, due to a surgery I was having. After I got back, six or seven of us got together in a meeting to discuss, among other things, the pros and cons of migrating the site over to Linux. I got the okay.

Now, the last week or two I’ve been looking into various issues. I mentioned I could do this whole project on existing hardware with minimal financial costs. Now I’ve got to prove it. We had just ordered some new PCs to replace some aging ones, which proved to be good timing. I’ve now come into possession of a number of those “aging ones” and am going to be utilizing them for this project. Right now, I have two PCs that have dual 650MHz CPUs, 512MB of RAM, and plenty of drive space to host our website. Thus far this year, we’re averaging around 1.15 million hits/month, so we’re not outrageously high traffic or anything like that and these PCs should be plenty sufficient to do the job. I also have plenty of spare parts on hand in case any should die, and I can always beef up the RAM and such a bit, if necessary (I don’t think it will be).

I’m also going to be doing a bit of “rearranging” of hardware. I currently have two unused 73GB SCSI drives and a couple of servers that I’m going to “swap around”:

• a dual 866MHz server currently doing nothing, and
• a 933 MHz server running Squid and a few in-house web apps.

The plan, which involves a lot of swapping of hardware, will end up like this:

• a 933MHz server running either MySQL or PostgreSQL, and
• a dual 866MHz server running Squid.

The server running Squid is by far the one that’s the “busiest”, as we have users (students, mostly) surfing the web all day long (duh!). It has the highest load average of any of them and will benefit the most (I think) from the extra CPU power. Then, the 933MHz server will be a dedicated database server to support the databases used by our website (which is currently completely static — more on that later) and other databases used internally (most of our database stuff is on Microsoft SQL Server).

There are a few things that I was still kinda unsure about, but that was pretty much taken care of today. The two PCs that will be running the web site will be using heartbeat in an active/passive failover mode. That way, if the primary server goes down or otherwise becomes unavailable, the other will step up and take its spot. This should help me sleep a bit better as I don’t have to worry so much about when one of them dies. Now, if they both at the same time, then we have issues. =)

Today I got the two PCs set up identically with most things set up before I “imaged” the second one (Apache, postfix, etc.) so those are ready to go. I just installed heartbeat tonight and I’m actually thinking about going back in tonight to install an extra NIC in each one (for monitoring with heartbeat — crossover cable, baby!) and hookup a modem cable between ‘em. Of course, I’ll have each of their NICs plugged into separate switches and such to help provide a bit more redundancy.

So, I still have some work ahead of me, but I have faith that everything is going to work out well. Oh, I mentioned earlier that our web site is completely static. If anything on it changes, it’s because someone changed it by hand. This wouldn’t be so bad if we didn’t have “dynamic content” on our site. Take our little “events calendar”, for example. Anytime a new event is announced or coming up or what have you, $webmaster has to go in and manually update the site. “Wouldn’t it be nice if he didn’t have to?” was one of the questions I asked in my initial persuasive e-mail. The thought being that we can set up a front-end web app so that authorized users can add their own events to the database backend. Then, whenever the web page is accessed, it will automatically pull the latest event info from the database. Common sense, yes, but evidentally it’s not so common. Oh, and have I mentioned that I hate ASP? Well, I do, and I won’t write any if I can avoid it. At least if it’s on Linux, I have my choice of C, Perl, PHP, etc. I shudder when I think about having to do this in ASP. Oh, and the same thing for podcasts. We just “launched” our first official podcast, and will have more coming up. It’d be nice if the right people could access a front-end to upload their MP3 files, provide the info (title, description, etc.) of the podcast and have it automatically made available via our XML feed. Once again, common sense.

So, long story short (have I said that already?), this is going to be a pretty fun project to get going. I’m a huge FOSS freak, so I view it as a victory for the open-source world. =) Stay tuned, and I’ll keep you updated on how it’s going.