March 09, 2008
Almost done with MCT
I graduate Tuesday, hurray! The final field exercises were fun, the final 15k hike sucked. I've gotten to fire a M203 grenade launcher (with training rounds full of orange death dust), a M240B machine gun (which rocked awesomely), the M-16A4 with an ACOG... I've gotten to do live fire buddy team rushes, combat patrols, defense emplacements, entry control points, grenade throwing, room clearing... MCT has basically been full of win.
Tuesday I should be heading up to Monterey for 581 days. I'll write more later :) Leave comments with laptop suggestions...
Posted by jeff at 04:47 PM | Comments (1)
March 02, 2008
MCT et cetera
So I lucked out, and my MCT training cycle is only 3 weeks. The poor bastards in the cycles after me (well, not the very next cycle, but all after) have to be here for 4 weeks. I'll graduate March 11. If you know the significance of that, shaddup :)
Regarding pictures of my high speed low drag hairdo: that'll probably have to wait until I get to Monterey. Yes, I could take pictures right now, but honestly I can't be bothered. Too much other stuff to do on my precious liberty time.
EDIT: Here, I put a pic up
I found out that I did get Arabic as my language, and I'll get my orders March 10. I don't know when the next Arabic cycle starts at DLI, so I don't know if it'll be possible for me to take any leave to bring my truck up to Monterey. I really hope I'll get some time, as I have dick for civilian clothes handy. I do plan on linking up with Baby Phil, so he can introduce me to hot Air Force womens.
Posted by jeff at 06:02 PM | Comments (1)
September 24, 2007
Asterisk goes down like a chump
So, this entry is entirely uninteresting and boring.
I finally got Caller ID working completely with our Asterisk setup here at work. The key seems to be that TAMU is sending the CID spill for off-campus numbers (numbers not inside the TAMU exchanges) late, possibly after the second ring. I hacked the Asterisk Zaptel driver to retry processing if it wasn't able to get useful data from the CID spill, on the assumption that good data is around the corner. I didn't put in any limitations on the goto I used, so the patch definitely isn't something that should be in production (alas, it is in our case). I sent an email to Digium, to see if they will be willing to patch in support for late spills.
Posted by jeff at 06:57 PM | Comments (0)
August 30, 2007
Storage woes
The CIS problem report is quoted below for posterity (and humor's) sake, but in brief:
Teague machine room tops 140F.
Clariion drive array fails, comes back up after a manual power cycle.
ACNC drive arrays do not visibly fail, but running scrub jobs on our ZFS pools reveals corruption and disk problems.
Then we discover we do not actually have enough spare drives on the shelf to replace all of those eaten by the heat.
And ZFS decides to hang on the command to add new hot spares to various pools.
It's been, seriously, ridiculous and a tad frightening. That's a lot of data to have to restore from backups, so the pressure is on to get these pools out of degraded states.
Around 5:45 this morning we lost power to the machine room and any systems that didn't have a backup UPS went down.* As of 2007/08/29 09:36: The power and air-conditioning failure in the CIS Data Center on Sunday 8/23 was the result of multiple campus power outages and a failure in the software that monitors environmental conditions in the computer room.
At around 10 pm Saturday a feeder supplying electrical power to the building blew out and caused a dirty power failure. The UPS picked it up and switched to the backup generator. After about 12 minutes the power was restored and the UPS switched back to utility power.
This was only one of three major feeder failures that occurred Saturday nightThe power surge generated by the feeder problem caused a fault in the chillers that supply utility chilled water to the Computing Services Center building. When that happened CIS was automatically switched over to backup chillers. Unfortunately, due to a malfunctioning valve, they shut down sometime early Sunday morning.
The building management system is monitored by the Operations Center staff 24 hours a day via the internet using a software product called APOGEE. This software is administered by the Physical Plant.s energy management group and the OC has access to it via the internet.
Network hubs were down in many locations all over campus as a result of the multiple power outages and the alarms that would normally have been generated by the APOGEE system were not available.The software that is used to monitor the power and air-conditioning equipment in the Data Center is a product purchased from Liebert called SiteScan. It failed as well and as a result the OC staff did not receive any notification that the Data Center was overheating until nearly 5AM when one of the vendors automated call home features sent out a notification that it had reached a temperature in excess of 100 degrees.
Soon after that the mechanical room where the UPS is located overheated to the point that the UPS shut down and went into by-pass mode. An attempt to put the UPS back online resulted in a total black out of the whole facility.
There are some obvious improvements that need to be made to assure that the OC is not depending on the Internet to monitor the machine room environment and we have already taken steps to correct that.
Anyone needing more information about this outage may contact John Rauser at 979-845-8461. This is also the number to call to register a complaint or offer helpful suggestions.
Posted by jeff at 05:26 PM | Comments (0)
August 10, 2007
OpenVPN success
I finally got the spare cycles to nail down our OpenVPN configuration, and it is wonderous (when I redesigned our web application architecture, I went ahead and put almost everything on a private network which I had been accessing via ssh... unfortunately, some of the interfaces I needed to access were web-based).
I now need to cook up some sort of web application for generating certificates for the rest of the office so that they can get on the tasty action.
Posted by jeff at 03:00 PM | Comments (0)
July 22, 2007
Career change
So, because I finally got the chance to tell my father and stepmother in person (my mother and stepfather had to make due with a phone call, driving to Alabama can bite me...), I can make a general announcement in my blag.
I've enlisted in the Unites States Marine Corps as a linguist. To be technically accurate, I swore in to the Delayed Enlistment Program, and while I'm currently on an open contract with an October 15th ship date, I will get my job changed to linguist before I leave.
I've already told most of the people who needed to know personally from me, well, personally :) For the rest of you, if you're curious about reasons then I'll write about a few.
First, I truly want to serve my country. If I had my college degree, there might have been other agencies I would choose first, but I absolutely respect the Marine Corps, and I'm excited about the chance to become a Marine.
Second, I'm sick and fucking tired of working with computers. My job is best described as a "computer plumber", and while it has interesting aspects and possibilities for significant compensation, I have been unhappy with my work for well over 6 years. Not the people I work with, mind you, but rather I've been unhappy being a puzzle solver that doesn't feel his work is really all that useful in the bigger picture.
Third, counter-intelligence has interested me as a field for... well, a very long time. This job in the Marines will give me some exposure to that world, and let me decide if I'm interested in making a career of it.
I'm currently intending to be an Arabic linguist. No, I didn't have any previous experience with the language before deciding. Since DEPing in, however, I have gotten a Rosetta Stone account to try to pick up some basics.
Posted by jeff at 03:24 PM | Comments (3)
January 24, 2006
Once more into the hellmouth...
At work, we're creating a new SAN out of more-or-less off the shelf gear. We have a hodge-podge of operating systems: Novell (primary SAN user), FreeBSD, Linux, Irix, Solaris. We decided to go with our favorite el-cheapo RAID cabinet vendor, ACNC for their high-density SATA-FC units, and the qlogic starter SAN kits for HBAs and switches.
Anywho, I've been on the spear's end for this, which is exciting. I was having problems getting FreeBSD to work with the qlogic HBAs, however. The isp driver was loading, but the JetStor unit was not being detected by the OS (despite being seen by the card's BIOS). It turns out that the ispfw module needs to not be a kernel module, but rather needs to be compiled in the kernel, for it to work properly with the isp driver being compiled in. Go figure.
Here's hoping Google indexes this, so some other poor sap doesn't spend half of forever cursing before figuring out they need a custom kernel :)
Posted by jeff at 02:07 PM | Comments (0)
January 17, 2006
podcasts, iTunes, ITM, etc.
Just a brain dump for the Internet. iTunes version 6.0 does not in fact pull album art from the itunes namespace in the RSS feed as one would hope. Instead, it appears that the ITMS might use the field, but iTunes itself relies the JPG being added to the mp3 (yes, this is possible. No, I didn't know about it either) using the "OTHER" filetype. I used eyeD3. Version 2.3 of the ID3 spec seems to work best.
Jesus christ was that hard to debug.
Posted by jeff at 07:14 PM | Comments (0)
September 15, 2005
Web programmer stabbing
Timeline:
Late June, Early August: We notice that at certain times of the day, websites are just outright inaccessible. It seems unrelated to a previous problem with a stray webdav semaphore file, and to be more related to the log analysis that ran every four hours. Over the next week or so I index the mysql tables used for storing Apache logs.
Aug 22-23: Severe user complaining is brought to my attention. I move the log analysis job to only one run a day (during the early morning, when it should not affect usage). There is still some slowness, so I put up a Zope 2.8.1 test server (for eventual migration).
Aug 31: After a week of little testing by the web programmer, it's requested that I instead make the NetApp NetCache box we have sitting on a rack (that was NEVER POWERED ON, and about which I HAD NO DOCUMENTATION) work to cache pages.
Sep 02: I get the NetCache configured for a test site, and notify my boss and the programmer.
Sep 04: I'm told to go ahead and start moving live sites to the cache for testing. I submit a request to get its port opened through the TAMU firewall.
Sep 06: I rerequest the port be opened. Urgently.
Sep 08: CIS helpfully closes the ports to the production webserver. Stabbings ensue.
Sep 13: I'm told that www.isc.tamu.edu and some other site I setup for testing around the 6th are fine, and to start moving more production live sites over. I move about 15 hosts over. I request that the programmer do even further testing on the 14th on the new site.
Sep 14: A fairly important site is found to be down. The web programmer apparently tried to enable page caching using plone.org documentation.
Sep 15: My boss relays frantic emails from users. The NetCache is delivering pages of logged-in users to everyone. Login credentials themselves don't seem to be affected, but this issue is considered to be huge and cross-browser.
Now, we got the problem fixed eventually (web programmer had to add a no-cache pragma for things to work again), but I still gotta wonder: WHERE WAS THE TESTING? Look, I'm lazy, I admit it. But testing the functionality of websites I shouldn't even have logins to seriously is not my responsibility. And if it's going to be my responsibility, I need to know ahead of time so that customer-visible changes get tested, and less shit gets broke.
Oh well.
Posted by jeff at 04:07 PM
July 22, 2005
Loadbalanced media servers
One of my more interesting projects over the past few months was an "upgrade" of our media streaming servers.
The configuration is a shitty Dell 100mbit switch hooked via a gigE uplink to CIS's network. On that switch there are 2 media servers, 2 load balancers, a webserver, and a NetApp fileserver.
The media servers were originally RedHat 7.something boxes, with the Real Helix server on them. I wiped one box and moved it to Slackware (I also shot a man in Reno just to watch him die.) The loadbalancers were originally running RedHat and the shittiest virtual software in existence . I notice they ported that crap to FreeBSD, and I have no idea why anyone in their right might would cripple themselves that way.
Anyway, I moved the loadbalancers to FreeBSD. They run CARP between them, and the Helix servers are set to use the CARP virtual address as their default gateway (yes, this works. No, it isn't something you are normally taught to do.) The FreeBSD boxen also run pf, and have a round robin rdr rule to balance media traffic between the linux Helix servers. I wrote up a script to remove a server from the pool if it pings out automatically, it's available here for anyone interested. It's not particularly complicated, but the design is rather robust. I was able to test it by rebooting the loadbalancer boxes with minimal service disruption, something that was often problematic under Linux.
Posted by jeff at 12:19 AM | Comments (0)