Advocating Freedom Considered Harmful*
I reviewed the web logs of numerous websites associated with my blog after discovering that my employer, Pacific Northwest National Laboratory was investigating me. The data reduction and presentation was a more difficult and time consuming process than I expected. However the tools developed and the proper presentation paid off. I discovered a strong focus on subject matter that was unrelated to my work as a cyber-security research scientist. From the initial days of the investigation through to the hours before I was fired, my advocation of freedom on my blog was the issue of most interest for the investigators. This abnormal interest explains why there were never any specific allegations of wrongdoing made, why I was never given an opportunity to defend myself against such allegations, why performance reviews and goals were removed from my personnel file, and why I was denied access to the company's Policy and Procedures manual.
I am a senior computer research scientist. My primary interest is in biometrics and encryption/authentication protocols. I have a high level security clearance and until late May of 2005 worked at Pacific Northwest National Laboratory (PNNL). At that time I was suspended without pay pending the results of an investigation. A week later I was fired. Only vague allegations of misconduct were offered as the reason. No opportunity was given to refute the allegations--"That discussion would not be productive."
I am also a civil rights activist. I have been a speaker on using the Internet at two national level Gun Rights Policy Conferences. I am the event director for an annual shooting event that brings people in from all over North America to shoot at high explosives. Newsweek, Outside Magazine, the Seattle Times, and one Seattle television station have made positive mention of me and my pro gun rights activities. I am the webmaster for numerous pro-freedom websites and maintain a blog commenting on topics of interest to me.
The first time I knew anything was wrong inappropriate posting of material on my blog was mentioned as being a point of concern. Therefore reviewing what they saw on my websites was the obvious starting point to discovering their motives in firing me. Log files can contain an overwhelming amount of information. And some web servers produce a very large number of files in a short period of time. For just one of my websites, boomershoot.org, the web server produced nearly 750 files containing over 36 megabytes of information during the month of May 2005. I was the webmaster of 13 websites plus I obtained access to the logs of about a half dozen more sites that were potential targets of the investigation. Assuming the logs could shed light on the mystery this information needed to be reduced and drastically reformatted for the lay person to understand it.
LAX SECURITY HELPED
Fortunately for me the PNNL investigators were very lax in the operational security of their web browsing. Ironically I had repeatedly expressed my concerns to my supervisor Bryan McMillan, numerous co-workers, and on one occasion to Safeguards and Security about the lax security for web browsing the lab as a whole. Nothing was done and I set up my own proxy at my own expense to hide my true origin from the websites I visited. The security was so bad that an outsider doing a nslookup on a PNNL IP address would get the individual machine name the user was browsing from. In most cases it was trivial for an insider to use the computer name to obtain the computer user name. Add in the search strings unknowingly left in web searches and an enemy operating honey pot websites or with access to log files from compromised web severs and the effect on security was disastrous. The enemy could obtain a vast amounts of information about the projects individual researchers were working on or topics they were interested in.
Had the investigators used random proxies for their browsing of my web sites it would have been far, far harder to follow their tracks and probably impossible to be convincing in the conclusions drawn from the information had I been successful in tracking them.
REDUCING AND COMBINING THE DATA
I wrote a simple program that would parse a log file and search for IP addresses of the form 130.20.* All of these belonged to PNNL machines. The fields containing the Date, Time, IP, Target file, Referrer, User Agent information were kept. All others were ignored. One additional field was added--the host name obtained from a call to gethostbyaddr(). Both for performance reasons and my own security reasons I hard coded a cache of the more common PNNL IP addresses that came up and their corresponding host names. If a new IP address was found it was added to the dynamic cache used for that run of the program. The output of the program was a set of records in comma separated value format. Each record was one request for a file on one of my websites. The resultant record for one file request from one of the investigators looked like this:
The alert reader may already note another twist of irony. The investigator machine is infected with the spyware FunWebProducts--a violation of company policy. The two main investigators had this spyware installed on their computers.
The program would accept wild card file names and was run on the set of files for each website and the set of records obtained from hundreds of files was output into a single .CSV file for that website. Each of those files was concatenated into one additional composite file. The .CSV file format is recognized by Microsoft Excel which was used to perform simple database operations--Primarily to sort the composite file by date and time as new log files became available and were added.
FINDING THE INVESTIGATORS
Discovering who the investigators were was surprisingly easy. I recognized most of the machine names found in the composite log file as those belonging to co-workers and my own. The few that were not recognized were either exceedingly infrequent visitors or spent a lot of time pouring through a many, many web pages. The investigators were found.
ANNOTATING THE LOG
The data had been reduced to just a handful of files and a little over a megabyte but was still far too abstract to be understandable. The target file "/archive/2004/02/09/177.aspx" only had a small amount of meaning to me and meant nothing to a lawyer, friend, or co-worker to whom I wanted to explain things.
Manually I could copy and paste the target filename into my browser with the proper prefix (in the above case it would be http;//blog0.joehuffman.org) and could fairly quickly view the file the investigator was interested in. In this case it was a blog posting titled Freedom of (some) expression is supported. By doing this I quickly found the information I was interested in. They were interested in my politics and civil rights activism. But the pattern wasn't easily visualized by others. I had to make it obvious to anyone what was going on. Numerous times I tried to explain, in words, to others what had happened. It just wasn't clicking with them. Because they were my friends and relatives they believed me but the facial expressions of understanding and outrage weren't there.
I should have bit the bullet and learned the API used for the blog database and automated a lot of the process for at least that website. But I didn't and so I manually copy and pasted every web page visited by the investigators into the browser. I created a day by day, second by second account of the investigators month of searching on my websites and those of my family and friends for whom I was able to obtain log files. For each hit on a web page I would note the time, the investigator, the action they made, and create a hyperlink of the web page or blog title. Using the previous record as an example:
The pattern was far more clear now but I still couldn't see the "light go on" when I would show this report to people.
MERGING THE PHYSICAL AND THE VIRTUAL WORLDS
I started looking at another set of data I had--my notes from the three contacts I had with management on this issue. I wondered what actually motivated them to confront me at that time. I looked at the annotated log just prior to the time of the contact. The result made everything much more clear. An example of this follows:
The morning before the meeting PUCK again returned to my web sites leaving still more evidence of his interest by viewing the following web pages:
I arrived at the meeting a minute or two early and there are two people besides Bryan McMillan (my boss) already there. One is from "Safeguards and Security" (I don't remember his name) and the other is Bryan's boss Marty Peterson.
Bryan told me they had some concerns--"Safeguards and Security" would explain. Bryan said that candor and cooperation were important. They were investigating my web site Boomershoot.org and from there found my blog. They didn't like some of the things they found. They were concerned that I had revealed that I had a security clearance and had revealed the general nature of my work. This might make me a target for foreign intelligence. I offered to delete and/or edit all of the postings. They agreed the blog needed to be fixed and I asked for a copy of the printouts (about two inches thick with tabs on probably 15 or 20 pages) they had on the table of the "offending material" to make sure I had found everything. They just told me to "clean up everything". I tried asking several different ways for the list. They declined.
IT ALL MAKES SENSE
It was no wonder why I wasn't allowed to have the printouts. Almost for certain they were unrelated to what they talked to me about. It also explained all the other strange behavior on their part. It explains why I was not confronted with any allegations and given an opportunity to refute them. It explained why my performance reviews and goals were removed from my personnel file before it was sent to me--the performance goals included documentation that I was required to write research papers on my work topics. Research papers would have had far more detailed information on my work than any of the vague postings I made on my blog about my work. That would have exposed it was a pretense for the initial meeting. Based on that pretense they initiated a further investigation into the content of my work computer hard drives. There was another meeting, again with no specific allegations or opportunity to refute any such allegations. The meetings were almost for certain for form only--not function. They were going through the motions of following procedures. Procedures which I was denied access to when I requested them. Specifics would have allowed me to refute their 'charges' which they could not allow to happen if they were to achieve the desired end result--My termination for being an advocate for gun ownership and freedom.
*A play on the title of the classic ACM paper from October 1995, Go To Statement Considered Harmful.
Research Notes and the raw data for this research project are available here.
Last update: January 01, 2007