Tims Blog

Perceived speed is a great way to make your end users think that your application is snappy.

It’s especially important in regards to web applications because people have no tolerance for waiting on the web. We want things to be responsive, snappy, fast. Developers can give us that, but it takes more foresight (and experience in failing to do this) than it does to take the easy way out.

The easy way out is to send everything back and forth between the browser. It’s easy because you do not need to keep track of anything. You put it all out there and you get it all back.

The problem arises when it comes time to deal with larger volumes of data. You did plan on your software managing larger volumes of data than you tested with…right?

Most people don’t. I know I didn’t. I figured with nessquik “why do I need to page this target list? I can’t imagine any single person having more than 50 or so targets”. And then I found myself adding site-audit functionality….with exclusion lists of 1500+ targets. In the words of the grail knight from The Last Crusade, “he chose poorly”

I had made the decision to send everything back and forth between the client and server, and my code on the backend operated this way. However, to do this was unacceptable because it caused the user experience to be abysmal. Loading several meg of HTML into the browser makes most browsers kack.

This problem turned me in the direction of paging. I could page the target list, only loading a small subset, 15 or so, and then loading the rest on demand, a page at a time.

This works, but requires that the backend be modified to support paging and the front end be modified to support paging. In addition to that, I also need to modify the backend and frontend to support updating of the audit targets in a totally different way; by using only the changes.

Since the whole list will usually never be on the screen, I can no longer rely on sending the whole list back to the server to be saved. I now needed to track the changes to the list and apply those changes on the backend when the audit is saved.

These changes are not overly complicated, but they require time, and time is a precious commodity.

Unfortunately, you usually do not have these revelations until you’ve experienced doing it wrong. Even for small applications, or applications that are home-grown and will stay in-house, you develop the false belief that your data volume will always ever be small. That’s usually not the case.

So I was asked to get on my knees and suck some major dick today at work in the form of an article for our org’s daily newsletter that makes its rounds every day into my spam folder.

Beware that there is quite a large amount of ass-kissing and sucking up to a particular vendor. The majority of the article isn’t worded as I worded it (editors) but it sure makes me come off as if I have my lips firmly attached to the ass of every member of splunk’s board of directors.

So please, for heaven’s sake. If you read the article, do not come to this blog and give me praise for how glowing of an article I wrote for splunk; it isn’t like that. Splunk is cool, no doubt, but they are far from where they were a couple years ago.

For whatever reason, the process of moving virtual machines in Virtualbox from my old computer to my new computer took way too long. This explains the website being down. Everything should be back to as normal as possible now though

Going on three days now I’ve been wfh and one thing I’ve come to realize is that while I’m more productive when I’m not in the office, I also feel like I’m more alienated from my job.

I also feel guilty for some reason, despite that I have legitimate reasons to be away. I guess I get the feeling that I should be taking these days as vacation, but truth be told, I’m not vacationing while I’m here; I sit in my office chair and do work. So why do I get the feeling that my coworkers are judging me.

Ok, shit can officially stop breaking at my house. kthxbye

• java decompiler project
• moar rawr
• lcd parts
• inspiron 9400 inverter
• dell lcd inverter
• burp suite
• attack and defense labs tools

Bluecoat Web Proxies include a command on the CLI that does not show up if you use the help context. Well, actually, this is an understatement. Bluecoat webproxies have way more that just this one command.

The command is called get-now-noblock and is available from the following enable menu

content-filter > local > download get-now-noblock

You can verify this by trying it, or do like I did and decompile the applet.jar file that provides their user interface.

Originally I thought it was speaking HTTP as I was trying to capture the URLs it was POSTing to so that I could script something that would let me control local database downloading remotely.

After decompiling the applet jar though I found out that the stupid thing is literally sending freaking CLI commands to the backend. What a cop out.

Hey, I can cop out like the best of them. So I made an expect script that will do what I need to do.

So in case you’ve been wondering why the site was down, it was because we had a severe failure of APC and surge protectors during a raging thunderstorm.

At about 3 am on Wednesday morning, a bolt of lightning struck so close to my house that the thunder from the hit shook the entire building, set the firealarms off, and tripped the circuit breaker for the bedrooms on my upper floor.

It scared the living shit out of my sister and I too. I literally lept out of bed.

The UPS’ that I have on everything in my house started beeping, and I ran over to make sure things were shutting down as they should.

I walked downstairs, visibly shaken, to see what else was fucked beyond all recognition. Amazingly the TV, XBox, Wii and bigger items like the AC, fridge, etc were ok.

The networking in the house wasn’t so lucky.

All my little netgear switches were shot to hell, as was my DSL modem, and my alarm clock in my room was acting wonky. Also, my backup server was turfed. And the server that hosts the webserver VM’s was turfed too; woohoo. Also my desktop wouldn’t turn back on for some reason.

Disks were fine in all of the machines, but I figured they’d be. So it was more a case of “ok, gotta scramble to fix fried hardware” than scramble to recover lost data.

In the end I had to drop about 90 bucks on new switches and a DSL modem to replace those that were toast.

I got the backup server back online, but the other server is screwed insofar as it’s embedded NIC doesn’t transmit anymore and when you put a NIC in it’s PCI slots, it doesn’t boot; fuck it.

I also dropped coin on a new Dell to replace servers/be new desktop. 600 bucks netted me an AMD 2.8 GHz quad core with 6 gig of ram and a 750 gig disk; man this stuff is getting cheap. All the rest of the broken shit and old hardware I had I tossed; good riddance.

So after a lot of copying stuff around, it’s all back up and going again.

I rode my motorcycle up to Blackhawk Campgrounds in Milton Wisconsin this weekend to hang out with friends at a camper. Man, 120 miles is a lot farther on a motorcycle than it is in a car.

The whole trip but about 350 miles on my bike; 2 days of riding. I got home the other day, after riding in ~90 degree weather at between 75 and 80 mph with killer head and cross winds, and walked in the house and collapsed. I was exhausted.

I slept for the next 13 hours and felt sore as hell when I woke up. I feel a better now, but man, the memories of these last 2 days. I got to sleep at about 4 am Sunday morning, crashing on a futon in the camper; it was the most comfortable futon I’ve ever slept on. I can’t say whether it was the AC or the fact that it was 4 am and I had had a busy day that contributed most to the head-hits-the-pillow-and-lights-out sleep.

Saturday started with the 120 mile ride north. I got there at ~10 am, already tired from the ride, and got the next hour to relax and doze until 11 when we did what I originally had intended to do when I got there; go riding.

So we skipped around Wisconsin for the next 100 or so miles stopping at dive bars along the way. The weather was fine while we were moving, let’s put it that way. Moving under 10 mph for any amount of time was miserable. We got back to the camper 5 hours later and I was ready to throw in the towel, but thats when they started on dinner and cracked open the booze. Hey, why not it’s 4 pm and its the weekend.

So from that point until roughly 4 am I ate food from a bunch of my friends friends campers, hung out, drank whatever was around, although I really hate beer, so I prefer the harder stuff mixed with anything that removes that alcohol flavor, bleh (I must have some repressed memories from when I was a kid; the taste of alcohol reminds me of hospitals, and I was in them waaaaay too much for your average kid; lousy CAH)

So the rest of the night and next morning was a mix of that. So it’s not unusual to feel not too happy about then having to top that off with another 120 mile ride home the next day. I woke up too early considering the night before, and plodded around outside until about 9 am. Once everyone else was awake and beginning to lounge again, I pulled myself together and said my goodbyes; getting on the road at roughly 1 pm.

At about 3:30, and after taking a wrong turn no thanks to my GPS and it’s burning desire to have me take a route I didn’t want to take, I pulled up at the house just as Natalie was getting ready to leave with some new biker friends she had met. I guess they had gone down to Starved Rock that day. I made some idle conversation but was really too tired to do anything so that’s when I made my way back inside, showered, and crashed in bed.

Thank god I have today to recover : )

I tried making a simple splunk app today that would chart a number of metrics pull from out netflow resolver stats.

The end result was this

Pretty cool!

It was much easier than I thought it was going to be and there is a LOT of stuff you can do in a splunk app. I wonder how I can make it even more awesome. Suggestions?

My netflow resolver had a slight problem with its first go around.

To sum up the problem, we have a server that sends us 2 netflow feeds. We receive the netflow and pipe it through some commands to get it to the point of ascii text that we then dump in a text file.

Then, when we start resolving things, the resolutions go to our site DNS server which then talks to the outside world. These external queries that our DNS servers are making are generating more netflow which we then log and that means more stuff to resolve via DNS. You can see the vicious loop that starts.

The end result of this is that node would run at 100% CPU. In addition to that, it eventually killed itself because it ran out of memory.

So I made some changes. First, I added the ability to blacklist certain flows from processing. So I just skip over any flows that are related to the DNS servers. This takes a HUGE load off of the whole system.

Next, I added the ability to blacklist certain IP addresses from resolution. So I now exclude addresses in each flow record that are “on site” addresses. This takes another huge load off the resolver.

Where did these changes bring me?

node now runs consistently at ~10% CPU. The total amount of RAM used is ~60 meg. The resolver stack tends to consistently have 1500 – 2000 jobs waiting to return from the DNS servers; aka, it keeps up.

This is really sweet. node ftw

Part of this node.js twiddling has been me learning the nuances of javascript. Among other things was learning about callback and variable scope. I’ve included the stackoverflow links that REALLY helped me learn this stuff.

• pywebsocket
• ape
• node
• js callbacks getting unexpected value for variable
• javascript date format
• websockets now available in google chrome
• jquery-websocket
• socket-io.node
• pashnit california motorcycle roads
• motorcycle roads in illinois
• how to install cigarette lighter type power outlets on motorcycle
• stopbadware ip address report