It’s an interesting experience to walk into a new team and company and proceed to start dismantling their infrastructure.

When I started in my team, the person who I replaced was viewed in a pretty negative light.

After he left, actually (and only during one-on-one conversations with his former colleagues) did I actually begin to grasp the level of contempt that they had for him.

You also see that contempt manifest itself in the significant waste in resources, poorly implemented tools and technologies, and technologies not used by the other colleagues.

Take for example the systems that were referred to as “filer” systems.

They were poorly named

When I started, they were named

  • f1.hype.f5net.com
  • f2.hype.f5net.com

They resided in a sub-domain that did not imply their location in the company.

From my past experience at Rackspace this is totally unacceptable. So I renamed them to something that implies cells and regions; as we used at RS.

They have poorly managed ZFS volumes

When I stepped into the group, nothing was monitored. I fixed this by establishing a nagios infrastructure to handle that.

Immediately though it started noticing problems. In particular, the disk pools in the filers were full. This was a problem because backups were no longer happening. However it was a double whammy when I found out that the filers were using ZFS as their filesystem.

The only reason they were using ZFS was because my colleague had installed FreeNAS on the filers; another questionable solution.

While I proceeded to clean those up, I found over 3000 snapshots on the volumes.

I spent a lot time determining what all could be deleted and learning a lot about ZFS along the way.

In addition to that I found that ZFS and big snapshots have terrible performance problems when dedup is turned on. So I turned that off as well and now I think I need to wait for all the existing snapshots to purge their way out of the system.

In any event…more premature optimization.

Useless services

He installed some plugins that were never used. Among them, syncthing and btsync.

When I went to investigate those, I found that directories had been set up for all of the team members to use; but only he had used it.

More wasted effort…

Conclusion

My former colleague talked a lot of shit about another former colleague that left the company right as I started. However, based off of this colleagues ability to establish and run production systems, I would have to say that he’s just as lacking in his professional skills as the person that he himself talked shit about.

I don’t make those same mistakes.

I’m only interested in the enterprise-grade solutions to the problem because my technical interests (my play time) is not done at work or with work resources.

And I also abhore debt; including the technical kind.

It’s been interesting to dismantle this empire though and see just how much stuff is not needed and how badly technologies were abused.