Infrastructure upgrades at OP-EZY HQ

So, you may have noticed the lack of posts here recently, but perhaps you may have seen the status updates on the OP-EZY website (http://status.op-ezy.co.uk/) or the OP-EZY International Facebook page which suggest things have been quite busy at OP-EZY HQ.

An infrastructure upgrade has been on the cards here for a while, after learning some tricks at work for better server management. Unfortunately, finding the time to rebuild the whole platform was difficult. After famously* not using any of my holiday leave last year and needing to take it all at once, I decided this year I’d burn through a few days here and there, and took a week off which was a sensible amount of time to upgrade the servers here.

*Well, not famous, but the butt end of a lot of jokes at work for a while.

The OP-EZY servers, a Dell Poweredge 1950 for dedicated virtual machine hosting, and a Raspberry Pi as the front facing box.

So what’s changed? Well, in the above image, you can see the Dell Poweredge which is used as a Virtual Machine server. Nothing much has changed hardware side there apart from the addition of two hard drives. As for the software, VMWare ESXi has been upgraded, The Ubuntu Linux 12.04 LTS Server Edition and Windows 2008 R2 VMs have been teared down and replaced with two Ubuntu Linux 14.04 LTS VMs and a Windows Server 2012 R2 VM (At the time of writing). A Raspberry Pi (Model B) (running Raspbian Wheezy with all the GUI apps and X Window server removed, which freed up about 1.5 gigabytes of space) has been added to act as a reverse proxy with content caching, perform DDNS updates (using a custom version of DDClient), and keeps the site secure by blocking malicious attacks. The Pi is also used to display a holding page should the main servers require downtime.

Security is something that’s been given a lot more thought this time. First defence lies with our DNS and CDN provider, Cloudflare, which can detect potential attacks on the sites and deals with DDoS attacks. Cloudflare also caches the site to reduce the bandwidth usage. Next is the Pi, which as I mentioned earlier, protects our systems from attacks which includes banning IPs (temporally) if it detects someone, or something attempting a brute force attack (or simply trying to guess passwords too many times). Its role is to also link users to the correct servers depending on what content they’re requesting, and cache the content that’s returned for faster loading times should the same content be requested again, even if it’s for a different user. It will also be able to perform load balancing duties if the platform is expanded. The Pi also has another role in that it’s also being used to monitor the other systems and can alert on errors.

Security of data is also something that’s been given an overhaul. In 2010, we deployed the first server to have RAIDed hard disks to protect against mechanical failure. Whilst this protected data a bit, if something was to happen (a wrong command wiping out a file, or the CMS to the sites got hacked) that would be instantly replicated to the backup drive. The new platform addresses this by taking hourly snapshots of the sites, captured on the second Linux server, then backing up the live snapshots nightly. Similarly, the database from the main server is replicated in real time to the second, and a nightly backup is taken. The nightly backups are stored for a number of days before being considered too old to be useful. The two Linux VMs are stored on separate RAIDed disks, which essentially means at any one time, a recent up to date snapshot exists on four physical disks. Again, if the platform expands, this will also increase.

Site wise, my blog is now running over HTTPS thanks to upgrading to Cloudflare Pro, various improvements have been made to WordPress to make it more secure and faster.

The platform still needs a bit of tweaking as some things have been misbehaving on the new system (The OP-EZY Music site was broken by a PHP upgrade for example) but overall the majority of the work has been completed, and so far things are holding out well, and the sites seem to be a little more responsive.

The Fujitsu ErgoPro x365 desktop computer which hosted the OP-EZY website for many years. – December 2005

Yes, I did recycle that image from an older post.

Things have come a long way since when the original platform launched in 2005 using an old run down Pentium II desktop I had acquired from high school. Who knows what the infrastructure will be like in another 9 years or so. Home hosting has just gotten serious.

Cheers