As you may be aware, FOSS Galaxy recently acquired a Dedicated Box to replace some of our VMs. For those of you interested in the technical details:
The new setup
Our new setup is as follows:
- 8 core xeon
- 32GiB Ram
- 2 x 2 TB disks (RAID 1, providing 2TB storage)
- 1 static ip address (host)
- 16 static ip addresses (allocated to VMs)
The server is hosting VMs using QEMU/KVM via libvirt. This makes it easy to manage the machines and their configuration both on the command line and graphically.
The server has 3 virtual networks (and 1 bridge) setup to provide access to the servers.
- default (192.168.122.0/24) – A NATed interface allowing access to the internet for VMs that don’t need dedicated public IP addresses
- fossgalaxy-lan (10.42.10.0/24) – An isolated network to allow communication between VMs.
- fossgalaxy-pub (18.104.22.168/28) – Network to expose our public IP address block to the VMs.
- openvpn (10.9.8.0/24) – VPN to allow management of VMs without exposing them to the net directly.
The hypervisor is running a few services to support the VMs. These services are kept to a minimum and include openssh, openvpn and haproxy.
- OpenSSH The hypervisor permits access over ssh directly over it’s public IP address. This is mostly to prevent the administrators getting locked out of the box as we don’t have KVM access.
- OpenVPN is used for day to day management tasks. It is setup to forward DNS requests to one of our VMs (described below). This allows administrators direct access to the VMs using their internal DNS names, allowing us to avoid exposing these services to the internet.
- HAProxy is used to allow access public access to some web services hosted by the virtual machines. It can also do SSL termination to make it easier to deploy SSL certificates.
- Gitlab server – This VM will replace nexus, our current Gitlab server. It will have 2 public IP addresses allocated to it (one for Gitlab and one for Gitlab Pages). As we will be able to provide a dedicated IP address for pages, we will be able to support custom domains for pages.
- FreeIPA server – Provides centralised user authentication, ssh key management, DNS services to ther other VMs and access control. This helps to prevent the nightmare that is having to maintain different user accounts on every VM. It also gives us a central store for user accounts for services such as E-mail.
- Postgresql Server – presently, all of our services use dedicated containers for their databases. This is a problem because it is quite resource heavy and makes backups troublesome. This VM will be replacing the “one container per database” approach with a central database server. It will also make it easier to scale front-end applications.
- Comms Server – A lot of our applications send or receive emails, a dedicated VM with a public IP address will be taking over this role going forward. This server will also be handling our protocol bridges and alerts.
- Docker Server – We’ve been using docker containers to run a large amount of our infrastructure for a few years now. We may move this to Kubernetes to provide better Gitlab integration but to start with this will be a standard docker server.