All servers are temporarily closing down in 30 minutes to move over the databases from the old Summit/Valley server to the new server. This shouldn’t take long, but since there can be unexpected issues we cannot give an ETA. As soon as Meadow and Atoll are back up we will begin moving Summit, Valley, and lobby to their new server – so they will be down for longer. We apologize for the short notice, we decided to do this as soon as we could because of the high player counts.
The community goal we started has been reached and I’ve ordered the server. Sadly some parts are out of stock so we won’t be able to upgrade right away but as soon as it’s delivered I’ll start work on it. For now, everyone can enjoy the 25% sale for the next 24 hours!
As many of you know the physical server Atoll and Meadow are on has been upgraded, the one Summit and Valley are on is still the same. I want to upgrade these but especially the initial costs aren’t cheap. That’s why I’ve decided to add a temporary Community Goal in the donation store. When it’s reached it will start a 25% sale for 24 hours as a thank you and a small celebration. Altitude would not be possible without the donations from our donors, thank you everyone!
Meadow, Atoll, and Creative will be temporarily down some time tomorrow to move them over to the new server. There will be a 5 minute warning before they go down and if there is an ETA it will be on our twitter page.
If they can’t be fixed all maps will have to be deleted to properly start syncing maps. I’ll talk to the developer of the plugin tomorrow to see if we can do anything to keep the maps. Expect an announcement as an update to this one somewhere this week.
Some of you are already familiar with how our server is set up, but let me give the rest of you a bit of info: We rent two dedicated servers and a virtual server from OVH, all hosted in Canada. The virtual server hosts our website as well as Waterfall/Bungeecord, which is a proxy for Minecraft servers. This proxy allows you to connect to all of our different servers while logged into each other (without this proxy, /server wouldn’t work). Then all of our Minecraft servers (Atoll, Meadow, Summit, Valley, Creative, Events, and Lobby) get split up among the two dedicated servers. This setup allows us to support everyone who wants to play. If we only had one server trying to host everything, the CPU would be overloaded.
Up until today, the proxy was running on a virtual server that had 2 cores @ 2Ghz and 100Mbps of network bandwidth. We weren’t sure at first if this would be enough, but over the last 5 months it has held up pretty well! Unfortunately, starting this week, every player online started experiencing very bad pings and seemingly random times. We noticed that during these times that CPU load jumped as much as 400%! We aren’t sure if this is a 1.14.x issue, but that’s a potential suspect.
Since this issue was happening pretty frequently, and we didn’t want to drop support for 1.14.x clients, we decided to upgrade our proxy hardware! We’ve moved the website and proxy to a brand new virtual server with 4 cores @ 2.3Ghz and 500Mbps of network bandwidth. So far, things appear to be running very well – CPU load average is about 1/2 of what we were seeing on the old hardware.
We are hoping that this will clear up that ping spike issue, and also provide a better experience overall for our players. Since this is a major change, there may be bugs over the next couple of weeks that we didn’t catch at first, so please bear in mind reboots may be more frequent than normal.
Our host has completed their emergency maintenance and connection to the US has been re-established. Our services should be accessible again.
They are still conducting repairs on the original line, but we do not expect any further downtime. You can view status updates from the host here:
UPDATE 3: They’ve replaced the damaged fiber and are in the process of rolling all of the connections over to the new link.
UPDATE 2: They’ve determined it is an issue with the fiber connection from Washington to Beauharnois (the datacenter location) so it only affects US players.
UPDATE: Our host has confirmed that the cluster of servers that our hardware is located within is currently having some network issues. They are already working on resolving it. There is no action needed by us – we just need to wait.
We are aware that there are some network issues involving our servers at the moment. All three of our physical machines appear to be very slow to respond and occasionally unable to send outgoing packets. As we have not changed any configurations recently, and things were running fine earlier this week, we believe the issue is on our hosts end NOT on our end.
At this time we are investigating the issue but believe that this will only be a matter of waiting for our host to resolve the issue. Sorry for the inconvenience.
The servers are now back online. Things appear to be working correctly, but, again, we will continue to monitor them throughout the next week to make sure there are no issues.
Remember: There may some unexpected bugs and reboots!
We are finally ready to try and transfer everything to our new hardware. I am shutting down the servers in 15 minutes to begin the process. Please note: the servers will be offline for several hours during this process!
Also, players should not notice any difference while using the server. From a client-perspective, everything still works the same. This new hardware changes a lot of things on the backend that will make things very different for administrators, but players will not notice anything different.