We hereby announce delays in our Support System due to two ongoing issues.

We ask for just a little patience now as everything is about to be fixed. We will get back to you soon. Please do not open multiple or new tickets, as we prioritize the oldest ones first.

1) ARP Resolving Issues in a Larger (VPS) VLAN (after upgrade to new Routing Stack and changing network style)

Two 40G interfaces (80 Gbit channel) from our Juniper MX960 to part of our network where we run Arista switches were split into 4x10G breakout mode from Juniper to Arista, with multiple links aggregated using LACP. ARP resolution failures and multicast delivery problems were observed. The root cause was traced back to the LACP hash algorithm, which was not appropriately distributing multicast or broadcast traffic (such as ARP requests) across the links.

This issue was mainly seen in a larger VLAN where we placed the VPS nodes due to the volume of ARP traffic.  
Dedicated server clients (or even most VPS customers) might not have noticed any issues as they are either in their own VLAN or routed with their own ASN/IPs in separate VLANs, or if not, in smaller shared VLANs.  
However, in this larger VPS VLAN (where VPS nodes are also connected via this channel), we saw degraded network speeds to some nodes and packet loss.

Resolution:
We are updating the configuration algorithm as well as switching back to a native 2x40Gbit setup to fully resolve the issue tonight.

We have already reviewed our entire network configuration and setup and have worked hard over the past 48 hours to address this.

2) Failure and Hardware Investigation/Replacement:  
10GKVMVPS28.hostslick.com (formerly 10GKVMVPS15.hostslick.com)

Unfortunately, during a very inconvenient time, we experienced a major failure with the 10GKVM15 node, which primarily served some special deals from last year. During the same days when we were busy with network upgrades to move to the new routing stack, this node failed.

We discovered that more than one disk failed in the same mirrored span, which in RAID10 means data loss. We have informed the affected clients about compensation and reinstalled their service.

Further investigation with the SSD supplier was conducted. We use SSDs from the same supplier (just a different model) in many servers. We have never had issues with them before, but it suddenly happened that this specific model used to build this node had many problems. We replaced the failed drives and reinstalled the node. After that, we re-provisioned the clients.

However, two days later, six more disks failed, which was extremely frustrating for us. We had to order new ones, as we did not have 2TB SSDs in stock (the node had 24x2TB SSDs), and we are still in contact with the manufacturer. We have never experienced such an issue before, despite having racks full of VPS nodes. We understand the frustration from affected clients and are doing everything we can to resolve the situation.

We are currently working to get the node back online, and we expect this to happen by Wednesday evening at the latest. During this time, we have created a dedicated support ticket status/category to better organize tickets related to this issue and respond to every client as soon as possible to restore your services fully.

Compensation:
We are offering €10 credit and an additional month of free service to every affected client.

Tuesday, October 15, 2024

« Tilbage