Our infrastructure team has finally spotted the network issue and acted preventing it to cause more problems. The 503 errors were a result of our firewall trying to prevent a DDoS attack.
A recent update to the integration with one of our partners, unfortunately collided with some bad QOS issues on the partner's network. As a result a portion of our network got flooded with failed connections, which interfered with the necessary connections between our load balancer and our edoc worker nodes, which are in the same internal network.
As an immediate action, we deactivated such integration, which will be moved to a separated network and have its architecture reviewed, and possibly discussed with the partner.
As a preventive action, we will be reviewing our network distribution for all products so we don't have applications getting penalised.
We are very sorry about this unfortunate incident.
