We are currently investigating an internal issue regarding incoming calls. We have already rolled back an update which have resolved the issue at 8:30.

We will come back later with more details.

Update 9:30

Last night we were notified about database inconsistency and we were forced to apply an emergency fix. That fix was implemented at 7:50 this morning.
At 8:06 we were notified about high database load because of failures of delivery of incoming calls. After investigation we decided to rollback the fix at 8:29. That resolved the issue with the delivery of incoming calls.

Our apologies for any inconvenience which may have caused by this outage.

Please find below the Root Cause Analyse:

Due to unexpected inconsistency of the semantics of the configuration language between versions, part of our internal configuration was invalidated which caused corrupted or invalid CallerID values to be used and to be sent to our subscribers. For example we have sent From headers like From: "<null>" <sip:<null>@sip.onecentral.eu>;tag=as1a26c112 instead of From: "31881118111" <sip:31881118111@sip.onecentral.eu>;tag=as1a26c112.

Most of our subscribers rejected those calls due to invalid format of the CallerID as well as invalid characters in the header.

In addition due to the corrupted state of INCOMING calls, a very large amount of calls were attempted to be terminated due to internal fail-overs which caused an higher than usual load of certain database components which in turn slowed down our investigation.