Since 8:50 CEST our messages are failing with internal errors.
We are also experiencing issues, not sure if it is only limited to UK
Thank you!
Did you manage get someone on the outage phone-line, the line keeps “Ringing” when we call ?
No I’m trying as well, but no answer
Also experiencing issues in South Africa, I wonder if it is related to the AWS issue in US East?
1 Like
The outage seems to be resolved, probably related to the AWS issues in US East.
Just seeing these messages.
What number did you call that rang and rang with no answer?
Also, it does appear things are working now. From my initial look, this appears to be related to an issue with one of our upstream providers. We’ll have a full answer once we firmly identify what happened.
I tried it and it is working right now.
Would you mind trying once more - I’m curious if the problem is the number or if it has to do with calling from outside of America. If the call is answered, you can just disconnect or tell them you are testing it.
We are also having the same issue with internal errors, on UK1 this morning. We are now processing email, but unable to rerun any failed email.
Thank you
Can you send me an example url for a message you can’t reprocess and that you don’t mind if I try?
@Paul @Tlek @Tim @Jono @Richard
Thank you all for reporting the issue today. From our initial investigation, the root cause does appear to be the AWS us-east-1 issue from today.
Interestingly, the AWS outage didn’t directly affect any of our services, but it did affect services for the upstream billing and provisioning system we use (ChargeBee). During email processing, our system retrieves information from ChargeBee to determine the active subscription level of our customer, and the ChargeBee service was affected by the AWS outage.
We’re going to continue to look into this, but I expect we will change how we interact with ChargeBee so that these requests are not part of the critical path in processing.
I am very interested to hear of any other reports of our phone not being answered. If you happen to have any timestamps for those calls, that will be helpful as well. Even if the outage/issue today was outside of our direct control, we want to ensure that our phones are answered and you can quickly reach us in the event of any issues like today.
We will publish a final report once we have fully investigated the issue.
Thanks for your support!
Thank you – checking this out.
@Richard I re-played this message and it was processed successfully. If you continue to have issues with replaying messages, feel free to email [email protected] so we can troubleshoot this directly with you. From what I can tell, you should be able to replay any failed message (just click the play button in the console). Please let me know if you are successful.
Hi Travis,
We did a test call and were able to talk to an operator.
Hello Travis,
Thank you for the feedback. Hereby the timestamps where I tried to reach your support by phone:
Note - these calls are from the Belgian Time Zone (UTC +2)
Thank you!! This will be helpful.
Just to confirm, did you let it ring for 16 minutes and 7 minutes and there was no answer?!
That’s correct, the time below the timestamp is the time I waited on the line