news

Resolved: Intermittent Service Interruptions for Cloud Customers

In the last few weeks, more than a few (any is too many) MODX Cloud customers experienced unexplained intermittent outages and errors (502/504 errors) on our MODX Cloud public platforms in Amsterdam and Texas. We know how upsetting this is for you and for your clients. We hate downtime as much as you do and we've been pained by this frustrating challenge.

We have been working on this since the outages began. We even brought in additional expertise to add new monitoring and scrutiny to the operation and performance of the MODX Cloud platforms.

We recently discovered that the issues were only affecting people with sites on PHP 5.6, but we could not figure out exactly why it was affecting 5.6 but not 7.1. We initially chalked it up to the massive differences in PHP 7.1 and how much more efficient it is.

Eureka!

On Friday, September 22, 2017, Elizabeth, our Systems Administrator, was reviewing some data and found something that, to her, looked like it was related to PHP's OPcache code (a standard code caching mechanism in PHP) that corresponded to an event on the server. I had an idea to check Google and found a report of a recent issue with PHP's OPcache. It turns out that a change in the behavior of how OPcache works was introduced in PHP 5.6.29. This change created a condition that would cause a deadlock in all running PHP instances, across all pools and masters, thus returning errors related to PHP vanishing or timing out (and the 502s/504s you'd see on the front end).

This answered our question as to why restarting PHP for the affected Clouds would resolve the issue for customers but then recur. Under normal circumstances, PHP would time out and recover resources on its own. Along the way, we found several other issues that we thought were the cause (but, ultimately not the root cause while definitely contributing in their own way).

In the interests of stabilizing the servers we downgraded all public platforms back to PHP 5.6.26. Our monitoring shows that the incidents of PHP locking up are now gone. We will continue to monitor performance and health on our servers, and we're hopeful that this critical bug is resolved in an upcoming PHP 5.6.x release.

The End is Near for PHP 5.6

In recent years, the PHP oversight group has increased the frequency of releases and published the support schedule for releases of PHP. PHP 5.6 is the last release of PHP 5 and will no longer be supported after December 31, 2018.

Current MODX Revolution, and most modern CMS and web applications, will run better and faster on PHP 7+. You can use PHP 7 now in MODX Cloud by switching to it in the Cloud Edit Interface on the Web Server Tab. Soon we'll be switching new Clouds to be PHP 7 by default.

We'll continue to support PHP 5.6 into 2018, however, we encourage all our customers to start moving their sites to PHP 7 now.

If you're not sure if your site or app can run on PHP 7, you can always clone your site by creating a new Cloud and Restoring a backup into that new Cloud. Once you've verified it's working, you can enable PHP 7 on the site and test it thoroughly.

Hi. We’re MODX.

We’re here to help you fix, build and grow fantastic sites. How can we help?




How can we help?

Tell us the general reason for reaching out so we can connect you with the right team.

MODX Diagnostics

MODX’s Open Source software is 100% free for anyone to download and use. As the team behind it for more than a decade, we know it inside, out, and then some.

Like any software, sometimes things break; we can usually fix them very fast. But, we do have to charge for our time to support our families and fund its ongoing development. There are almost an unlimited variety of things that can cause problems, including server upgrades, corrupt files, accidental changes, outdated software, database hiccups and more. We will save you a lot of time and frustration, and get you back in action.

With our MODX Diagnostic service, we determine the source of issues, and often fix them on the spot. For more extensive problems needing more time, like hacked sites or overdue upgrades, we provide additional estimates and guidance. MODX Diagnostics cost $99 for standard business hours support (US Central Time), or $500 for priority, rush or after-hours emergencies.

If you don’t have budget for professional support from the source, you look for answers in the MODX Forums or Documentation, or seek help from MODXers in the Community Slack, or from MODX Professionals near you.

  I’m not ready to pay, let’s talk…

After submitting this form and completing payment, we will collect your access credentials in a secure support ticket. We look forward to helping restore your site back to full health.

Hi! We’d love to work together.

If you have a simple problem that needs our assistance, please request quick fix help here.

What should we keep in mind?

The project involves:
(select all that apply)
What are you planning?
(select all that apply)

Some other considerations

Specific project information

Commercial Support Customers

Customers with a current Commercial Support agreeement can get help using this form. Learn more about MODX Preferred Support.

Let’s get started

What seems to be the issue?

Contact MODX

We welcome conversations, ideas, inquiries and even the occassional cold sales call, but support and requests about how to use MODX software sent via this form cannot be guaranteed a response. That said, we try to respond to everyone that reaches out to us within two business days.

To report a security issue or file a bug for MODX software, please email security [at] modx.com to reach our security team. If you are looking for help with MODX, many times you can find an answer in the MODX Forums or MODX Documentation, from MODXers in realtime at the MODX Community Slack Channel, or from a MODX Professional near you.

How can we help?