Using memcached for MODX Caching

Using memcached to cache all or part of the MODX cache directly in memory can be a great way to speed up a MODX site. I get asked often how to set this up, so I'll try and oblige by taking everyone through the local development environment I am using for testing MODX Revolution with memcached.

Preparations

To use MODX with memcached, there are a few additional pre-requisites for your environment. First, you must have memcached installed locally or on an accessible network box with an appropriate amount of RAM available for it to use. Second, you must have the memcache (or memcached) extension for PHP installed and configured properly. Note there are two different extensions available to PHP for communicating with memcached servers, memcache, which MODX Revolution has supported since it's introduction (xPDOMemCache), and memcached, which MODX Revolution 2.2.5+ will include support for (xPDOMemCached).

I use nginx and php-fpm for my local OS X Lion testing environment, installed via homebrew. Homebrew is an amazing piece of software, and I recommend getting familiar with it if you do development on OS X Lion. My first step in getting memcached working with my environment was installing the appropriate PHP extension and memcached. By using homebrew to install it, I can run my memcached daemons as my current user easily, and it's dependency management allows me to simply install the PHP extension and have everything else automatically installed for me, including libevent and memcached itself.

To install the memcache extension in my homebrew managed PHP configuration, this is all that was needed:

[opengeek@drumshaman ~]$ brew install php54-memcache

I followed a few simple instructions to complete the PHP configuration of the memcache extension, restarted my php-fpm, and that's it.

Now, we will need multiple deamons to keep the MODX cache partitions isolated. I'm going to start 9 daemons starting at port 11211 for this example, each configured to use up to 64Mb of RAM. This is more memcached instances than partitions, but I'll explain why when we get to the MODX configuration.

[opengeek@drumshaman ~]$ memcached -p 11211 -d
[opengeek@drumshaman ~]$ memcached -p 11212 -d
[opengeek@drumshaman ~]$ memcached -p 11213 -d
[opengeek@drumshaman ~]$ memcached -p 11214 -d
[opengeek@drumshaman ~]$ memcached -p 11215 -d
[opengeek@drumshaman ~]$ memcached -p 11216 -d
[opengeek@drumshaman ~]$ memcached -p 11217 -d
[opengeek@drumshaman ~]$ memcached -p 11218 -d
[opengeek@drumshaman ~]$ memcached -p 11219 -d
[opengeek@drumshaman ~]$ memcached -p 11220 -d

Configuring MODX

The slightly trickier part of getting MODX to work completely with memcached is the configuration. Since the MODX cache configuration settings are part of what gets cached according to those settings, we have to inform MODX of these details before it loads the system settings. For this reason, there is a $config_options array in your MODX config file where you can specify settings which are applied before the system settings are loaded.

But before we do that, let's set up all of the partition configuration settings which will tell MODX which memcached instances will store which pieces of the cache. Each partition can have it's own memcached_server settings, which can specify one or more memcached servers using comma-delimited format.

Cache partition settings for memcached

MODX Cache Partition System Settings for memcached

There are a few other partitions, mostly manager related (e.g. lexicon_topics, action_map), which you can configure as well, but if you do not set them, they will use the default partition. This is good enough for the example, but in real deployments, you may want to optimize these partitions as well. Otherwise, the only other System Setting you need to set in the manager is the Caching Handler Class, or cache_handler. Give this a value of cache.xPDOMemCache to have all cache partitions in MODX use the PHP memcache extension (or cache.xPDOMemCached if using the memcached PHP extension in 2.2.5+). Note that when you set this in the manager, it will change the cache handler immediately, but we aren't quite finished yet.

As we mentioned earlier, since the cache settings are themselves stored in the cache, we need to configure MODX so it knows about certain cache settings before any cache loading occurs. MODX provides a simple method for specifying pre-load options via the $config_options array located in the main MODX config file.

Specifically, the partitions for system_settings and db need to be pre-configured before System Settings is loaded from the cache itself. I also enable cache_db here so it can apply to loading System Settings. Without this, System Settings could never utilize db caching, though it is debatable if that is even useful. Finally, you'll also notice the default memcached_server configuration for port 11211 is made here.

$config_options = array (
  'cache_handler' => 'cache.xPDOMemCached',
  'memcached_server' => 'localhost:11211',
  'system_settings_memcached_server' => 'localhost:11212',
  'cache_db' => true,
  'db_memcached_server' => 'localhost:11217,localhost:11218,localhost:11219,localhost:11220',
);

Notice I specified four memcached server instances for the db partition. This gives it 4 times as much memory, 256Mb, to utilize for caching, as memcached keeps track of what server instance a particular key is stored on. You can decide how to configure all of your partitions based on the total amount of data you are storing in them. One way to predict this is by calculating size of the various directory roots of your file-based cache data, which should be similar to how much RAM you will be using for each partition when fully loaded with all cacheable items.

That's it. You have now configured MODX to use memcached to store and serve cacheable data directly from RAM.

Caveats and Gotchas

Though memcached can be a great tool for increasing the scalability and/or performance of your dynamic MODX site, there are a few potential pitfalls you will want to be aware of before deciding if it is the right solution to your perceived problems.

  • A single cache entry is limited to 1Mb in size
  • You can delete only individual entries or flush the entire partition
  • Very fast with local servers, vulnerable to network overhead otherwise

1Mb cache entry limitation

Most cache entries in MODX will never get this large. But if you have tens of thousands of Resources, you may want to exclude Context Settings from using memcached, because if your Context cache entry is larger than 1Mb, it will not be cached at all, and will be loaded from the database on each request. Similar could occur with very large cacheable Resources or huge database result sets. It's not something that will affect many, but those with very large MODX sites should definitely be aware of the possibility.

Cannot remove multiple entries by key prefix or regex

One drawback of using memcached, and the reason why you need to partition the MODX cache into separate, isolated instances, is that you cannot remove a set of cache entries by key prefix or regex as you can with the default file-based (xPDOFileCache) or even with APC-based (xPDOAPCCache) caching. You can delete individual entries, or you can flush the entire instance. This could be a problem with certain custom cache implementations that depend on being able to delete an entire node of cache entries based on their "path" specified in the key name, but does not otherwise negatively affect MODX functionality.

Local versus remote servers

Finally, running MODX against local memcached servers can be extremely fast. But don't expect to connect across just any network to remote memcached servers and see the same kind of performance. Anytime the PHP application has to communicate with a remote server over the TCP/IP protocol, you add overhead that is dependent on the speed of the network communications between those servers. If you want to use separate servers to run your memcached instances, make sure their is adequate network performance between your web server(s) and your memcached server(s).

Monitoring Tools and Information

I found a nice PHP-based tool called phpMemcacheAdmin for monitoring and debugging my memcached servers. It might be useful if you want to give memcached a try with MODX, as it allows you to view all your instances, particular clusters of instances you can configure, individual slabs, and even the slab items stored in each. There are other tools for doing this, but it is convenient having a simple web interface for the job.

I know there are many of you who have been asking how to setup MODX with memcached, so here is a starting point. Please let me know if you have problems getting this working, suggestions for better ways of configuring MODX or memcached, or just want to share you experiences using MODX with memcached. Your participation will help us all deliver more amazing web experiences to our customers and improve the MODX product documentation with regard to cache configuration.

Nice to meet you…

If you’re reading this, chances are you need assistance. Let’s get started:




How can we help?

Tell us the general reason for reaching out so we can connect you with the right team.

MODX Diagnostics

MODX’s Open Source software is 100% free for anyone to download and use. As the team behind it for more than a decade, we know it inside, out, and then some.

Like any software, sometimes things break; we can usually fix them very fast. But, we do have to charge for our time to support our families and fund its ongoing development. There are almost an unlimited variety of things that can cause problems, including server upgrades, corrupt files, accidental changes, outdated software, database hiccups and more. We will save you a lot of time and frustration, and get you back in action.

With our MODX Diagnostic service, we determine the source of issues, and often fix them on the spot. For more extensive problems needing more time, like hacked sites or overdue upgrades, we provide additional estimates and guidance. MODX Diagnostics cost $99 for standard business hours support (US Central Time), or $500 for priority, rush or after-hours emergencies.

If you don’t have budget for professional support from the source, you look for answers in the MODX Forums or Documentation, or seek help from MODXers in the Community Slack, or from MODX Professionals near you.

  I’m not ready to pay, let’s talk…

After submitting this form and completing payment, we will collect your access credentials in a secure support ticket. We look forward to helping restore your site back to full health.

Hi! We’d love to work together.

If you have a simple problem that needs our assistance, please request quick fix help here.

What should we keep in mind?

The project involves:
(select all that apply)
What are you planning?
(select all that apply)

Some other considerations

Specific project information

Commercial Support Customers

Customers with a current Commercial Support agreeement can get help using this form. Learn more about MODX Preferred Support.

Let’s get started

What seems to be the issue?

Contact MODX

We welcome conversations, ideas, inquiries and even the occassional cold sales call, but support and requests about how to use MODX software sent via this form cannot be guaranteed a response. That said, we try to respond to everyone that reaches out to us within two business days.

To report a security issue or file a bug for MODX software, please email security [at] modx.com to reach our security team. If you are looking for help with MODX, many times you can find an answer in the MODX Forums or MODX Documentation, from MODXers in realtime at the MODX Community Slack Channel, or from a MODX Professional near you.

How can we help?