Developing high-load projects in any language requires a special approach and the use of specific tools, but when it comes to PHP applications, the situation can escalate to the point where it becomes necessary to develop, for example, . This note will discuss the well-known pain of distributed session storage and data caching in memcached, and how we addressed these issues in one of our 'ward' projects.
The culprit is a PHP application based on the symfony 2.3 framework, which the business has no plans to upgrade. In addition to the quite standard session storage, this project was actively using a 'cache everything' policy in memcached: responses to requests to the database and API servers, various flags, locks for synchronizing code execution, and much more. In such a situation, a failure of memcached becomes critical for the application's operation. Moreover, losing the cache leads to serious consequences: the DBMS starts to break down, API services ban requests, etc. Stabilizing the situation can take tens of minutes, during which the service will be horribly slow or even completely unavailable.
We needed to ensure the ability to horizontally scale the application with minimal effort, i.e., with minimal changes to the source code and full preservation of functionality. To make the cache not only fault-tolerant but also to try to minimize data loss from it.
What is wrong with memcached itself?
In general, the PHP extension for memcached 'out of the box' supports distributed storage of data and sessions. The mechanism of consistent key hashing allows for even distribution of data across many servers, addressing each specific key to a particular server in the group, while built-in failover mechanisms ensure high availability of the caching service (but, unfortunately, not the data.).
Session storage is handled slightly better: it is possible to configure memcached.sess_number_of_replicas, resulting in data being saved immediately on multiple servers, and in the event of a failure of one memcached instance, data will be served from others. However, if the server returns online without data (which usually happens after a restart), some keys will be redistributed in its favor. Essentially, this would mean loss of session data, as there’s no option to “fallback” to another replica in case of a miss.
The standard tools of the library are primarily aimed at horizontal scaling: they allow the cache to grow to gigantic sizes and ensure access to it from code running on different servers. However, in our situation, the volume of stored data does not exceed a few gigabytes, and the performance of one or two nodes is more than sufficient. Accordingly, the only useful capability that standard tools could provide would be ensuring the availability of memcached while keeping at least one cache instance operational. However, even this opportunity could not be utilized... It’s worth noting the antiquity of the framework used in the project, which made getting the application to work with a pool of servers impossible. We also shouldn’t forget about the data loss from sessions: the customer was visibly stressed due to users’ mass logouts.
Ideally, it was required to replicate records in memcached and bypass replicas in case of a miss or error. We were assisted in implementing this strategy by .
mcrouter
This is a router for memcached, developed by Facebook to address its issues. It supports the text-based memcached protocol, which allows to scale memcached installations to insane sizes. A detailed description of mcrouter can be found in . In addition to other it can do what we need:
- replicate records;
- fallback to other servers in the group in case of an error.
Let’s get to work!
mcrouter Configuration
I’ll jump straight to the config:
{
"pools": {
"pool00": {
"servers": [
"mc-0.mc:11211",
"mc-1.mc:11211",
"mc-2.mc:11211"
},
"pool01": {
"servers": [
"mc-1.mc:11211",
"mc-2.mc:11211",
"mc-0.mc:11211"
},
"pool02": {
"servers": [
"mc-2.mc:11211",
"mc-0.mc:11211",
"mc-1.mc:11211"
},
"route": {
"type": "OperationSelectorRoute",
"default_policy": "AllMajorityRoute|Pool|pool00",
"operation_policies": {
"get": {
"type": "RandomRoute",
"children": [
"MissFailoverRoute|Pool|pool02",
"MissFailoverRoute|Pool|pool00",
"MissFailoverRoute|Pool|pool01"
]
}
}
}
}Why three pools? Why do the servers repeat? Let's figure out how this works.
- In this configuration, mcrouter selects the path that the request will take based on the request command. This is indicated by the type
OperationSelectorRoute. - GET requests are handled by
RandomRoute, which randomly selects a pool or route from the array objectschildren. Each element of this array is in turn a handler ofMissFailoverRoute, which will iterate through each server in the pool until it receives a response with the data, which will then be returned to the client. - If we were using only
MissFailoverRoutea pool of three servers, all requests would first go to the first instance of memcached, while the others would only receive requests as a fallback when data is missing. This approach would lead to excessive load on the first server in the list, which is why it was decided to generate three pools with addresses in different sequences and select them randomly. - All other requests (which are write operations) are processed using
AllMajorityRoute. This handler sends requests to all servers in the pool and waits for responses from at least N/2 + 1 of them. The use ofAllSyncRoutefor write operations had to be discarded, as this method requires a positive response from all servers in the group—otherwise, it will returnSERVER_ERROR. Although mcrouter will still store data in the available caches, the PHP calling function will return an error and generate a notice.AllMajorityRouteis not as strict and allows for up to half of the nodes to be taken out of service without the aforementioned issues.
The main downside of this scheme is that if the data is not actually in the cache, each client request will effectively result in N requests to memcached—toward the with everything servers in the pool. The number of servers in the pools can be reduced, for example, to two: sacrificing storage reliability, we achieve bhigher level of isolation, as if one controller is broken, the problem is confined to that specific context).greater speed and lower load from requests to missing keys.
NB: Useful links for studying mcrouter may also include and (including closed ones), which represent a treasure trove of various configurations.
Building and running mcrouter
The application (and memcached itself) runs in Kubernetes for us—therefore, mcrouter is located there as well. For building the container we use , the config for which will look as follows:
NB: The listings provided in the article are published in the repository .
configVersion: 1
project: mcrouter
deploy:
namespace: '[[ env ]]'
helmRelease: '[[ project ]]-[[ env ]]'
---
image: mcrouter
from: ubuntu:16.04
mount:
- from: tmp_dir
to: /var/lib/apt/lists
- from: build_dir
to: /var/cache/apt
ansible:
beforeInstall:
- name: Install prerequisites
apt:
name: [ 'apt-transport-https', 'tzdata', 'locales' ]
update_cache: yes
- name: Add mcrouter APT key
apt_key:
url: https://facebook.github.io/mcrouter/debrepo/xenial/PUBLIC.KEY
- name: Add mcrouter Repo
apt_repository:
repo: deb https://facebook.github.io/mcrouter/debrepo/xenial xenial contrib
filename: mcrouter
update_cache: yes
- name: Set timezone
timezone:
name: "Europe/Moscow"
- name: Ensure a locale exists
locale_gen:
name: en_US.UTF-8
state: present
install:
- name: Install mcrouter
apt:
name: [ 'mcrouter' ]()
… and we outline the Helm chart. Interestingly — it only includes a config generator based on the number of replicas (if anyone has a more concise and elegant option — feel free to share in the comments):
{{- $count := (pluck .Values.global.env .Values.memcached.replicas | first | default .Values.memcached.replicas._default | int) -}}
{{- $pools := dict -}}
{{- $servers := list -}}
{{- /* Filling the array with two copies of servers: "0 1 2 0 1 2" */ -}}
{{- range until 2 -}}
{{- range $i, $_ := until $count -}}
{{- $servers = append $servers (printf "mc-%d.mc:11211" $i) -}}
{{- end -}}
{{- end -}}
{{- /* Shifting through the array, we get N slices: "[0 1 2] [1 2 0] [2 0 1]" */ -}}
{{- range $i, $_ := until $count -}}
{{- $pool := dict "servers" (slice $servers $i (add $i $count)) -}}
{{- $_ := set $pools (printf "MissFailoverRoute|Pool|poold" $i) $pool -}}
{{- end -}}
---
apiVersion: v1
kind: ConfigMap
metadata:
name: mcrouter
data:
config.json: |
{
"pools": {{- $pools | toJson | replace "MissFailoverRoute|Pool|" "" -}},
"route": {
"type": "OperationSelectorRoute",
"default_policy": "AllMajorityRoute|Pool|pool00",
"operation_policies": {
"get": {
"type": "RandomRoute",
"children": {{- keys $pools | toJson }}
}
}
}
}()
We deploy it in the testing environment and check:
# php -a
Interactive mode enabled
php > # Проверяем запись и чтение
php > $m = new Memcached();
php > $m->addServer('mcrouter', 11211);
php > var_dump($m->set('test', 'value'));
bool(true)
php > var_dump($m->get('test'));
string(5) "value"
php > # Работает! Тестируем работу сессий:
php > ini_set('session.save_handler', 'memcached');
php > ini_set('session.save_path', 'mcrouter:11211');
php > var_dump(session_start());
PHP Warning: Uncaught Error: Failed to create session ID: memcached (path: mcrouter:11211) in php shell code:1
Stack trace:
#0 php shell code(1): session_start()
#1 {main}
thrown in php shell code on line 1
php > # Не заводится… Попробуем задать session_id:
php > session_id("zzz");
php > var_dump(session_start());
PHP Warning: session_start(): Cannot send session cookie - headers already sent by (output started at php shell code:1) in php shell code on line 1
PHP Warning: session_start(): Failed to write session lock: UNKNOWN READ FAILURE in php shell code on line 1
PHP Warning: session_start(): Failed to write session lock: UNKNOWN READ FAILURE in php shell code on line 1
PHP Warning: session_start(): Failed to write session lock: UNKNOWN READ FAILURE in php shell code on line 1
PHP Warning: session_start(): Failed to write session lock: UNKNOWN READ FAILURE in php shell code on line 1
PHP Warning: session_start(): Failed to write session lock: UNKNOWN READ FAILURE in php shell code on line 1
PHP Warning: session_start(): Failed to write session lock: UNKNOWN READ FAILURE in php shell code on line 1
PHP Warning: session_start(): Unable to clear session lock record in php shell code on line 1
PHP Warning: session_start(): Failed to read session data: memcached (path: mcrouter:11211) in php shell code on line 1
bool(false)
php >Searching for the error text did not yield results, however, the query “” prominently featured the oldest unresolved issue of the project — the memcached binary protocol.
NB: The ASCII protocol in memcached is slower than the binary one, and the built-in key consistent hashing tools only work with the binary protocol. However, this doesn't create any issues for the specific case.
It's done: just switch to the ASCII protocol and everything will work... However, in this case, the habit of looking for answers in played a cruel joke. You won't find the correct answer there... unless, of course, you scroll to the end, where in the section ‘User contributed notes’ you'll find the right and .
Yes, the correct name of the option is memcached.sess_binary_protocol. It needs to be disabled, after which the sessions will start working. You just need to place the container with mcrouter in the pod with PHP!
Conclusion
Thus, with just infrastructure changes, we managed to solve the task at hand: the issue with memcached's fault tolerance has been resolved, and the reliability of cache storage has increased. In addition to the obvious benefits for the application, it provided flexibility while working on the platform: when all components have backups, the administrator's life becomes much easier. Yes, this method has its downsides, it may seem like a ‘hack’, but if it saves money, buries the problem, and doesn't cause new issues — why not?
P.S.
Also read in our blog:
- ‘Practical Use of dapp’ (example: symfony-demo): and ;
- «».
Source: habr.com
