We are publishing the transcript of the talk from the conference again HighLoad++ 2016, which took place in Skolkovo, near Moscow, on November 7-8 last year. Vladimir Protasov discusses how to extend NGINX functionality with OpenResty and Lua.

Hello everyone, my name is Vladimir Protasov, I work at Parallels. Let me tell you a bit about myself. For three-quarters of my life, I have been writing code. I became a programmer to the bone: I sometimes see code in my dreams. A quarter of my life has been spent in industrial development, writing code that goes straight to production. The code you use, though you might not realize it.

To give you an idea of how bad things were. When I was a young junior, I came in and was handed these two-terabyte databases. Nowadays, everyone has high load systems. I went to conferences, asking: "Guys, tell me, you have big data, everything is cool? How large are your databases?" They replied: "We have 100 gigabytes!" I thought: "Cool, 100 gigabytes!" But inside, I was trying to keep a poker face. You think, yes, these guys are cool, but then you go back and deal with these multi-terabyte databases. And that was as a junior. Can you imagine what a hit that was?

I know over 20 programming languages. This is what I had to figure out during my work. You are handed code in Erlang, C, C++, Lua, Python, Ruby, and some other languages, and you have to work with all of it. In general, it was necessary. I couldn’t quite count the exact number, but I lost track around 20.

Since everyone here knows what Parallels is and what we do, I won’t talk about how great we are or what we do. I will just mention that we have 13 offices worldwide, over 300 employees, and development teams in Moscow, Tallinn, and Malta. If you want, you can move to Malta if it’s cold in winter and you need to warm up.

Specifically, our department writes in Python 2. We are in business and don’t have time to implement trendy technologies, which is why we struggle. We use Django because it has everything we need, and we’ve discarded what was unnecessary. We also use MySQL, Redis, and NGINX. Plus, we have a lot of other cool stuff. We have MongoDB, rabbits running around, we have all kinds of things — but that’s not my area, and I don’t deal with it.

OpenResty

I've talked about myself. Let's dive into what I will be discussing today:

What is OpenResty and how is it used?
Why reinvent the wheel when we already have Python, NodeJS, PHP, Go, and other great tools that everyone is happy with?
And a few real-life examples. I had to significantly cut down my presentation because it was initially about 3.5 hours long, so there will be few examples.

OpenResty is NGINX. Thanks to it, we have a fully functional web server that's well-written and operates quickly. I believe most of us use NGINX in production. You all know it's fast and great. It features excellent synchronous I/O, so we don't need to reinvent the wheel like we did with the gevent library in Python. Gevent is cool, but if you write some C code and something goes wrong, debugging with gevent can drive you insane. I had an experience: it took me two whole days to figure out what went wrong. If someone hadn't spent weeks digging into it, found the problem, and posted it online, which Google then discovered, we would have gone completely mad.

NGINX already includes caching and static content handling. You don’t have to worry about how to set it up correctly so that nothing gets sluggish and you don't lose any descriptors. NGINX is easy to deploy; you don’t have to think about what to use—WSGI, PHP-FPM, Gunicorn, Unicorn. You set up NGINX, hand it over to the admins, and they know how to work with it. NGINX processes requests in a structured manner. I'll talk a bit more about that later. In short, it has a phase when it first receives the request, when it processes it, and when it delivers the content to the user.

NGINX is awesome, but there is one problem: it’s not flexible enough even with all the great features the team has crammed into the config, despite the customization options. This power is lacking. That's why the team at Taobao embedded Lua into it about eight years ago. What does that provide?

Size. It's lightweight. LuaJIT adds around 100-200 kilobytes of memory overhead with minimal performance overhead.
SpeedThe LuaJIT interpreter is close to C in many situations, sometimes it falls behind Java, and in some cases, it surpasses it. For a while, it was considered state of the art, the best JIT compiler. Now, there are more advanced ones, but they are quite heavy, like V8. Some JS interpreters and the Java HotSpot are faster in some cases, but still lose out in others.
Ease of learning. If, for instance, you have a codebase in Perl, and you are not Booking, you won’t find Perl programmers. Because they are not available; they've all been taken, and teaching them is long and complicated. If you want programmers in something else, you might also have to retrain them or find some. With Lua, it’s simple. Any junior can learn Lua in three days. It took me about two hours to get acclimated. After two hours, I was already writing production code. Within a week, it was deployed to production.

As a result, it looks like this:

There’s a lot going on here. OpenResty has gathered a bunch of modules, both Lua-based and engine-based. And everything is ready for you — deploy it and it works.

Examples

Enough of the theory, let's move on to the code. Here’s a simple Hello World:

What do we have here? This is the engine’s location. We don’t worry about writing our own routing, nor do we take any pre-made one — we already have it in NGINX, and we’re living well and lazily.

content_by_lua_block is a block that tells us that we are delivering content using a Lua script. We take the engine variable remote_addr and pass it to string.format. This is the same as sprintf, but in Lua, and it’s correct. And we send it to the client.

As a result, it will look like this:

But let's return to the real world. In production, no one deploys Hello World. Our applications usually interact with a database or something else, and most of the time, they wait for a response.

It just sits and waits. This isn’t very good. When 100,000 users come in, it becomes quite difficult. So, let's create a simple application as an example. We’ll be looking for pictures, for instance, of kittens. But we won’t just look randomly, we’ll expand the keywords and, if a user searches for "kittens", we’ll find them cats, fluffballs, and more. First, we need to get the request data on the backend. Here’s what it looks like:

Two lines allow you to retrieve GET parameters, no complexities. Then, we can obtain this information from a database with a table by keyword and extension through a regular SQL query. It's straightforward. It looks like this:

We connect the library resty.mysql, which we already have included. We don't need to install anything, everything is ready. We specify how to connect and make the SQL query:

There's a bit of a scare here, but everything works. Here, 10 is the limit. We are retrieving 10 records; we're lazy and don't want to show more. I forgot about the limit in SQL.

Next, we find images for all the queries. We gather a bunch of queries and populate a Lua table called reqs, and then we do ngx.location.capture_multi.

All these requests go in parallel, and we receive the responses back. The execution time equals the response time of the slowest one. If all of them respond in 50 milliseconds and we sent a hundred queries, the response will come back in 50 milliseconds.

Since we're lazy and don't want to write HTTP processing and caching, we'll make NGINX do everything for us. As you saw, there was a request to url/fetch, here it is:

We make a simple proxy_pass, specify where to cache, how to do it, and it all works.

But that's not enough; we also need to return data to the user. The simplest idea is to serialize everything in JSON, easy enough in two lines. We set the Content-Type and return JSON.

But there’s one complication: the user doesn’t want to read JSON. We need to involve frontend developers. Sometimes we don’t want to do this at first. And SEO specialists will say that if we're searching for images, they don’t care. But if we're giving them some content, they’ll say that search engines are indexing nothing.

What to do about this? Naturally, we will return HTML to the user. Generating it manually is not practical, so we want to use templates. For this, there is a library lua-resty-template.

You probably saw the three scary letters OPM. OpenResty comes with its own package manager, through which you can install a bunch of different modules, in particular, lua-resty-template. This is a simple template engine, similar to Django templates. You can write code there and make substitutions for variables.

As a result, everything will look something like this:

We took the data and rendered the template again in two lines. The user is happy; they received kittens. Since we expanded the query, they also got a sea otter along with the kittens. Who knows, maybe that’s what they were actually looking for but couldn’t articulate their request properly.

Everything is great, but we are in development, and we don't want to show this to users yet. Let's implement authentication. To do this, let's see how NGINX processes requests in terms of OpenResty:

The first phase — access, when the user just arrived, and we looked at them based on headers, IP address, and other data. We can immediately block them if they don't seem suitable. This can be used for authentication, or if we receive a high volume of requests, we can easily cut them off at this phase.
rewrite. We rewrite some request data.
content. We deliver content to the user.
headers filter. We modify the response headers. If we used proxy_pass, we can rewrite some headers before delivering them to the user.
body filter. We can modify the body.
log — logging. We can write logs to Elasticsearch without an additional layer.

Our authentication will look something like this:

We will add this to the location, which we described earlier, and insert the following code:

We check if we have a cookie token. If not, we redirect to authentication. Users are clever and might guess they need to set a cookie token. Therefore, we will also store it in Redis:

The code for working with Redis is very simple and doesn’t differ from other languages. The input/output is non-blocking, both there and here. If you write synchronous code, it works asynchronously. It’s somewhat like using gevent, but done well.

Let’s create the authentication itself:

We state that we need to read the request body. We get the POST arguments and check if the username and password are correct. If they are incorrect, we redirect to authentication. If they are correct, we store the token in Redis:

Don't forget to set the cookie; this is done in just two lines:

This is a simple, theoretical example. We certainly won't create a service that shows people kittens. But who knows. So let's go over what can be done in production.

Minimalist backend. Sometimes we need to output just a little bit of data on the backend: sometimes we need to insert a date, display a list, indicate how many users are currently on the site, attach a counter, or show some statistics. Something small like that. Minimal pieces can be done very easily. This will turn out quickly, easily, and great.
Data Preprocessing. Sometimes we want to embed advertisements into our web page, and we fetch these ads using API requests. This can be done very easily here. We do not overload our backend, which is already working hard. We can gather and assemble things here. We can combine some JS or, conversely, break apart something, preprocess it before sending it to the user.
Facade for Microservices. This is also a very good case, which I have implemented. Before this, I worked at Tenzor, a company that deals with electronic reporting, providing reporting for about half of the legal entities in the country. We created a service where many things were done with this same mechanism: routing, authorization, and more.
OpenResty can be used as glue for your microservices, providing a unified access point and interface for everything. Since microservices can be written in different languages, such as Node.js, PHP, Python, and there may be some piece running on Erlang, we understand we don’t want to rewrite the same code everywhere. Therefore, OpenResty can be implemented on the front end.
Statistics and Analytics. Usually, NGINX is placed at the entry point, and all requests pass through it. It is very convenient to gather here. We can calculate something immediately and send it somewhere, for example, to Elasticsearch, Logstash, or simply record it in a log and then send it somewhere.
Multiplayer Systems. For example, online games can also be done very well. Today in Cape Town, Alexander Gladys will talk about how to quickly prototype a multiplayer game using OpenResty.
Request Filtering (WAF). Nowadays, it's trendy to create various web application firewalls, and there are many services that provide them. With OpenResty, you can create a web application firewall that easily filters requests according to your requirements. If you're using Python, you know that PHP won't be injected at all, unless, of course, you spawn it from the console somewhere. You know you have MySQL and Python. There may be attempts at directory traversal and trying to inject something into the database. Therefore, you can quickly and cheaply filter out suspicious requests right at the front.
Community. Since OpenResty is built on NGINX, it has a bonus — which is the NGINX community.It is quite large, and a considerable portion of the questions you will encounter initially has already been addressed by the NGINX community.
Lua developers. Yesterday, I talked with some guys who came to the HighLoad++ training day and heard that only Tarantool is written in Lua. That's not true; many things are made with Lua. Examples include: OpenResty, the XMPP server Prosody, the game engine Love2D, and Lua is scripted in Warcraft and other places. There are many Lua developers, and they have a large and responsive community. All of my questions about Lua were solved within a few hours. When you write to the mailing list, literally within minutes there are plenty of responses explaining what and how, connecting the dots. It's really great. Unfortunately, not everywhere is there such a kind and friendly community.
There is a GitHub page for OpenResty where you can open an issue if something breaks. There is a mailing list on Google Groups where you can discuss general issues, and there's a mailing list in Chinese—just in case you don't speak English but have knowledge of Chinese.

Summary

I hope I've conveyed that OpenResty is a very convenient framework focused on the web.
It has a low entry barrier since the code resembles what we write, and the language is quite simple and minimalist.
It provides asynchronous I/O without callbacks, so we won't have the spaghetti code that we can sometimes end up with in NodeJS.
It has an easy deployment since we only need NGINX with the required module and our code, and it all works right away.
A large and responsive community.

I didn't go into detail about how routing is done; it would have turned into a very long story.

Thank you for your attention!