In this article, I want to share my experience with NJS, a JavaScript interpreter for Nginx developed by Nginx Inc, describing its main features through a real example. NJS is a subset of the JavaScript programming language that allows extending the functionality of Nginx. To the question Dmitry Volyntsev answered in detail. In short: NJS is nginx-way, while JavaScript is more progressive, 'native', and does not have GC, unlike Lua.
A long time ago…
At my previous job, I inherited a GitLab with a variety of CI/CD pipelines that used docker-compose, dind, and other functionalities, which were migrated to kaniko. The images that were previously used in CI remained in their original form. They worked fine until the day our GitLab's IP changed, and CI turned into a pumpkin. The problem was that one of the Docker images used in CI contained git, which pulled Python modules via SSH. For SSH, a private key is needed, and… it was in the image along with known_hosts. Any CI would fail with a key verification error due to the mismatch between the actual IP and the one listed in known_hosts. A new image was quickly built from the existing Dockerfiles and the option StrictHostKeyChecking no. But the unpleasant taste remained, and the desire to move the libraries to a private PyPI repository arose. An additional bonus of switching to a private PyPI would be a simpler pipeline and a proper description of requirements.txt.
The choice is made, gentlemen!
We run everything in the clouds and Kubernetes, and ultimately wanted to create a small service that represented a stateless container with external storage. Since we use S3, that was the priority. Ideally, it would also authenticate with GitLab (we can code that ourselves if needed).
A quick search yielded several results: s3pypi, pypicloud, and a 'manual' option for creating HTML files for the repository. The last option was quickly dismissed.
s3pypi: This is a CLI for using hosting on S3. We upload the files, generate HTML, and upload it to the same bucket. It works for home use.
Pypicloud seemed like an interesting project, but after reading the documentation, I was disappointed. Despite good documentation and the ability to extend it for my needs, it turned out to be overly complex and difficult to configure. Modifying the code for my tasks would, by initial estimates, take 3-5 days. The service also requires a database. We left it on the back burner in case we couldn't find anything else.
A deeper search revealed a module for Nginx, ngx_aws_auth. Testing it resulted in XML displayed in the browser, showing the contents of the S3 bucket. The last commit, at the time of my search, was a year ago. The repository appeared abandoned.
Turning to the original source and reading , I understood that XML can be converted to HTML on the fly and delivered to pip. After searching a bit more with the terms Nginx and S3, I stumbled upon a JS authentication example for S3 written for Nginx. That’s how I got acquainted with NJS.
Using this example as a basis, within an hour I observed the same XML in my browser as with the ngx_aws_auth module, but everything was already written in JS.
I was very fond of the Nginx solution. Firstly, it has good documentation and many examples; secondly, we get all the benefits of Nginx for file handling (out of the box); thirdly, anyone who knows how to write configs for Nginx will be able to understand what’s what. Moreover, its minimalism compared to Python or Go (if starting from scratch) is a plus for me, let alone compared to Nexus.
TL;DR Within 2 days, the test version of PyPi was already used in CI.
Initially, a check is performed: does the client device support power via PoE? A voltage of 2.8 to 10 volts is supplied, and the input resistance is determined. If the results obtained are satisfactory for powering via PoE, the power device proceeds to the next stage.
In Nginx, the module ngx_http_js_module, is included in the official docker image. We import our script using the directive js_importin the Nginx configuration. The function call is made with the directive js_content. To set variables, the directive js_set, which only takes a function described in the script as an argument. However, we can only execute subrequests in NJS with Nginx; no XMLHttpRequest for you. For this, the corresponding location must be added to the Nginx configuration. And in the script, a subrequest must be described to this location. To be able to call a function from the Nginx config, the function’s name must be exported in the script. export default.
nginx.conf
load_module modules/ngx_http_js_module.so;
http {
js_import imported_name from script.js;
server {
listen 8080;
...
location = /sub-query {
internal;
proxy_pass http://upstream;
}
location / {
js_content imported_name.request;
}
}script.js
function request(r) {
function call_back(resp) {
// handler's code
r.return(resp.status, resp.responseBody);
}
r.subrequest('/sub-query', { method: r.method }, call_back);
}
export default {request}When a request is made in the browser http://localhost:8080/ we reach location /where the directive js_content calls function webhook is described in our script script.js. In turn, the function webhook makes a subrequest to location = /sub-query, using the method (in this example GET) obtained from the argument (r), implicitly passed when this function is called. The response to the subrequest will be handled in the function call_back.
Trying S3
To make a request to a private S3 storage, we need:
ACCESS_KEY
SECRET_KEY
S3_BUCKET
From the used HTTP method, the current date/time, S3_NAME, and URI, a specific type of string is generated which is signed (HMAC_SHA1) using the SECRET_KEY. Then, the string in the form of AWS $ACCESS_KEY:$HASH, can be used in the authorization header. The same date/time that was used to generate the string in the previous step must be added to the header X-amz-date. In code, it looks like this:
nginx.conf
load_module modules/ngx_http_js_module.so;
http {
js_import s3 from s3.js;
js_set $s3_datetime s3.date_now;
js_set $s3_auth s3.s3_sign;
server {
listen 8080;
...
location ~* /s3-query/(?.*) {
internal;
proxy_set_header X-amz-date $s3_datetime;
proxy_set_header Authorization $s3_auth;
proxy_pass $s3_endpoint/$s3_path;
}
location ~ "^/(?[w-]*)[\/]?(?[w-.]*)$" {
js_content s3.request;
}
}s3.js(example of AWS Sign v2 authorization, marked as deprecated)
var crypt = require('crypto');
var s3_bucket = process.env.S3_BUCKET;
var s3_access_key = process.env.S3_ACCESS_KEY;
var s3_secret_key = process.env.S3_SECRET_KEY;
var _datetime = new Date().toISOString().replace(/[:-]|.d{3}/g, '');
function date_now() {
return _datetime
}
function s3_sign(r) {
var s2s = r.method + 'nnnn';
s2s += `x-amz-date:${date_now()}n`;
s2s += '/' + s3_bucket;
s2s += r.uri.endsWith('/') ? '/' : r.variables.s3_path;
return `AWS ${s3_access_key}:${crypt.createHmac('sha1', s3_secret_key).update(s2s).digest('base64')}`;
}
function request(r) {
var v = r.variables;
function call_back(resp) {
r.return(resp.status, resp.responseBody);
}
var _subrequest_uri = r.uri;
if (r.uri === '/') {
// root
_subrequest_uri = '/?delimiter=/';
} else if (v.prefix !== '' && v.postfix === '') {
// directory
var slash = v.prefix.endsWith('/') ? '' : '/';
_subrequest_uri = '/?prefix=' + v.prefix + slash;
}
r.subrequest(`/s3-query${_subrequest_uri}`, { method: r.method }, call_back);
}
export default {request, s3_sign, date_now}A little explanation about _subrequest_uri: this is a variable that generates a request to S3 depending on the original URI. If you need to get the content of the "root", you need to generate a URI request with the specified delimiter delimiter, which will return a list of all XML elements CommonPrefixes, corresponding to directories (in the case of PyPI, a list of all packages). If you need to get a list of contents in a specific directory (a list of all versions of packages), then the URI request must contain the prefix field with the name of the directory (package) that ends with a slash \. Otherwise, there may be collisions when querying the contents of the directory, for example. There are directories aiohttp-request and aiohttp-requests, and if the request specifies /?prefix=aiohttp-request, then the response will contain the contents of both directories. If there is a slash at the end, /?prefix=aiohttp-request/, then the response will only contain the required directory. And if we request a file, the resulting URI should not differ from the original.
We save, restart Nginx. In the browser, we enter our Nginx address, the result of the request will be XML, for example:
List of directories
myback-space
10000
/
false
new/
old/From the list of directories, only the elements we need will be required. CommonPrefixes.
By adding the required directory to our address in the browser, we will receive its contents also in XML format:
List of files in the directory
myback-space
old/
10000
false
old/giphy.mp4
2020-08-21T20:27:46.000Z
"00000000000000000000000000000000-1"
1350084
02d6176db174dc93cb1b899f7c6078f08654445fe8cf1b6ce98d8855f66bdbf4
STANDARD
old/hsd-k8s.jpg
2020-08-31T16:40:01.000Z
"b2d76df4aeb4493c5456366748218093"
93183
02d6176db174dc93cb1b899f7c6078f08654445fe8cf1b6ce98d8855f66bdbf4
STANDARDFrom the list of files, we will take only the elements we need. Key.
We then need to parse the resulting XML and return it as HTML, replacing the Content-Type header with text/html in advance.
function request(r) {
var v = r.variables;
function call_back(resp) {
var body = resp.responseBody;
if (r.method !== 'PUT' && resp.status < 400 && v.postfix === '') {
r.headersOut['Content-Type'] = "text/html; charset=utf-8";
body = toHTML(body);
}
r.return(resp.status, body);
}
var _subrequest_uri = r.uri;
...
}
function toHTML(xml_str) {
var keysMap = {
'CommonPrefixes': 'Prefix',
'Contents': 'Key',
};
var pattern = `<k>(?<v>.*?)</k>`;
var out = [];
for(var group_key in keysMap) {
var reS;
var reGroup = new RegExp(pattern.replace(/k/g, group_key), 'g');
while(reS = reGroup.exec(xml_str)) {
var data = new RegExp(pattern.replace(/k/g, keysMap[group_key]), 'g');
var reValue = data.exec(reS);
var a_text = '';
if (group_key === 'CommonPrefixes') {
a_text = reValue.groups.v.replace(///g, '');
} else {
a_text = reValue.groups.v.split('/').slice(-1);
}
out.push(`<a href="/${reValue.groups.v}">${a_text}</a>`);
}
}
return '<html><body>n' + out.join('</br>n') + 'n</html></body>'
}Trying PyPI
We check that nothing is broken anywhere with reliable packages.
# Создаем для тестов новое окружение
python3 -m venv venv
. ./venv/bin/activate
# Скачиваем рабочие пакеты.
pip download aiohttp
# Загружаем в приватную репу
for wheel in *.whl; do curl -T $wheel http://localhost:8080/${wheel%%-*}/$wheel; done
rm -f *.whl
# Устанавливаем из приватной репы
pip install aiohttp -i http://localhost:8080We repeat this with our libraries.
# Создаем для тестов новое окружение
python3 -m venv venv
. ./venv/bin/activate
pip install setuptools wheel
python setup.py bdist_wheel
for wheel in dist/*.whl; do curl -T $wheel http://localhost:8080/${wheel%%-*}/$wheel; done
pip install our_pkg --extra-index-url http://localhost:8080In CI, creating and uploading a package looks like this:
pip install setuptools wheel
python setup.py bdist_wheel
curl -sSfT dist/*.whl -u "gitlab-ci-token:${CI_JOB_TOKEN}" "https://pypi.our-domain.com/${CI_PROJECT_NAME}"Authentication
In GitLab, it is possible to use JWT for authenticating/authorizing external services. By utilizing the auth_request directive in Nginx, we can redirect authentication data into a subrequest containing a call to a function in the script. In the script, there will be another subrequest to the GitLab URL, and if the authentication data is provided correctly, GitLab will return a 200 code and allow the upload/download of the package. Why not use a single subrequest and send the data directly to GitLab? Because we would then have to edit the Nginx configuration file every time there are changes to our authorization, which is quite a tedious task. Additionally, if a read-only root filesystem policy is used in Kubernetes, this adds more complexity when trying to replace nginx.conf through configmaps. It becomes absolutely impossible to configure Nginx through a configmap when simultaneously using policies that prohibit attaching volumes (PVC) and a read-only root filesystem (this can happen as well).
By using NJS as an intermediary, we gain the ability to change specified parameters in the Nginx config using environment variables and perform checks in the script (for example, for incorrectly specified URLs).
nginx.conf
location = /auth-provider {
internal;
proxy_pass $auth_url;
}
location = /auth {
internal;
proxy_set_header Content-Length "";
proxy_pass_request_body off;
js_content auth.auth;
}
location ~ "^/(?[w-]*)[\/]?(?[w-.]*)$" {
auth_request /auth;
js_content s3.request;
}s3.js
var env = process.env;
var env_bool = new RegExp(/[Tt]rue|[Yy]es|[Oo]n|[TtYy]|1/);
var auth_disabled = env_bool.test(env.DISABLE_AUTH);
var gitlab_url = env.AUTH_URL;
function url() {
return `${gitlab_url}/jwt/auth?service=container_registry`
}
function auth(r) {
if (auth_disabled) {
r.return(202, '{"auth": "disabled"}');
return null
}
r.subrequest('/auth-provider',
{method: 'GET', body: ''},
function(res) {
r.return(res.status, "");
});
}
export default {auth, url}Perhaps the question arises: - Why not use ready-made modules? Everything is already done there! For example, var AWS = require('aws-sdk') and there's no need to reinvent the wheel with S3 authentication!
Let's move on to the downsides
For me, the inability to import external JS modules has been an unpleasant but expected feature. The example above describing require('crypto') is and require only works for them. There is also no opportunity to reuse code from scripts, and one has to copy-paste it across different files. I hope that someday this functionality will be implemented.
For the current project, gzip compression should be disabled in Nginx. gzip off;
This is because the gzip module is not available in NJS, and it cannot be connected, hence there is no way to work with compressed data. However, this is not much of a downside for this case. There is not much text, and the files being transferred are already compressed, so additional compression will not particularly help. It is also not such a heavy or critical service that we need to bother about delivering content a few milliseconds faster.
Debugging the script is lengthy and can only be done through prints in error.log. Depending on the logging level set to info, warn, or error, you can use the three methods r.log, r.warn, r.error respectively. I try to debug some scripts in Chrome (v8) or the NJS console tool, but not everything is possible to check there. When debugging code, aka functional testing, the history looks something like this:
docker-compose restart nginx
curl localhost:8080/
docker-compose logs --tail 10 nginxand such sequences can run into hundreds.
Writing code using subqueries and variables for them can turn into a tangled mess. Sometimes you find yourself jumping between different IDE windows trying to figure out the sequence of your code's actions. It's not challenging, but it can be quite stressful at times.
There is no full support for ES6.
There may be other drawbacks as well, but I haven't encountered any more. Please share your information if you have had negative experiences with NJS.
Conclusion
NJS is a lightweight open-source interpreter that allows you to implement various scripts in JavaScript within Nginx. Great attention was paid to performance during its development. Of course, it still lacks many features, but the project is evolving thanks to a small team that actively adds new features and fixes bugs. I hope that one day NJS will allow the inclusion of external modules, making Nginx's functionality nearly limitless. However, there is NGINX Plus, and some features are likely to be missing!
and
/ Выступление Дмитрия Волныева на Saint HighLoad++ 2019
/ Выступление Василия Сошникова на HighLoad++ 2019
Source: habr.com
