Recently, I wrote , where I briefly mentioned that it's possible to create your own configuration system so that the service works well in Kubernetes, retrieves secrets, and runs conveniently locally, even completely outside of Docker. It's nothing complicated, but the described "recipe" might be useful to someone 🙂 The code is in Python, but the logic is not tied to the language.
The backstory of the question is as follows: there once was a project, initially a small monolith with utilities and scripts, but over time it grew, dividing into services that, in turn, became microservices, and then began to scale. Initially, everything was run on bare VPSs, and the processes of setting up and deploying code were automated using Ansible. Each service had a YAML config created with the necessary settings and keys, and a similar config file was used for local runs, which was very convenient as this config is loaded into a global object accessible from anywhere in the project.
However, the growth of the number of microservices, their interconnections, and , forewarned a move to Kubernetes, which is still in process. Together with addressing the mentioned challenges, Kubernetes offers its approaches to infrastructure management, including and . The mechanism is standard and reliable, so it’s literally a shame not to take advantage of it! However, I would like to maintain my current format of working with the config: firstly, to use it uniformly across different microservices of the project, and secondly, to be able to run code on a local machine using one simple config file.
In this regard, the mechanism for constructing the configuration object was refined to work with both our classic config file and the Kubernetes secrets. A stricter config structure was also defined, in Python 3 terms, like this:
Dict[str, Dict[str, Union[str, int, float]]]
That is, the final config is a dictionary with named sections, each of which is a dictionary with values of simple types. The sections describe the configuration and access to resources of a certain kind. Here’s an example snippet from our config:
admin panel:
django_secret: "ExtraLongAndHardCode"
db_main:
engine: mysql
host: 256.128.64.32
user: cool_user
password: "SuperHardPassword"
redis:
host: 256.128.64.32
pw: "SuperHardPassword"
port: 26379
smtp:
server: smtp.gmail.com
port: 465
email: info@test.com
pw: "SuperHardPassword"At the same time, the field engine database can be set to SQLite, while redis configured to mock, specifying the filename for saving – these parameters are correctly recognized and processed, allowing for easy local code execution for debugging, unit testing, and various other needs. This is particularly relevant for us, as there are many such needs – part of our code is designed for various analytical calculations, running not only on orchestrated servers but also via different scripts, and on analysts' computers, who need to develop and debug complex data processing pipelines without worrying about backend issues. By the way, it’s worth sharing that our main tools, including the configuration code assembler, are installed via setup.py – together, this unites our code into a single ecosystem, independent of the platform and method of use.
The pod description in Kubernetes looks like this:
containers:
- name: enter-api
image: enter-api:latest
ports:
- containerPort: 80
volumeMounts:
- name: db-main-secret-volume
mountPath: /etc/secrets/db-main
volumes:
- name: db-main-secret-volume
secret:
secretName: db-main-secretThat is, each secret describes one section. The secrets themselves are created like this:
apiVersion: v1
kind: Secret
metadata:
name: db-main-secret
type: Opaque
stringData:
db_main.yaml: |
engine: sqlite
filename: main.sqlite3Together, this leads to the creation of YAML files at the path /etc/secrets/db-main/section_name.yaml
And for local launches, a config located in the root directory of the project or at the path specified in the environment variable is used. The code responsible for these conveniences can be found in the spoiler.
config.py
__author__ = 'AivanF'
__copyright__ = 'Copyright 2020, AivanF'
import os
import yaml
__all__ = ['config']
PROJECT_DIR = os.path.abspath(__file__ + 3 * '/..')
SECRETS_DIR = '/etc/secrets'
KEY_LOG = '_config_log'
KEY_DBG = 'debug'
def is_yes(value):
if isinstance(value, str):
value = value.lower()
if value in ('1', 'on', 'yes', 'true'):
return True
else:
if value in (1, True):
return True
return False
def update_config_part(config, key, data):
if key not in config:
config[key] = data
else:
config[key].update(data)
def parse_big_config(config, filename):
'''
Parse YAML config with multiple sections
'''
if not os.path.isfile(filename):
return False
with open(filename) as f:
config_new = yaml.safe_load(f.read())
for key, data in config_new.items():
update_config_part(config, key, data)
config[KEY_LOG].append(filename)
return True
def parse_tiny_config(config, key, filename):
'''
Parse YAML config with a single section
'''
with open(filename) as f:
config_tiny = yaml.safe_load(f.read())
update_config_part(config, key, config_tiny)
config[KEY_LOG].append(filename)
def combine_config():
config = {
# To debug config load code
KEY_LOG: [],
# To debug other code
KEY_DBG: is_yes(os.environ.get('DEBUG')),
}
# For simple local runs
CONFIG_SIMPLE = os.path.join(PROJECT_DIR, 'config.yaml')
parse_big_config(config, CONFIG_SIMPLE)
# For container's tests
CONFIG_ENVVAR = os.environ.get('CONFIG')
if CONFIG_ENVVAR is not None:
if not parse_big_config(config, CONFIG_ENVVAR):
raise ValueError(
f'No config file from EnvVar:n'
f'{CONFIG_ENVVAR}'
)
# For K8s secrets
for path, dirs, files in os.walk(SECRETS_DIR):
depth = path[len(SECRETS_DIR):].count(os.sep)
if depth > 1:
continue
for file in files:
if file.endswith('.yaml'):
filename = os.path.join(path, file)
key = file.rsplit('.', 1)[0]
parse_tiny_config(config, key, filename)
return config
def build_config():
config = combine_config()
# Preprocess
for key, data in config.items():
if key.startswith('db_'):
if data['engine'] == 'sqlite':
data['filename'] = os.path.join(PROJECT_DIR, data['filename'])
# To verify correctness
if config[KEY_DBG]:
print(f'** Loaded config:n{yaml.dump(config)}')
else:
print(f'** Loaded config from: {config[KEY_LOG]}')
return config
config = build_config()The logic here is quite simple: we combine large configs from the project directory and the path from the environment variable, along with small config sections from Kubernetes secrets, and then preprocess them a bit. Plus some variables. I should note that when searching for files from secrets, a depth restriction is used, as K8s creates another hidden folder for each secret where the secrets are actually stored, while a level above contains just the link.
I hope what I've described will be useful to someone 🙂 Any comments and recommendations regarding security or other areas for improvement are welcome. I'm also interested in the community's opinion on whether we should add support for ConfigMaps (which are not currently used in our project) and whether to showcase the code on GitHub / PyPI? Personally, I think these aspects are too individual for projects to be universal, and a brief look at others' implementations, like the one mentioned here, along with discussions on nuances, advice, and best practices, which I hope to see in the comments 😉
Only registered users can participate in the survey. , please.
Should it be published as a project/library?
0,0%Yes, I would use it / contribute
33,3%Yes, that sounds great
41,7%No, those who need it will do it themselves in their own format and for their own needs
25,0%I will refrain from answering
12 users voted. 3 users abstained.
Source: habr.com
