Aloha kākou! He mea hoʻomohala CV wau ma CROC. Ke hoʻokō nei mākou i nā papahana ma ke kahua o CV no 3 mau makahiki i kēia manawa. I loko o kēia manawa, hana mākou i nā mea he nui, no ka laʻana: nānā mākou i nā mea hoʻokele i ka wā e holo ana ʻaʻole lākou i inu, ʻaʻole puhi, ʻaʻole kamaʻilio ma ke kelepona, nānā i ke alanui, ʻaʻole i nā moeʻuhane a i nā ao. ; Ua hoʻopaʻa mākou i nā poʻe e holo ana ma nā ala i hoʻolaʻa ʻia a lawe i kekahi mau wahi kaʻa; hōʻoia i ka hana ʻana o nā limahana i nā mahiole, nā mīkina lima, a pēlā aku; ʻike i kahi limahana e makemake ana e komo i ka hale; Ua helu mākou i nā mea a pau.
No ke aha kaʻu e hana nei i kēia mau mea?
Ma ke kaʻina hana o ka hoʻokō ʻana i nā papahana, pā mākou i nā puʻupuʻu, nā puʻupuʻu he nui, kekahi o nā pilikia āu i kamaʻāina ai a kamaʻāina paha i ka wā e hiki mai ana.
E hoʻohālike kākou i ke kūlana
E noʻonoʻo kākou ua loaʻa iā mākou kahi hana ma kahi ʻoihana ʻōpio "N", nona nā hana e pili ana iā ML. Hana mākou ma kahi papahana ML (DL, CV), a laila no kekahi kumu e hoʻololi mākou i kahi hana ʻē aʻe, hoʻomaha maʻamau, a hoʻi i kā mākou neuron a i ʻole kekahi.
- Hiki mai ka manawa o ka ʻoiaʻiʻo, pono ʻoe e hoʻomanaʻo i kahi āu i kū ai, he aha nā hyperparameters āu i hoʻāʻo ai a, ʻo ka mea nui loa, he aha nā hopena i alakaʻi ai lākou. Hiki i nā koho he nui no ka mea nāna i mālama i ka ʻike ma nā neʻe a pau: ma ke poʻo, configs, notepad, ma kahi ʻoihana hana i ke ao. Ua ʻike wau i kahi koho i ka wā i mālama ʻia ai nā hyperparameters e like me nā laina i ʻōlelo ʻia ma ke code, ma ke ʻano holoʻokoʻa, he lele o ka nani. E noʻonoʻo ʻoe ʻaʻole ʻoe i hoʻi i kāu papahana, akā i ka papahana o kahi kanaka i haʻalele i ka hui a ua hoʻoili ʻoe i kahi code a me kahi kumu hoʻohālike i kapa ʻia ʻo model_1.pb. No ka hoʻopiha ʻana i ke kiʻi a hōʻike i nā ʻeha āpau, e noʻonoʻo kākou he loea hoʻomaka ʻoe.
- Hele i mua. No ka holo ʻana i ke code, pono mākou a me nā mea a pau e hana pū me ia e hana i kahi kaiapuni. He mea pinepine ia no kekahi kumu ʻaʻole lākou i haʻalele iā ia i hoʻoilina no mākou. Hiki ke lilo kēia i mea hana ʻole. ʻAʻole ʻoe makemake e hoʻopau manawa ma kēia ʻanuʻu, ʻeā?
- Hoʻomaʻamaʻa mākou i kahi hiʻohiʻona (e laʻa, kahi mea ʻike kaʻa). Hele mākou i kahi e lilo ai i mea maikaʻi loa - ʻo ia ka manawa e mālama ai i ka hopena. E kapa kākou iā car_detection_v1.pb. A laila hoʻomaʻamaʻa mākou i kekahi - car_detection_v2.pb. Ma hope o kekahi manawa, aʻo nui aʻe kā mākou mau hoa a i ʻole mākou iho, me ka hoʻohana ʻana i nā ʻano hana like ʻole. ʻO ka hopena, ua hoʻokumu ʻia kahi pūʻulu o nā mea artifact, pono e hōʻiliʻili ʻia ka ʻike e pili ana (akā, e hana mākou i kēia ma hope, no ka mea, ʻoi aku ka nui o kā mākou mea nui).
- OK ua pau i kēia manawa! He kumu hoʻohālike kā mākou! Hiki iā mākou ke hoʻomaka i ka hoʻomaʻamaʻa ʻana i ke kumu hoʻohālike e hiki mai ana, e hoʻomohala i kahi hoʻolālā e hoʻoponopono i kahi pilikia hou, a i ʻole hiki iā mākou ke hele i kahi kī? A ʻo wai ka mea e hoʻolaha?
ʻIke ʻana i nā pilikia
ʻO ka hana ʻana i kahi papahana a huahana paha ka hana a nā poʻe he nui. A i ka wā lōʻihi, haʻalele ka poʻe a hele mai, ʻoi aku ka nui o nā papahana, a ʻoi aku ka paʻakikī o nā papahana ponoʻī. Hoʻokahi ala a i ʻole, ʻo nā kūlana mai ka pōʻai i hōʻike ʻia ma luna nei (a ʻaʻole wale) i kekahi mau hui pū ʻana e hiki mai ana mai ka hoʻololi ʻana a i ka ʻike. ʻO kēia ka hopena o ka manawa pau ʻole, ka huikau, nā aʻalolo, ka ʻoluʻolu ʻole o ka mea kūʻai aku, a i ka hopena, nalowale ke kālā. ʻOiai mākou e hahai maʻamau i ka rake kahiko, ke manaʻoʻiʻo nei au ʻaʻohe mea makemake e hoʻihoʻi hou i kēia mau manawa.
No laila, ua hele mākou i hoʻokahi pōʻai hoʻomohala a ʻike mākou aia nā pilikia e pono e hoʻoholo. No ka hana ʻana i kēia pono ʻoe:
- mālama pono i nā hopena hana;
- e maʻalahi ke kaʻina hana o ka hoʻopili ʻana i nā limahana hou;
- e maʻalahi i ke kaʻina hana o ka hoʻolālā ʻana i kahi ʻano hoʻomohala;
- hoʻonohonoho i ke kaʻina hana hoʻohālike;
- loaʻa kahi ala kūpono e hōʻoia i nā hiʻohiʻona;
- ʻimi i kahi mea hana hoʻokele mokuʻāina;
- ʻimi i kahi ala e hoʻopuka ai i nā hiʻohiʻona i ka hana.
ʻIke ʻia he mea pono e hele mai me kahi workflow e hiki ai iā ʻoe ke maʻalahi a maʻalahi hoʻi e hoʻokele i kēia pōʻai ola? Kapa ʻia kēia hana ʻo MLOps
ʻO MLOps, a i ʻole DevOps no ke aʻo ʻana i ka mīkini, e ʻae i ka ʻepekema data a me nā hui IT e hui pū a hoʻonui i ka wikiwiki o ka hoʻomohala ʻana a me ka hoʻolālā ʻana ma o ka nānā ʻana, hōʻoia, a me ka hoʻokele ʻana no nā kumu aʻo mīkini.
Hiki paha iā 'oe He aha ka manaʻo o ka poʻe Google e pili ana i kēia mau mea? Mai ka ʻatikala ua maopopo ʻo MLOps he mea voluminous.
Eia hou ma kaʻu ʻatikala e wehewehe wau i kahi hapa o ke kaʻina hana. No ka hoʻokō, e hoʻohana wau i ka mea hana MLflow, no ka mea ... He papahana open-source kēia, koi ʻia kahi helu liʻiliʻi o ke code e hoʻopili a aia ka hoʻohui ʻana me nā frameworks ml kaulana. Hiki iā ʻoe ke ʻimi i ka Pūnaewele no nā mea hana ʻē aʻe, e like me Kubeflow, SageMaker, Trains, etc., a loaʻa paha kahi mea i kūpono i kāu mau pono.
"Kūkulu" nā MLO me ka hoʻohana ʻana i ka mea hana MLFlow
ʻO MLFlow kahi kahua kumu no ka hoʻokele ola ʻana o nā hiʻohiʻona ml ().
Aia ʻo MLflow i ʻehā mau ʻāpana:
- MLflow Tracking - uhi i nā pilikia o ka hoʻopaʻa ʻana i nā hopena a me nā ʻāpana i alakaʻi i kēia hopena;
- MLflow Project - hiki iā ʻoe ke hoʻopaʻa i ka code a hoʻopuka hou iā ia ma kekahi kahua;
- MLflow Models - kuleana no ka hoʻoili ʻana i nā hiʻohiʻona i ka hana;
- MLflow Registry - hiki iā ʻoe ke mālama i nā hiʻohiʻona a mālama i ko lākou mokuʻāina i loko o kahi waihona kikowaena.
Hoʻohana ʻo MLflow i ʻelua hui:
- ʻO ka hoʻomaka ʻana he pōʻai holoʻokoʻa o ka hoʻomaʻamaʻa, nā ʻāpana a me nā ana a mākou e makemake ai e hoʻopaʻa inoa;
- ʻO kahi hoʻokolohua he "kumuhana" e holo pū ana.
Hoʻokomo ʻia nā pae āpau o ka laʻana ma ka ʻōnaehana hana ʻo Ubuntu 18.04.
1. E hoʻolālā i ke kikowaena
I mea e hiki ai iā mākou ke hoʻokele maʻalahi i kā mākou papahana a loaʻa nā ʻike āpau e pono ai, e kau mākou i kahi kikowaena. He ʻelua mau mea nui ka server tracking MLflow:
- hale kūʻai hope - kuleana no ka mālama ʻana i ka ʻike e pili ana i nā hiʻohiʻona i hoʻopaʻa ʻia (kākoʻo iā 4 DBMSs: mysql, mssql, sqlite, a me postgresql);
- hale kūʻai artifact - kuleana no ka mālama ʻana i nā mea waiwai (kākoʻo i nā koho mālama 7: Amazon S3, Azure Blob Storage, Google Cloud Storage, FTP server, SFTP Server, NFS, HDFS).
I a hale kūʻai mea hana No ka maʻalahi, e lawe i kahi kikowaena sftp.
- hana i hui
$ sudo groupadd sftpg - hoʻohui i kahi mea hoʻohana a hoʻonoho i kahi ʻōlelo huna nona
$ sudo useradd -g sftpg mlflowsftp $ sudo passwd mlflowsftp - ka hoʻoponopono ʻana i ʻelua mau hoʻonohonoho komo
$ sudo mkdir -p /data/mlflowsftp/upload $ sudo chown -R root.sftpg /data/mlflowsftp $ sudo chown -R mlflowsftp.sftpg /data/mlflowsftp/upload - hoʻohui i kekahi mau laina i /etc/ssh/sshd_config
Match Group sftpg ChrootDirectory /data/%u ForceCommand internal-sftp - e hoʻomaka hou i ka lawelawe
$ sudo systemctl restart sshd
I a hale kūʻai hope E lawe kākou i ka postgresql.
$ sudo apt update
$ sudo apt-get install -y postgresql postgresql-contrib postgresql-server-dev-all
$ sudo apt install gcc
$ pip install psycopg2
$ sudo -u postgres -i
# Create new user: mlflow_user
[postgres@user_name~]$ createuser --interactive -P
Enter name of role to add: mlflow_user
Enter password for new role: mlflow
Enter it again: mlflow
Shall the new role be a superuser? (y/n) n
Shall the new role be allowed to create databases? (y/n) n
Shall the new role be allowed to create more new roles? (y/n) n
# Create database mlflow_bd owned by mlflow_user
$ createdb -O mlflow_user mlflow_dbNo ka hoʻomaka ʻana i ke kikowaena, pono ʻoe e hoʻokomo i nā pūʻulu python aʻe (Ke paipai nei au e hana i kahi kaiapuni virtual kaʻawale):
pip install mlflow
pip install pysftpE hoʻomaka i kā mākou kikowaena
$ mlflow server
--backend-store-uri postgresql://mlflow_user:mlflow@localhost/mlflow_db
--default-artifact-root sftp://mlflowsftp:mlflow@sftp_host/upload
--host server_host
--port server_port2. Hoʻohui i ka nānā ʻana
I mea e nalowale ai nā hopena o kā mākou hoʻomaʻamaʻa ʻana, e hoʻomaopopo nā hanauna e hiki mai ana o nā mea hoʻomohala i nā mea e hana nei, a no nā hoa kahiko a me ʻoe e hiki ke hoʻopaʻa mālie i ke kaʻina aʻo, pono mākou e hoʻohui i ka nānā ʻana. ʻO ka hahai ʻana, ʻo ia ka mālama ʻana i nā ʻāpana, metrics, artifacts a me nā ʻike hou aʻe e pili ana i ka hoʻomaka ʻana o ke aʻo ʻana, i kā mākou hihia, ma ke kikowaena.
Eia kekahi laʻana, ua hana wau i kahi liʻiliʻi ma Keras no ka hoʻokaʻawale ʻana i nā mea a pau i loko . No ka hoʻohui ʻana i ka nānā ʻana, hana wau i kahi faila mlflow_training.py.
Eia nā laina kahi e hana ai nā mea hoihoi loa:
def run(self, epochs, lr, experiment_name):
# getting the id of the experiment, creating an experiment in its absence
remote_experiment_id = self.remote_server.get_experiment_id(name=experiment_name)
# creating a "run" and getting its id
remote_run_id = self.remote_server.get_run_id(remote_experiment_id)
# indicate that we want to save the results on a remote server
mlflow.set_tracking_uri(self.tracking_uri)
mlflow.set_experiment(experiment_name)
with mlflow.start_run(run_id=remote_run_id, nested=False):
mlflow.keras.autolog()
self.train_pipeline.train(lr=lr, epochs=epochs)
try:
self.log_tags_and_params(remote_run_id)
except mlflow.exceptions.RestException as e:
print(e)Eia ʻo self.remote_server kahi wahī liʻiliʻi ma luna o nā ʻano mlflow.tracking. MlflowClient (Ua hana au ia no ka maʻalahi), me ke kōkua o iaʻu e hana i kahi hoʻokolohua a holo ma luna o ke kikowaena. A laila, hōʻike wau i kahi e hoʻohui ʻia ai nā hopena hoʻomaka (mlflow.set_tracking_uri(self.tracking_uri)). Hiki iaʻu ke hoʻopaʻa inoa ʻakomi mlflow.keras.autolog(). Ke kākoʻo nei ʻo MLflow Tracking i ka hoʻopaʻa inoa ʻana no TensorFlow, Keras, Gluon XGBoost, LightGBM, Spark. Inā ʻaʻole ʻoe i ʻike i kāu framework a i ʻole ka waihona, a laila hiki iā ʻoe ke hoʻopaʻa inoa i nā manawa a pau. Ke hoʻomaka nei mākou i ke aʻo ʻana. E hoʻopaʻa inoa i nā hōʻailona a me nā ʻāpana hoʻokomo ma ke kikowaena mamao.
Loaʻa i nā laina ʻelua a ʻo ʻoe, e like me nā mea ʻē aʻe, i ka ʻike e pili ana i nā neʻe āpau. Anu?
3. Hoʻolālā mākou i ka papahana
I kēia manawa e maʻalahi ka hoʻomaka ʻana i ka papahana. No ka hana ʻana i kēia, hoʻohui i ka faila MLproject a me conda.yaml i ke kumu o ka papahana.
MLproject
name: flow_segmentation
conda_env: conda.yaml
entry_points:
main:
parameters:
categories: {help: 'list of categories from coco dataset'}
epochs: {type: int, help: 'number of epochs in training'}
lr: {type: float, default: 0.001, help: 'learning rate'}
batch_size: {type: int, default: 8}
model_name: {type: str, default: 'Unet', help: 'Unet, PSPNet, Linknet, FPN'}
backbone_name: {type: str, default: 'resnet18', help: 'exampe resnet18, resnet50, mobilenetv2 ...'}
tracking_uri: {type: str, help: 'the server address'}
experiment_name: {type: str, default: 'My_experiment', help: 'remote and local experiment name'}
command: "python mlflow_training.py
--epochs={epochs}
--categories={categories}
--lr={lr}
--tracking_uri={tracking_uri}
--model_name={model_name}
--backbone_name={backbone_name}
--batch_size={batch_size}
--experiment_name={experiment_name}"He mau waiwai ko MLflow Project:
- Inoa - ka inoa o kāu papahana;
- Kaiapuni - i koʻu hihia, hōʻike ʻo conda_env ua hoʻohana ʻia ʻo Anaconda e holo a aia ka wehewehe hilinaʻi i ka faila conda.yaml;
- Nā Koho Komo - hōʻike i nā faila a me nā ʻāpana e hiki ai iā mākou ke holo (e hoʻopaʻa ʻia nā ʻāpana āpau i ka wā e hoʻomaka ai ke aʻo ʻana)
conda.yaml
name: flow_segmentation
channels:
- defaults
- anaconda
dependencies:
- python==3.7
- pip:
- mlflow==1.8.0
- pysftp==0.2.9
- Cython==0.29.19
- numpy==1.18.4
- pycocotools==2.0.0
- requests==2.23.0
- matplotlib==3.2.1
- segmentation-models==1.0.1
- Keras==2.3.1
- imgaug==0.4.0
- tqdm==4.46.0
- tensorflow-gpu==1.14.0Hiki iā ʻoe ke hoʻohana i ka docker e like me kāu kaiapuni runtime, no ka ʻike hou aku e ʻoluʻolu e nānā .
4. E hoʻomaka kākou i ke aʻo ʻana
Hoʻopili mākou i ka papahana a hele i ka papa kuhikuhi papahana:
git clone https://github.com/simbakot/mlflow_example.git
cd mlflow_example/No ka holo ʻana, pono ʻoe e hoʻokomo i nā hale waihona puke
pip install mlflow
pip install pysftpNo ka mea i ka laʻana aʻu e hoʻohana ai i conda_env, pono e hoʻokomo ʻia ʻo Anaconda ma kāu kamepiula (akā hiki iā ʻoe ke hoʻopuni i kēia ma ka hoʻokomo ʻana i nā pūʻolo pono āpau iā ʻoe iho a pāʻani me nā ʻāpana hoʻomaka).
Ua pau nā hana hoʻomākaukau a hiki iā mākou ke hoʻomaka i ka hoʻomaʻamaʻa. Mai ke kumu o ka papahana:
$ mlflow run -P epochs=10 -P categories=cat,dog -P tracking_uri=http://server_host:server_port .Ma hope o ke komo ʻana i ke kauoha, e hana ʻia kahi conda environment a hoʻomaka ke aʻo ʻana.
I ka laʻana ma luna, ua hala au i ka helu o nā manawa no ka hoʻomaʻamaʻa ʻana, nā ʻāpana a mākou e makemake ai e hoʻokaʻawale (hiki iā ʻoe ke ʻike i ka papa inoa piha ) a me ka helu wahi o kā mākou kikowaena mamao.
Hiki ke loaʻa kahi papa inoa piha o nā ʻāpana i ka faila MLproject.
5. E loiloi i nā hopena aʻo
Ma hope o ka pau ʻana o ke aʻo ʻana, hiki iā mākou ke hele i ka polokalamu kele i ka helu o kā mākou kikowaena
Maanei mākou e ʻike ai i ka papa inoa o nā hoʻokolohua āpau (hema hema), a me ka ʻike e pili ana i nā holo (waena). Hiki iā mākou ke ʻike i ka ʻike kikoʻī hou aku (nā ʻāpana, metric, artifacts a me kekahi ʻike hou aʻe) no kēlā me kēia hoʻolaha.
No kēlā me kēia metric hiki iā mākou ke nānā i ka mōʻaukala o nā loli
ʻO kēlā mau mea. I kēia manawa, hiki iā mākou ke nānā i nā hopena ma ke ʻano "manual", a hiki iā ʻoe ke hoʻonohonoho i ka hōʻoia maʻalahi me ka hoʻohana ʻana i ka MLflow API.
6. E hoʻopaʻa inoa i ke kumu hoʻohālike
Ma hope o ka nānā ʻana i kā mākou kumu hoʻohālike a hoʻoholo ua mākaukau no ke kaua, hoʻomaka mākou e hoʻopaʻa inoa, no kēia koho mākou i ka hoʻomaka ʻana e pono ai mākou (e like me ka hōʻike ʻana ma ka paukū mua) a iho i lalo.
Ma hope o ka hāʻawi ʻana i kā mākou kumu hoʻohālike i ka inoa, loaʻa iā ia kahi mana. Inā mālama ʻoe i kekahi kumu hoʻohālike me ka inoa hoʻokahi, e hoʻonui ʻia ka mana.
No kēlā me kēia hiʻohiʻona, hiki iā mākou ke hoʻohui i kahi wehewehe a koho i kekahi o nā mokuʻāina ʻekolu (Staging, Production, Archived); a laila, me ka hoʻohana ʻana i ka API, hiki iā mākou ke komo i kēia mau mokuʻāina, a me ka hoʻololi ʻana, hāʻawi i ka maʻalahi.
Loaʻa iā mākou ke komo maʻalahi i nā hiʻohiʻona āpau
a me kā lākou mau mana
E like me ka paukū mua, hiki ke hana i nā hana a pau me ka API.
7. E hoʻolālā i ke kumu hoʻohālike
I kēia pae, ua loaʻa iā mākou kahi hoʻohālike i aʻo ʻia (keras). ʻO kahi hiʻohiʻona pehea e hiki ai iā ʻoe ke hoʻohana:
class SegmentationModel:
def __init__(self, tracking_uri, model_name):
self.registry = RemoteRegistry(tracking_uri=tracking_uri)
self.model_name = model_name
self.model = self.build_model(model_name)
def get_latest_model(self, model_name):
registered_models = self.registry.get_registered_model(model_name)
last_model = self.registry.get_last_model(registered_models)
local_path = self.registry.download_artifact(last_model.run_id, 'model', './')
return local_path
def build_model(self, model_name):
local_path = self.get_latest_model(model_name)
return mlflow.keras.load_model(local_path)
def predict(self, image):
image = self.preprocess(image)
result = self.model.predict(image)
return self.postprocess(result)
def preprocess(self, image):
image = cv2.resize(image, (256, 256))
image = image / 255.
image = np.expand_dims(image, 0)
return image
def postprocess(self, result):
return resultEia hou self.registry kahi wili liʻiliʻi ma luna o mlflow.tracking.MlflowClient, no ka maʻalahi. ʻO ke kumu ke komo wau i kahi kikowaena mamao a ʻimi i kahi kumu hoʻohālike ma laila me ka inoa i kuhikuhi ʻia, a me ka mana hana hou loa. Ma hope aʻe, hoʻoiho au i ka artifact kūloko i ka waihona ./model a kūkulu i ke kumu hoʻohālike mai kēia papa kuhikuhi mlflow.keras.load_model(local_path). I kēia manawa hiki iā mākou ke hoʻohana i kā mākou kumu hoʻohālike. Hiki i nā mea hoʻomohala CV (ML) ke hoʻomaikaʻi maʻalahi i ke kumu hoʻohālike a hoʻopuka i nā mana hou.
I ka hopena
Ua hōʻike au i kahi ʻōnaehana e hiki ai:
- e mālama i ka ʻike e pili ana i nā hiʻohiʻona ML, ka holomua aʻo ʻana a me nā hopena;
- hoʻolālā wikiwiki i kahi hoʻolālā hoʻomohala;
- nānā a nānā i ka holomua o ka hana ma nā kumu hoʻohālike;
- He mea maʻalahi ke hoʻololi a mālama i ke kūlana o nā hiʻohiʻona;
- He mea maʻalahi ke kau ʻana i nā ʻano hoʻohālike.
ʻO kēia hiʻohiʻona he mea pāʻani a lawelawe ma ke ʻano he kumu hoʻomaka no ke kūkulu ʻana i kāu ʻōnaehana ponoʻī, hiki ke komo i ka automation o ka loiloi o nā hopena a me ka hoʻopaʻa inoa ʻana o nā kumu hoʻohālike (nā helu 5 a me 6, kēlā me kēia) a i ʻole ʻoe e hoʻohui i ka versioning o nā datasets, a i ʻole kekahi mea ʻē aʻe ? ʻO ka manaʻo aʻu e hoʻāʻo nei e hana, pono ʻoe i nā MLOps ma ke ʻano holoʻokoʻa, ʻo MLflow kahi ala e hoʻopau ai.
E kākau i nā pilikia āu i hālāwai ai ʻaʻole au i hōʻike?
He aha kāu e hoʻohui ai i ka ʻōnaehana e hoʻokō ai i kāu mau pono?
He aha nā mea hana a me nā ala āu e hoʻohana ai e hoʻoponopono i nā pilikia āpau a i ʻole kekahi hapa?
PS E waiho wau i ʻelua mau loulou:
papahana github -
MLflow -
ʻO kaʻu leka uila hana no nā nīnau - [pale ʻia ka leka uila]
Hoʻokipa pinepine kā mākou hui i nā hanana like ʻole no nā loea IT, no ka laʻana: ma Iulai 8 ma 19:00 i ka manawa Moscow e loaʻa kahi hui CV ma kahi ʻano pūnaewele, inā makemake ʻoe, hiki iā ʻoe ke komo, kākau inoa. .
Source: www.habr.com