Dumelang bohle! Ke moqapi oa CV ho CROC. E se e le lilemo tse tharo re ntse re kenya tšebetsong merero ea CV. Nakong ena, re entse sohle ho tloha ho beha bakhanni leihlo ho ea ho ho netefatsa hore ha ba noe, ha ba tsube, kapa ha ba bue ka mohala ha ba ntse ba khanna, ho isa mahlo a bona tseleng ’me ha ba lore kapa ba tjamela marung; ho tsebahatsa ba khannang litseleng tse inehetseng le ho nka libaka tse ngata tsa ho paka makoloi; ho etsa bonnete ba hore basebetsi ba roala lihelmete, liatlana, joalo-joalo; ho hlwaya basebetsi ba batlang ho kena setsing; ho bala tsohle tse ka khonehang.
Ke fumana eng?
Ha re ntse re kenya tšebetsong merero, re kopane le mathata a mangata le matetetso, 'me u tseba mathata a mang kapa u tla kopana le ona nakong e tlang.
A re etsiseng boemo
Ha re nahane re kene k'hamphaning e nyane e bitsoang "N," eo mosebetsi oa eona o kenyelletsang ho ithuta ka mochini. Re sebetsa morerong oa ML (DL, CV), ebe ka lebaka le itseng re fetohela mosebetsing o mong, re khefutse, ebe re khutlela ho neural network ea rona kapa ea motho e mong.
- Nako ea 'nete e fihla, ho hlokahala hore ka tsela e itseng u hopole hore na u tlohetse hokae, ke li-hyperparameter life tseo u li lekileng' me, ho bohlokoa ka ho fetisisa, ke liphello life tseo ba li lebisitseng. Ho ka ba le mefuta e mengata e fapaneng ea hore na motho o bolokile data joang bakeng sa lits'ebetso tsohle tsa bona: lihloohong tsa bona, lifaeleng tsa config, bukeng ea lintlha, kapa sebakeng sa mosebetsi se thehiloeng marung. Ke bone nyeoe moo li-hyperparameter li neng li bolokiloe e le mela e boletsoeng ka har'a khoutu - mohopolo oa 'nete. Joale ak'u nahane u se u sa khutlele morerong oa hau, empa morerong oa motho ea tlohileng k'hamphani 'me a rua khoutu le mohlala o bitsoang model_1.pb. Ho u fa setšoantšo se felletseng le ho fetisa bohloko, ha re nahane hore o setsebi sa maemo a tlase.
- Ha re tsoeleng pele. Ho tsamaisa khoutu, rona le bohle ba tla sebetsa le eona re hloka ho theha tikoloho. Hangata ho etsahala hore, ka mabaka a itseng, ha rea ka ra siuoa le sena e le lefa. Hona hape e ka ba mosebetsi o sa reng letho. Ha u batle ho senya nako ka mohato ona, na ha ho joalo?
- Re koetlisa mohlala (mohlala, mochine oa koloi). Re fihla moo ho leng molemo haholo—ke nako ea ho boloka sephetho. Ha re e bitse car_detection_v1.pb. Ebe re koetlisa e 'ngoe-car_detection_v2.pb. Ka mor'a nako e itseng, basebetsi-'moho le rona kapa re koetlisa ka ho eketsehileng, ho sebelisa mehaho e fapaneng. Sephetho ke sehlopha sa lintho tsa khale, tlhahisoleseling e lokelang ho bokelloa ka hloko (empa re tla etsa joalo hamorao; hajoale, re na le litaba tse ngata tse hatellang tseo re lokelang ho li ela hloko).
- Ho joalo, ho joalo! Re na le mohlala! Na re ka qala ho koetlisa mohlala o latelang, ho theha moralo bakeng sa bothata bo bocha, kapa na re lokela ho ea noa tee? Mme ke mang ya tla e tsamaisa?
Ho khetholla mathata
Ho sebetsa morerong kapa sehlahisoa ke mosebetsi oa batho ba bangata. 'Me ha nako e ntse e ea, batho ba tla le ho tsamaea, palo ea merero e eketseha,' me merero ka boeona e fetoha e rarahaneng. Ka tsela e 'ngoe kapa e' ngoe, maemo ho tloha potolohong e hlalositsoeng ka holimo (le a mang) ka motsoako o fapaneng a tla etsahala ho tloha ho pheta-pheta ho ea ho pheta-pheta. Sena sohle se fella ka tšenyo ea nako, pherekano, khatello ea maikutlo, mohlomong ho se khotsofale ha bareki, 'me qetellong, ho lahleheloa ke chelete. Le hoja bohle re atisa ho etsa liphoso tse tšoanang, ke lumela hore ha ho motho ea batlang ho pheta maemo ana khafetsa.
Ka hona, re se re qetile cycle e le ‘ngoe ea ntlafatso ‘me rea bona hore ho na le litaba tse lokelang ho lokisoa. Ho etsa sena, re hloka ho:
- polokelo e bonolo ea liphetho tsa mosebetsi;
- etsa hore mokhoa oa ho kenyelletsa basebetsi ba bacha o be bonolo;
- nolofatsa mokhoa oa ho tsamaisa tikoloho ea ntlafatso;
- theha mokhoa oa ho fetolela mohlala;
- ho ba le mokhoa o bonolo oa ho netefatsa mehlala;
- fumana sesebelisoa sa tsamaiso ea boemo ba mohlala;
- Fumana mokhoa oa ho fana ka mehlala ho tlhahiso.
Kamoo ho bonahalang kateng, re hloka ho hlahisa mokhoa oa ho sebetsa o tla re lumella ho laola mokhoa ona oa bophelo habonolo le ka mokhoa o bonolo? Mokhoa ona o bitsoa MLOps.
MLOps, kapa DevOps bakeng sa ho ithuta ka mochini, e thusa lihlopha tsa bo-ramahlale ba data le litsebi tsa IT ho sebelisana le ho potlakisa lebelo la nts'etsopele ea mohlala le phepelo ka ho fana ka tlhokomelo, netefatso, le taolo bakeng sa mefuta ea ho ithuta ea mochini.
O ka khona Bashemane ba Google ba nahana eng ka see sohle? Ho hlakile ho tsoa sengolong hore MLOps ke ntho e kholo haholo.
Sehloohong se latelang, ke tla hlalosa karolo feela ea ts'ebetso. Ke tla sebelisa MLflow bakeng sa ts'ebetsong, kaha ke mohloli o bulehileng, o hloka khouto e nyane, 'me e hokahana le meralo e tsebahalang ea ML. U ka batla marang-rang bakeng sa lisebelisoa tse ling, tse kang Kubeflow, SageMaker, Trains, le tse ling, 'me mohlomong u fumane e lumellanang le litlhoko tsa hau hantle.
Ho aha li-MLOps ka MLFlow
MLFlow ke sethala sa mohloli o bulehileng oa ho laola bophelo ba mefuta ea ml ().
MLflow e na le likarolo tse 'ne:
- MLflow Tracking - e sebetsana le taba ea liphetho tsa ho rekota le litekanyo tse lebisitseng sephethong sena;
- MLflow Project - e u lumella ho paka khoutu le ho e hlahisa sethaleng sefe kapa sefe;
- MLflow Models - e ikarabellang bakeng sa ho tsamaisa mehlala ho tlhahiso;
- MLflow Registry - e u lumella ho boloka mehlala le ho laola boemo ba bona sebakeng sa polokelo se bohareng.
MLflow e sebetsa ho likarolo tse peli:
- thakholoa ke potoloho e felletseng ea ho ithuta, liparamente le methapo eo re batlang ho ingolisa ka eona;
- Teko ke "theme" e kopanyang tse qalang.
Mehato eohle ea mohlala e kenngoa ts'ebetsong tsamaisong ea ts'ebetso Ubuntu 18.04.
1. Kenya seva
Ho laola projeke ea rona habonolo le ho fumana lintlha tsohle tse hlokahalang, re tla tsamaisa seva. Seva ea ho latela MLflow e na le likarolo tse peli tsa mantlha:
- lebenkele la backend - le ikarabellang bakeng sa ho boloka tlhahisoleseling mabapi le mehlala e ngolisitsoeng (e tšehetsa 4 DBMS: mysql, mssql, sqlite, le postgresql);
- polokelo ea lintho tsa khale - e ikarabellang bakeng sa ho boloka lintho tsa khale (e tšehetsa likhetho tsa 7 tsa polokelo: Amazon S3, Azure Blob Storage, Google Cloud Storage, FTP seva, SFTP Server, NFS, HDFS).
E le lebenkele la lintho tsa khale Ho nolofatsa, ha re nkeng seva sa sftp.
- theha sehlopha
$ sudo groupadd sftpg - eketsa mosebedisi ebe u mo seta phasewete
$ sudo useradd -g sftpg mlflowsftp $ sudo passwd mlflowsftp - Re ntse re lokisa litlhophiso tse 'maloa tsa phihlello.
$ sudo mkdir -p /data/mlflowsftp/upload $ sudo chown -R root.sftpg /data/mlflowsftp $ sudo chown -R mlflowsftp.sftpg /data/mlflowsftp/upload - eketsa mela e 'maloa ho /etc/ssh/sshd_config
Match Group sftpg ChrootDirectory /data/%u ForceCommand internal-sftp - qala tshebeletso hape
$ sudo systemctl restart sshd
E le lebenkele la mokokotlo Ha re nke postgresql.
$ sudo apt update
$ sudo apt-get install -y postgresql postgresql-contrib postgresql-server-dev-all
$ sudo apt install gcc
$ pip install psycopg2
$ sudo -u postgres -i
# Create new user: mlflow_user
[postgres@user_name~]$ createuser --interactive -P
Enter name of role to add: mlflow_user
Enter password for new role: mlflow
Enter it again: mlflow
Shall the new role be a superuser? (y/n) n
Shall the new role be allowed to create databases? (y/n) n
Shall the new role be allowed to create more new roles? (y/n) n
# Create database mlflow_bd owned by mlflow_user
$ createdb -O mlflow_user mlflow_dbHo tsamaisa seva, o hloka ho kenya liphutheloana tse latelang tsa Python (ke khothaletsa ho theha tikoloho e arohaneng):
pip install mlflow
pip install pysftpHa re hlahise seva ea rona
$ mlflow server
--backend-store-uri postgresql://mlflow_user:mlflow@localhost/mlflow_db
--default-artifact-root sftp://mlflowsftp:mlflow@sftp_host/upload
--host server_host
--port server_port2. Eketsa ho latedisa
Ho etsa bonnete ba hore liphetho tsa rona tsa koetliso ha li lahlehe, hore meloko e tlang ea bahlahisi ba utloisise se etsahetseng, le hore bahlahisi ba phahameng le uena u ka sekaseka ts'ebetso ea koetliso ka mokhoa o khutsitseng, re hloka ho eketsa tlhaiso-leseling. Ho latela ho bolela ho boloka liparamente, metrics, lintho tsa khale, le tlhahisoleseling efe kapa efe e eketsehileng mabapi le thupelo ea koetliso - molemong oa rona, ho seva.
Ka mohlala, ke entse e nyenyane ho Keras bakeng sa ho arola ntho e 'ngoe le e' ngoe e teng Ho kenyelletsa tracking, ke thehile faele ea mlflow_training.py.
Mona ke mela eo lintho tse khahlang haholo li etsahalang:
def run(self, epochs, lr, experiment_name):
# getting the id of the experiment, creating an experiment in its absence
remote_experiment_id = self.remote_server.get_experiment_id(name=experiment_name)
# creating a "run" and getting its id
remote_run_id = self.remote_server.get_run_id(remote_experiment_id)
# indicate that we want to save the results on a remote server
mlflow.set_tracking_uri(self.tracking_uri)
mlflow.set_experiment(experiment_name)
with mlflow.start_run(run_id=remote_run_id, nested=False):
mlflow.keras.autolog()
self.train_pipeline.train(lr=lr, epochs=epochs)
try:
self.log_tags_and_params(remote_run_id)
except mlflow.exceptions.RestException as e:
print(e)Mona, self.remote_server ke wrapper e nyenyane e pota-potileng mekhoa ea mlflow.tracking.mlflowClient (Ke e bōpile bakeng sa boiketlo), eo ke e sebelisang ho etsa teko le ho e tsamaisa ho seva. Ka mor'a moo, ke hlalosa hore na liphetho tsa ho matha li lokela ho romeloa hokae (mlflow.set_tracking_uri(self.tracking_uri)). Ke nolofalletsa ho rema lifate ka mlflow.keras.autolog(). MLflow Tracking hajoale e ts'ehetsa ho rema lifate ka boiketsetso bakeng sa TensorFlow, Keras, Gluon XGBoost, LightGBM, le Spark. Haeba u sa fumane moralo kapa laeborari ea hau, u ka kena ka mokhoa o hlakileng kamehla. Ha re qaleng koetliso. Ngolisa li-tag le li-parameter tsa ho kenya ho seva e hole.
Ke mela e 'maloa feela, 'me uena, joalo ka batho ba bang, le khona ho fumana leseli mabapi le lits'ebetso tsohle. Ho pholile?
3. Re theha morero ka molao
Joale ha re etse hore ho tsamaisa morero ho be bonolo kamoo ho ka khonehang. Ho etsa sena, kenya faele ea MLproject le conda.yaml motso oa morero.
MLproject
name: flow_segmentation
conda_env: conda.yaml
entry_points:
main:
parameters:
categories: {help: 'list of categories from coco dataset'}
epochs: {type: int, help: 'number of epochs in training'}
lr: {type: float, default: 0.001, help: 'learning rate'}
batch_size: {type: int, default: 8}
model_name: {type: str, default: 'Unet', help: 'Unet, PSPNet, Linknet, FPN'}
backbone_name: {type: str, default: 'resnet18', help: 'exampe resnet18, resnet50, mobilenetv2 ...'}
tracking_uri: {type: str, help: 'the server address'}
experiment_name: {type: str, default: 'My_experiment', help: 'remote and local experiment name'}
command: "python mlflow_training.py
--epochs={epochs}
--categories={categories}
--lr={lr}
--tracking_uri={tracking_uri}
--model_name={model_name}
--backbone_name={backbone_name}
--batch_size={batch_size}
--experiment_name={experiment_name}"MLflow Project e na le thepa e 'maloa:
- Lebitso - lebitso la morero oa hau;
- Tikoloho - tabeng ea ka, conda_env e bontša hore Anaconda e sebelisoa bakeng sa ho qala 'me tlhaloso ea ho itšetleha e ka faeleng ea conda.yaml;
- Lintlha tsa ho Kena - e hlalosa hore na ke lifaele life le hore na re ka tsamaisa liparamente life (mekhahlelo eohle e kenngoa ka bo eona ha e qala koetliso)
conda.yaml
name: flow_segmentation
channels:
- defaults
- anaconda
dependencies:
- python==3.7
- pip:
- mlflow==1.8.0
- pysftp==0.2.9
- Cython==0.29.19
- numpy==1.18.4
- pycocotools==2.0.0
- requests==2.23.0
- matplotlib==3.2.1
- segmentation-models==1.0.1
- Keras==2.3.1
- imgaug==0.4.0
- tqdm==4.46.0
- tensorflow-gpu==1.14.0U ka sebelisa docker e le sebaka sa nako ea ho sebetsa, bakeng sa lintlha tse ling ka kopo sheba ho .
4. Qala koetliso
Kopanya morero 'me u ee bukeng ea morero:
git clone https://github.com/simbakot/mlflow_example.git
cd mlflow_example/Ho sebetsa o hloka ho kenya lilaeborari
pip install mlflow
pip install pysftpKaha ke sebelisa conda_env ka mohlala, Anaconda e tlameha ho kenngoa khomphuteng ea hau (empa u ka khona ho pota-pota sena ka ho kenya liphutheloana tsohle tse hlokahalang le ho bapala ka mekhahlelo ea ho qala).
Mehato eohle ea boitokisetso e felile, 'me re ka qala ho thakhola lithupelo. Ho tsoa ho motso oa projeke:
$ mlflow run -P epochs=10 -P categories=cat,dog -P tracking_uri=http://server_host:server_port .Kamora ho kenya taelo, tikoloho ea conda e tla etsoa ka bo eona mme koetliso e tla qala.
Mohlala o ka holimo, ke fetisitse palo ea nako ea ho koetlisa, lihlopha tseo re batlang ho li arola (lenane le feletseng le ka fumanoa mona). ) le aterese ea seva ea rona e hole.
Lethathamo le felletseng la liparamente tse ka bang teng le ka fumanoa faeleng ea MLproject.
5. Lekola liphello tsa ho ithuta
Ka mor'a ho qeta koetliso, re ka ea atereseng ea seva sa rona ho sebatli
Mona re bona lethathamo la liteko tsohle (holimo ka ho le letšehali) le ho tsamaisa tlhahisoleseding (bohareng). Re ka bona lintlha tse qaqileng haholoanyane (litekanyetso, metrics, li-artifacts, le lintlha tse ling tse ling) bakeng sa papali ka 'ngoe.
Bakeng sa metric ka 'ngoe re ka sheba nalane ea liphetoho
Sena se bolela hore hajoale re ka sekaseka liphetho ka letsoho, empa hape o ka theha netefatso ea othomathiki o sebelisa MLflow API.
6. Ngolisa mohlala
Ka mor'a hore re hlahlobisise mohlala oa rona mme re etse qeto ea hore o se o loketse ntoa, re tsoela pele ho o ngolisa. Ho etsa sena, re khetha tlhahiso eo re e hlokang (joalokaha ho bontšitsoe serapeng se fetileng) ebe re theoha.
Ka mor'a hore re rehe mohlala oa rona, e fumana phetolelo. Ha re boloka mohlala o mong o nang le lebitso le tšoanang, phetolelo e tla eketseha ka bo eona.
Bakeng sa mohlala o mong le o mong, re ka eketsa tlhaloso ebe re khetha e 'ngoe ea linaha tse tharo (Staging, Production, Archived). Joale re ka fihlella linaha tsena re sebelisa API, eo, hammoho le phetolelo, e fanang ka phetoho e eketsehileng.
Re boetse re na le phihlello e bonolo ho mefuta eohle
le liphetolelo tsa bona
Joalo ka ntlheng e fetileng, lits'ebetso tsohle li ka etsoa ho sebelisoa API.
7. Kenya mohlala
Nakong ena, re se re ntse re e-na le mohlala o koetlisitsoeng (Keras). Mona ke mohlala oa kamoo e ka sebelisoang kateng:
class SegmentationModel:
def __init__(self, tracking_uri, model_name):
self.registry = RemoteRegistry(tracking_uri=tracking_uri)
self.model_name = model_name
self.model = self.build_model(model_name)
def get_latest_model(self, model_name):
registered_models = self.registry.get_registered_model(model_name)
last_model = self.registry.get_last_model(registered_models)
local_path = self.registry.download_artifact(last_model.run_id, 'model', './')
return local_path
def build_model(self, model_name):
local_path = self.get_latest_model(model_name)
return mlflow.keras.load_model(local_path)
def predict(self, image):
image = self.preprocess(image)
result = self.model.predict(image)
return self.postprocess(result)
def preprocess(self, image):
image = cv2.resize(image, (256, 256))
image = image / 255.
image = np.expand_dims(image, 0)
return image
def postprocess(self, result):
return resultMona, self.registry e boetse e le sephutheloana se senyenyane se pota-potileng mlflow.tracking.MlflowClient, bakeng sa boiketlo. Maikutlo ke hore ke fihlella seva e hole mme ke batle mohlala o nang le lebitso le boletsoeng, haholo mofuta oa morao-rao oa tlhahiso. Ebe, ke khoasolla lintho tse entsoeng ka matsoho sebakeng sa heno ho ./model foldareng ebe ke aha mohlala ho tsoa bukeng ena: mlflow.keras.load_model(local_path). Hona joale re ka sebelisa mohlala oa rona. Baetsi ba CV (ML) ba ka ntlafatsa mohlala ka bolokolohi le ho phatlalatsa mefuta e mecha.
Qetellong
Ke hlahisitse sistimi e lumellang:
- boloka boitsebiso bo bohareng mabapi le mefuta ea ML, tsoelo-pele ea koetliso le liphetho;
- ho tsamaisa tikoloho ea ntlafatso kapele;
- hlokomela le ho sekaseka tsoelopele ea mosebetsi oa mehlala;
- phetolelo e bonolo le taolo ea boemo ba mohlala;
- Ho bonolo ho tsamaisa mehlala e hlahisoang.
Mohlala ona ke ntho ea ho bapala 'me e sebetsa e le qalo ea ho haha sistimi ea hau, e ka kenyelletsang tlhahlobo ea sephetho sa othomathike le ngoliso ea mohlala (lintho 5 le 6, ka ho latellana), kapa u ka eketsa mofuta oa dataset, kapa mohlomong ho hong ka ho felletseng? Ke ne ke leka ho fetisa mohopolo oa hore o hloka MLOps ka kakaretso, mme MLflow ke mokhoa oa ho fihlela sepheo.
Ke kopa o mpolelle hore na ke mathata afe ao u kopaneng le 'ona ao ke sa kang ka a bua.
U ka eketsa eng ho sistimi ho fihlela litlhoko tsa hau?
Ke lisebelisoa life le mekhoa eo u e sebelisang ho rarolla mathata kaofela kapa a mang?
P.S. Ke tla siea lihokelo tse 'maloa:
Morero oa GitHub -
MLflow -
Lengolo-tsoibila la ka la lipotso ke ikryakin@croc.ru
Nako le nako re tšoara liketsahalo tse fapaneng bakeng sa litsebi tsa IT k'hamphaning ea rona, mohlala: ka la 8 Phupu ka 19:00 PM nako ea Moscow, ho tla ba le kopano ea inthaneteng ea CV. Haeba u thahasella, u ka nka karolo; ngoliso ea hlokahala. .
Source: www.habr.com
