Lumelang bohle! Ke moqapi oa CV ho CROC. E se e le lilemo tse 3 re ntse re kenya tšebetsong merero lefapheng la CV. Ka nako ena, re ne re etsa lintho tse ngata, mohlala: re ne re beha bakhanni leihlo hore ha ba ntse ba khanna ha ba noe, ba sa tsube, ba sa bue le fono, ba sheba tsela, eseng litoro kapa maru. ; Re rekotile batšehetsi ba khanna litseleng tse inehetseng le ho lula libakeng tse 'maloa tsa ho paka; a netefatsa hore basebetsi ba roala lihelmete, liatlana, joalo-joalo; hlwaya mosebeletsi ya batlang ho kena setsing; Re ile ra bala sohle seo re ka se khonang.
Tsena tsohle ke li etsetsa eng?
Ha re ntse re kenya ts'ebetsong merero, re otla likhahla, mathata a mangata, mathata a mang ao u a tsebang kapa ao u tla a tseba nakong e tlang.
A re etsiseng boemo
A re ke re nahane hore re fumane mosebetsi k'hamphaning e nyenyane "N", eo mesebetsi ea eona e amanang le ML. Re sebetsa morerong oa ML (DL, CV), ebe ka lebaka le itseng re fetohela mosebetsing o mong, ka kakaretso re phomola, ebe re khutlela ho neuron ea rona kapa ea motho e mong.
- Nako ea 'nete e tla, ho hlokahala hore ka tsela e itseng u hopole hore na u emisitse hokae, ke li-hyperparameter life tseo u li lekileng' me, ho bohlokoa ka ho fetisisa, ke liphello life tseo ba li lebisitseng. Ho ka ba le likhetho tse ngata bakeng sa hore na ke mang ea bolokileng tlhahisoleseling ho tsohle tse hlahisoang: hloohong, configs, notepad, sebakeng sa ho sebetsa marung. Ke ile ka bona khetho ha li-hyperparameter li bolokiloe e le mela e hlalositsoeng khoutu, ka kakaretso, sefofane se majabajaba. Joale nahana hore ha ua khutlela morerong oa hau, empa morerong oa motho ea tlohileng k'hamphani 'me u futsitse khoutu le mohlala o bitsoang model_1.pb. Ho tlatsa setšoantšo le ho fetisa bohloko bohle, a re nke hore le uena u setsebi sa ho qala.
- Tsoela pele. Ho tsamaisa khoutu, rona le bohle ba tla sebetsa le eona re hloka ho theha tikoloho. Hangata ho etsahala hore ka mabaka a itseng ha baa ka ba mo tlohela e le lefa la rōna. Sena se ka boela sa fetoha mosebetsi oo e seng oa bohlokoa. Ha u batle ho senya nako ka mohato ona, na ha ho joalo?
- Re koetlisa mohlala (mohlala, mochine oa koloi). Re fihla moo e bang hantle haholo - ke nako ea ho boloka sephetho. Ha re e bitse car_detection_v1.pb. Ebe re koetlisa e 'ngoe - car_detection_v2.pb. Nako e itseng hamorao, basebetsi-'moho le rona kapa rona re ruta ka ho eketsehileng, re sebelisa mehaho e fapaneng. Ka lebaka leo, ho thehoa lintho tse ngata tse entsoeng ka matsoho, tlhahisoleseding eo ka eona e lokelang ho bokelloa ka hloko (empa re tla etsa sena hamorao, hobane hona joale re na le litaba tse tlang pele).
- Ho lokile, tsohle li felile Joale! Re na le mohlala! Na re ka qala ho koetlisa mohlala o latelang, ho nts'etsapele moaho oa ho rarolla bothata bo bocha, kapa na re ka ea noa tee? Mme ke mang ya tla tsamaisa?
Ho khetholla mathata
Ho sebetsa morerong kapa sehlahisoa ke mosebetsi oa batho ba bangata. 'Me ha nako e ntse e ea, batho ba tloha' me ba tla, ho na le merero e mengata, 'me merero ka boeona e fetoha e rarahaneng. Ka tsela e 'ngoe kapa e' ngoe, maemo a ho tloha potolohong e hlalositsoeng ka holimo (eseng feela) ka metsoako e itseng e tla etsahala ho tloha ho pheta-pheta ho ea ho pheta-pheta. Sena sohle se fella ka ho senya nako, pherekano, methapo, mohlomong ho se khotsofale ha bareki, 'me qetellong, ho lahleheloa ke chelete. Le hoja kaofela ha rōna re atisa ho latela rake la khale, ke lumela hore ha ho motho ea batlang ho pheta linako tsena khafetsa.
Ka hona, re fetile har’a potoloho e le ’ngoe ea ntlafatso ’me re bona hore ho na le mathata a lokelang ho rarolloa. Ho etsa sena o hloka:
- boloka hantle liphetho tsa mosebetsi;
- etsa hore mokhoa oa ho kenyelletsa basebetsi ba bacha o be bonolo;
- nolofatsa mokhoa oa ho tsamaisa tikoloho ea ntlafatso;
- lokisa mokhoa oa ho fetolela mohlala;
- ho ba le mokhoa o bonolo oa ho netefatsa mehlala;
- fumana sesebelisoa sa taolo ea mmuso oa mohlala;
- fumana mokhoa oa ho fana ka mehlala ho tlhahiso.
Kamoo ho bonahalang kateng hoa hlokahala ho tla le mokhoa oa ho sebetsa o tla u lumella ho laola mokhoa ona oa bophelo habonolo le ka mokhoa o bonolo? Mokhoa ona o bitsoa MLOps
MLOps, kapa DevOps bakeng sa ho ithuta ka mochini, e lumella mahlale a data le lihlopha tsa IT ho sebelisana 'moho le ho eketsa lebelo la nts'etsopele ea mohlala le phepelo ka ho beha leihlo, netefatso, le taolo bakeng sa mefuta ea ho ithuta ea mochini.
O ka khona Bashanyana ba Google ba nahana eng ka see sohle? Ho tsoa sengolong ho hlakile hore MLOps ke ntho e makatsang.
Ho feta moo sehloohong sa ka ke tla hlalosa karolo feela ea ts'ebetso. Bakeng sa ts'ebetsong, ke tla sebelisa sesebelisoa sa MLflow, hobane ... Ena ke morero oa mohloli o bulehileng, ho hlokahala palo e nyane ea khoutu ho hokahanya 'me ho na le kopanyo le meralo e tsebahalang ea ml. U ka batla marang-rang bakeng sa lisebelisoa tse ling, joalo ka Kubeflow, SageMaker, Literene, joalo-joalo, 'me mohlomong u fumane e lumellanang hantle le litlhoko tsa hau.
"Building" MLOps ho sebelisa mohlala oa ho sebelisa sesebelisoa sa MLFlow
MLFlow ke sethala sa mohloli o bulehileng oa taolo ea lifecycle ea mefuta ea ml ().
MLflow e kenyelletsa likarolo tse 'ne:
- MLflow Tracking - e akaretsa litaba tsa ho rekota sephetho le li-parameter tse lebisitseng sephethong sena;
- MLflow Project - e u lumella ho paka khoutu le ho e hlahisa sethaleng sefe kapa sefe;
- MLflow Models - e ikarabellang bakeng sa ho tsamaisa mehlala ho tlhahiso;
- MLflow Registry - e u lumella ho boloka mehlala le ho laola boemo ba bona sebakeng sa polokelo se bohareng.
MLflow e sebetsa likarolong tse peli:
- thakholoa ke potoloho e felletseng ea lithupelo, litekanyetso le methapo eo re batlang ho ingolisa ka eona;
- Teko ke "sehlooho" se tsamaeang 'moho.
Mehato eohle ea mohlala e kenngoa ts'ebetsong ea ts'ebetso ea Ubuntu 18.04.
1. Kenya seva
E le hore re laole morero oa rona habonolo mme re fumane boitsebiso bohle bo hlokahalang, re tla romela seva. Seva ea ho latela MLflow e na le likarolo tse peli tsa mantlha:
- lebenkele la backend - le ikarabellang bakeng sa ho boloka tlhahisoleseding mabapi le mehlala e ngolisitsoeng (e tšehetsa 4 DBMSs: mysql, mssql, sqlite, le postgresql);
- polokelo ea lintho tsa khale - e ikarabellang bakeng sa ho boloka lintho tsa khale (e tšehetsa likhetho tsa 7 tsa polokelo: Amazon S3, Azure Blob Storage, Google Cloud Storage, FTP seva, SFTP Server, NFS, HDFS).
E le lebenkele la lintho tsa khale Ho nolofatsa, ha re nkeng seva sa sftp.
- theha sehlopha
$ sudo groupadd sftpg - eketsa mosebedisi ebe u mo seta phasewete
$ sudo useradd -g sftpg mlflowsftp $ sudo passwd mlflowsftp - ho lokisa litlhophiso tse 'maloa tsa phihlello
$ sudo mkdir -p /data/mlflowsftp/upload $ sudo chown -R root.sftpg /data/mlflowsftp $ sudo chown -R mlflowsftp.sftpg /data/mlflowsftp/upload - eketsa mela e 'maloa ho /etc/ssh/sshd_config
Match Group sftpg ChrootDirectory /data/%u ForceCommand internal-sftp - qala tshebeletso hape
$ sudo systemctl restart sshd
E le lebenkele la mokokotlo Ha re nke postgresql.
$ sudo apt update
$ sudo apt-get install -y postgresql postgresql-contrib postgresql-server-dev-all
$ sudo apt install gcc
$ pip install psycopg2
$ sudo -u postgres -i
# Create new user: mlflow_user
[postgres@user_name~]$ createuser --interactive -P
Enter name of role to add: mlflow_user
Enter password for new role: mlflow
Enter it again: mlflow
Shall the new role be a superuser? (y/n) n
Shall the new role be allowed to create databases? (y/n) n
Shall the new role be allowed to create more new roles? (y/n) n
# Create database mlflow_bd owned by mlflow_user
$ createdb -O mlflow_user mlflow_dbHo qala seva, o hloka ho kenya liphutheloana tse latelang tsa python (ke khothaletsa ho theha tikoloho e arohaneng):
pip install mlflow
pip install pysftpHa re qale seva ea rona
$ mlflow server
--backend-store-uri postgresql://mlflow_user:mlflow@localhost/mlflow_db
--default-artifact-root sftp://mlflowsftp:mlflow@sftp_host/upload
--host server_host
--port server_port2. Eketsa ho latedisa
E le hore liphetho tsa koetliso ea rona li se ke tsa lahleha, meloko e tlang ea bahlahisi ba ka utloisisa se neng se etsahala, le hore ma-comrade a seng a hōlile le uena u khone ho hlahloba mokhoa oa ho ithuta ka mokhoa o khutsitseng, re lokela ho eketsa tatellano. Ho latela ho bolela ho boloka li-parameter, metrics, artifacts le tlhahisoleseding efe kapa efe e eketsehileng mabapi le qalo ea koetliso, molemong oa rona, ho seva.
Ka mohlala, ke entse e nyenyane ho Keras bakeng sa ho arola tsohle tse ka hare . Ho kenyelletsa tracking, ke thehile faele mlflow_training.py.
Mona ke mela eo lintho tse khahlang haholo li etsahalang:
def run(self, epochs, lr, experiment_name):
# getting the id of the experiment, creating an experiment in its absence
remote_experiment_id = self.remote_server.get_experiment_id(name=experiment_name)
# creating a "run" and getting its id
remote_run_id = self.remote_server.get_run_id(remote_experiment_id)
# indicate that we want to save the results on a remote server
mlflow.set_tracking_uri(self.tracking_uri)
mlflow.set_experiment(experiment_name)
with mlflow.start_run(run_id=remote_run_id, nested=False):
mlflow.keras.autolog()
self.train_pipeline.train(lr=lr, epochs=epochs)
try:
self.log_tags_and_params(remote_run_id)
except mlflow.exceptions.RestException as e:
print(e)Mona self.remote_server ke sephutheloana se senyenyane holim'a mekhoa ea ho latela mlflow.tracking. MlflowClient (Ke e entse bakeng sa boiketlo), ka thuso eo ke e etsang teko le ho e tsamaisa ho seva. Ka mor'a moo, ke bontša moo liphetho tsa ho qala li lokela ho kopanngoa (mlflow.set_tracking_uri(self.tracking_uri)). Ke nolofalletsa ho rema lifate mlflow.keras.autolog(). Hajoale MLflow Tracking e ts'ehetsa ho rema lifate ka boiketsetso bakeng sa TensorFlow, Keras, Gluon XGBoost, LightGBM, Spark. Haeba ha u so fumane moralo kapa laeborari ea hau, u ka lula u kena ka mokhoa o hlakileng. Re qala koetliso. Ngolisa li-tag le li-parameter tsa ho kenya ho seva e hole.
Mehala e 'maloa, 'me uena, joalo ka batho bohle, le khona ho fumana leseli mabapi le lits'ebetso tsohle. Ho pholile?
3. Re rala morero
Joale ha re e nolofatseng ho qala morero. Ho etsa sena, kenya faele ea MLproject le conda.yaml motsong oa morero.
MLproject
name: flow_segmentation
conda_env: conda.yaml
entry_points:
main:
parameters:
categories: {help: 'list of categories from coco dataset'}
epochs: {type: int, help: 'number of epochs in training'}
lr: {type: float, default: 0.001, help: 'learning rate'}
batch_size: {type: int, default: 8}
model_name: {type: str, default: 'Unet', help: 'Unet, PSPNet, Linknet, FPN'}
backbone_name: {type: str, default: 'resnet18', help: 'exampe resnet18, resnet50, mobilenetv2 ...'}
tracking_uri: {type: str, help: 'the server address'}
experiment_name: {type: str, default: 'My_experiment', help: 'remote and local experiment name'}
command: "python mlflow_training.py
--epochs={epochs}
--categories={categories}
--lr={lr}
--tracking_uri={tracking_uri}
--model_name={model_name}
--backbone_name={backbone_name}
--batch_size={batch_size}
--experiment_name={experiment_name}"MLflow Project e na le thepa e 'maloa:
- Lebitso - lebitso la morero oa hau;
- Tikoloho - tabeng ea ka, conda_env e bontša hore Anaconda e sebelisoa ho sebetsa 'me tlhaloso ea ho itšetleha e ka faeleng ea conda.yaml;
- Lintlha tsa ho Kena - e bonts'a hore na ke lifaele life le hore na re ka tsamaisa liparamente life (mekhahlelo eohle e kenngoa ka bo eona ha e qala koetliso)
conda.yaml
name: flow_segmentation
channels:
- defaults
- anaconda
dependencies:
- python==3.7
- pip:
- mlflow==1.8.0
- pysftp==0.2.9
- Cython==0.29.19
- numpy==1.18.4
- pycocotools==2.0.0
- requests==2.23.0
- matplotlib==3.2.1
- segmentation-models==1.0.1
- Keras==2.3.1
- imgaug==0.4.0
- tqdm==4.46.0
- tensorflow-gpu==1.14.0U ka sebelisa docker e le sebaka sa hau sa nako ea ho sebetsa, bakeng sa lintlha tse ling ka kopo sheba ho .
4. Ha re qaleng ho koetlisa
Re kopanya morero ebe re ea bukeng ea morero:
git clone https://github.com/simbakot/mlflow_example.git
cd mlflow_example/Ho sebetsa o hloka ho kenya lilaeborari
pip install mlflow
pip install pysftpHobane mohlala oo ke o sebelisang conda_env, Anaconda e tlameha ho kenngoa khomphuteng ea hau (empa u ka khona ho pota-pota sena ka ho kenya liphutheloana tsohle tse hlokahalang le ho bapala ka mekhahlelo ea ho qala).
Mehato eohle ea boitokisetso e phethiloe mme re ka qala ho thakhola koetliso. Ho tsoa ho motso oa projeke:
$ mlflow run -P epochs=10 -P categories=cat,dog -P tracking_uri=http://server_host:server_port .Kamora ho kenya taelo, tikoloho ea conda e tla etsoa ka bo eona mme koetliso e tla qala.
Mohlaleng o kaholimo, ke fetisitse palo ea linako tsa koetliso, mekhahlelo eo re batlang ho e arola (o ka bona lenane le felletseng. ) le aterese ea seva ea rona e hole.
Lethathamo le felletseng la liparamente tse ka bang teng le ka fumanoa faeleng ea MLproject.
5. Lekola liphello tsa ho ithuta
Kamora ho qeta koetliso, re ka kena ho sebatli ho ea atereseng ea seva sa rona
Mona re bona lethathamo la liteko tsohle (holimo ka ho le letšehali), hammoho le lintlha tse mabapi le li-runs (bohareng). Re ka sheba lintlha tse qaqileng haholoanyane (litekanyetso, metrics, li-artifacts le lintlha tse ling tse eketsehileng) bakeng sa phatlalatso e 'ngoe le e 'ngoe.
Bakeng sa metric ka 'ngoe re ka sheba nalane ea liphetoho
Tseo. Hona joale, re ka sekaseka liphetho ka mokhoa oa "manual", 'me u ka boela ua theha netefatso ea othomathike ho sebelisa MLflow API.
6. Ngolisa mohlala
Ka mor'a hore re hlahlobisise mohlala oa rona mme re etse qeto ea hore e se e loketse ntoa, re tsoela pele ho e ngolisa, bakeng sa sena re khetha tlhahiso eo re e hlokang (joalokaha ho bontšitsoe serapeng se fetileng) ebe re theoha.
Ka mor'a hore re fane ka mohlala oa rona lebitso, o na le phetolelo. Haeba u boloka mofuta o mong o nang le lebitso le tšoanang, mofuta ona o tla ntlafatsoa ka bohona.
Bakeng sa mohlala o mong le o mong, re ka eketsa tlhaloso ebe re khetha e 'ngoe ea linaha tse tharo (Staging, Production, Archived); ka mor'a moo, re sebelisa API, re ka fihlella linaha tsena, tseo, hammoho le phetolelo, li fanang ka phetoho e eketsehileng.
Re boetse re na le phihlello e bonolo ho mefuta eohle
le liphetolelo tsa bona
Joalo ka serapeng se fetileng, lits'ebetso tsohle li ka etsoa ho sebelisoa API.
7. Kenya mohlala
Mothating ona, re se re na le mohlala o koetlisitsoeng (keras). Mohlala oa hore na u ka e sebelisa joang:
class SegmentationModel:
def __init__(self, tracking_uri, model_name):
self.registry = RemoteRegistry(tracking_uri=tracking_uri)
self.model_name = model_name
self.model = self.build_model(model_name)
def get_latest_model(self, model_name):
registered_models = self.registry.get_registered_model(model_name)
last_model = self.registry.get_last_model(registered_models)
local_path = self.registry.download_artifact(last_model.run_id, 'model', './')
return local_path
def build_model(self, model_name):
local_path = self.get_latest_model(model_name)
return mlflow.keras.load_model(local_path)
def predict(self, image):
image = self.preprocess(image)
result = self.model.predict(image)
return self.postprocess(result)
def preprocess(self, image):
image = cv2.resize(image, (256, 256))
image = image / 255.
image = np.expand_dims(image, 0)
return image
def postprocess(self, result):
return resultMona self.registry ke hape sephutheloana se senyenyane holim'a mlflow.tracking.MlflowClient, bakeng sa boiketlo. Taba ke hore ke fihlella seva se hole mme ke batla mohlala moo ka lebitso le boletsoeng, le mofuta oa morao-rao oa tlhahiso. Ka mor'a moo, ke khoasolla lintho tse entsoeng ka matsoho sebakeng sa heno ho ./model foldareng ebe ke aha mohlala ho tsoa bukeng ena mlflow.keras.load_model(local_path). Hona joale re ka sebelisa mohlala oa rona. Baetsi ba CV (ML) ba ka ntlafatsa mofuta ona habonolo mme ba phatlalatsa mefuta e mecha.
Qetellong
Ke hlahisitse sistimi e lumellang:
- boloka boitsebiso bo bohareng mabapi le mefuta ea ML, tsoelo-pele ea koetliso le liphetho;
- ho tsamaisa tikoloho ea ntlafatso kapele;
- hlokomela le ho sekaseka tsoelopele ea mosebetsi oa mehlala;
- ho bonolo ho fetolela le ho laola boemo ba mehlala;
- Ho bonolo ho tsamaisa mehlala e hlahisoang.
Mohlala ona ke ntho ea ho bapala 'me e sebetsa e le qalo ea ho haha sistimi ea hau, e ka kenyelletsang boiketsetso ba tlhahlobo ea liphetho le ngoliso ea mehlala (lintlha tsa 5 le 6, ka ho latellana) kapa o tla eketsa mofuta oa li-dataset, kapa mohlomong ho hong? Taba eo ke neng ke leka ho e bua ke hore o hloka MLOps ka kakaretso, MLflow ke mokhoa oa ho fihlela pheletso.
Ngola mathata ao u kopaneng le ona ao ke sa kang ka a bontša?
U ka eketsa eng ho sistimi ho etsa hore e finyelle litlhoko tsa hau?
Ke lisebelisoa life le mekhoa eo u e sebelisang ho rarolla mathata ohle kapa karolo ea ona?
PS Ke tla siea lihokelo tse 'maloa:
morero oa github -
MLflow -
Imeile ea ka ea mosebetsi bakeng sa lipotso - [imeile e sirelelitsoe]
Khampani ea rona nako le nako e tšoara liketsahalo tse sa tšoaneng bakeng sa litsebi tsa IT, mohlala: ka la 8 July ka 19:00 Moscow nako ea Moscow ho tla ba le kopano ea CV ka mokhoa oa inthaneteng, haeba u thahasella, u ka nka karolo, ho ngolisa. .
Source: www.habr.com