🥇MLOps - Buuga karinta, cutubka 1

Salaamu calaykum! Waxaan ahay horumariye CV ka ah CROC. Muddo hadda laga joogo 3 sano ayaanu mashaariic ka fulinaynay dhinaca CV-ga. Muddadaas waxaan sameynay waxyaabo badan, tusaale ahaan: waxaan la soconay darawalada si aysan u cabbin, sigaar cabin, aysan ku hadlin taleefanka, eegin wadada, oo aysan u eegin riyo ama daruuraha. ; Waxaan duubnay dadka ku kaxeeya waddooyin gaar ah oo qaada dhawr meelood oo baabuurta la dhigto; la hubiyay in shaqaaluhu ay xidhaan koofiyadaha, galoofyada, iwm. aqoonsaday shaqaale raba inuu galo xarunta; Waxaan tirinay wax kasta oo aan kari karnay.

Maxaan waxan oo dhan u samaynayaa?

Geedi socodka fulinta mashaariicda, waxaan ku dhuftey kuuskuus, kuuskuus badan, qaar ka mid ah dhibaatooyinka aad ama aad taqaanid ama aad baran doonto mustaqbalka.

Aan ku dayano xaalada

Aynu qiyaasno inaan shaqo ka helnay shirkad dhallinyaro ah "N", kuwaas oo hawlahooda ay la xiriiraan ML. Waxaan ka shaqeynaa mashruuca ML (DL, CV), ka dib sabab qaar ka mid ah waxaan u wareegnaa shaqo kale, guud ahaan nasasho, oo ku soo laabano annaga ama qof kale neuron.

Waqtiga runta ayaa imanaya, waxaad u baahan tahay inaad si uun u xasuusato meesha aad joogsatay, waxa hyperparameters aad isku dayday iyo, ugu muhiimsan, maxay natiijooyinka ay keeneen. Waxaa jiri kara xulashooyin badan oo loogu talagalay cidda ku kaydisay macluumaadka dhammaan bilaabista: madaxa, qaabeynta, qoraalka qoraalka, jawiga shaqada ee daruuraha. Waxaan ku dhacay inaan arko ikhtiyaar markii hyperparameters lagu kaydiyay sida khadadka faallooyinka ee koodhka, guud ahaan, duulimaad qurux badan. Hadda ka soo qaad inaadan ku soo noqon mashruucaagii, balse u soo noqotay mashruucii qof shirkadda ka tagay oo aad ka dhaxashay kood iyo model la yidhaahdo model_1.pb. Si aad u dhammaystirto sawirka oo aad u gudbiso dhammaan xanuunka, aynu ka fikirno inaad sidoo kale tahay khabiir bilow ah.
Horey u soco Si loo socodsiiyo koodhka, annaga iyo qof kasta oo la shaqayn doona waxaanu u baahanahay inaanu abuurno deegaan. Inta badan waxay dhacdaa in sabab qaar ka mid ah ayan ka tegin isaga oo ah dhaxalkeenna. Tani waxay sidoo kale noqon kartaa hawl aan fududayn. Ma rabto inaad wakhti ku lumiso tallaabadan, miyaad tahay?
Waxaan tababarnaa nooc (tusaale ahaan, qalabka baabuurka). Waxaan gaadhnay meesha ay aad u wanaagsan tahay - waa waqtigii lagu badbaadin lahaa natiijada. Aynu u yeedhno gaadhi_helid_v1.pb. Kadibna waxaan tababarnaa mid kale - car_detection_v2.pb. Muddo ka dib, asxaabteena ama anaga lafteena ayaa aad iyo aad u baraya, anagoo adeegsanayna naqshado kala duwan. Natiijo ahaan, farabadan oo farshaxan ah ayaa la sameeyay, macluumaadka ku saabsan kuwaas oo u baahan in si taxadar leh loo ururiyo (laakiin tan dambe ayaan sameyn doonaa, sababtoo ah hadda waxaan haynaa arrimo badan oo mudnaanta leh).
OK way dhammaatay Hadda! Waxaan leenahay model ah! Ma bilaabi karnaa tababbarka qaabka xiga, samaynta qaab-dhismeedka si loo xalliyo dhib cusub, mise ma aadi karnaa shaah? yaase hawl gelin doona?

Aqoonsiga dhibaatooyinka

Ka shaqaynta mashruuc ama badeecad waa shaqada dad badan. Muddo ka dib, dadku way iska tagaan oo way yimaadaan, waxaa batay mashruucyo, mashaariicda laftooduna way sii adkaanayaan. Hal hab ama si kale, xaaladaha ka imanaya wareegga kor lagu sharraxay (oo kaliya maaha) ee isku-dhafka qaarkood waxay ka dhici doonaan soo noqnoqoshada ilaa soo noqnoqoshada. Dhammaan tani waxay keenaysaa waqti lumis, jahawareer, neerfayaasha, suurtogalnimada qanacsanaanta macaamiisha, iyo ugu dambeyntii, luminta lacag. In kasta oo aan dhammaanteen raacno isla ragii duugga ahaa, waxaan aaminsanahay in qofna uusan rabin in uu soo noolaado daqiiqadahan marar badan.

Haddaba, waxa aynu soo marnay hal wareeg oo horumarineed oo aynu aragno in ay jiraan dhibaatooyin u baahan in la xalliyo. Si aad tan u samayso waxaad u baahan tahay:

si habboon u kaydi natiijooyinka shaqada;
ka dhig habka ka qayb galka shaqaalaha cusub mid fudud;
fududaynta habka geynta deegaan horumarineed;
habayn habka versioning model;
leeyihiin hab ku habboon oo lagu ansixiyo moodooyinka;
Soo hel nooc ah qalab maamulka gobolka;
hel hab aad ku gaadhsiiso moodooyinka wax soo saarka.

Sida muuqata waa lagama maarmaan in la yimaado hab-socodka shaqada kaas oo kuu ogolaanaya inaad si fudud oo habboon u maareyso wareegga nolosha? Dhaqankan waxaa loo yaqaan MLOps

MLOps, ama DevOps ee barashada mashiinka, waxay u ogolaataa sayniska xogta iyo kooxaha IT-ga inay iska kaashadaan oo ay kordhiyaan xawliga horumarinta moodeelka iyo geynta iyada oo loo marayo la socodka, ansixinta, iyo maamulka moodooyinka barashada mashiinka.

Waxaad awoodi kartaa akhriMaxay ragga Google ka qabaan waxaas oo dhan? Maqaalka waxaa iska cad in MLOps ay tahay wax aad u weyn.

Dheeraad ah maqaalkayga waxaan ku tilmaami doonaa qayb ka mid ah habka. Hirgelinta, waxaan isticmaali doonaa qalabka MLflow, sababtoo ah... Kani waa mashruuc il furan, qadar yar oo kood ah ayaa loo baahan yahay in lagu xidho waxaana jira is dhexgalka qaabab ml oo caan ah. Waxaad ka raadin kartaa internetka qalabyada kale, sida Kubeflow, SageMaker, Trains, iwm, oo laga yaabo inaad hesho mid ku habboon baahiyahaaga.

"Dhismaha" MLOps iyadoo la isticmaalayo tusaale ahaan isticmaalka qalabka MLFlow

MLFlow waa il furan oo loogu talagalay maaraynta meertada nolosha ee moodooyinka ml (https://mlflow.org/).

MLflow waxay ka kooban tahay afar qaybood:

MLflow Tracking - waxay dabooshaa arrimaha duubista natiijooyinka iyo cabbirrada keenay natiijadan;
Mashruuca MLflow - wuxuu kuu ogolaanayaa inaad xirto koodka oo aad ku daabacdo goob kasta;
Moodooyinka MLflow - mas'uul ka ah geynta moodooyinka wax soo saarka;
Diiwaanka MLflow - wuxuu kuu ogolaanayaa inaad kaydiso moodooyinka oo aad ku maamusho gobolkooda meel dhexe.

MLflow waxay ku shaqaysaa laba qaybood:

Daah-furku waa wareeg buuxa oo tababar ah, cabbirro iyo cabbirro kuwaas oo aan rabno in aan isku diiwaan-gelinno;
Tijaabadu waa "mowduuc" oo wada socda.

Dhammaan tillaabooyinka tusaalaha waxaa lagu hirgeliyay nidaamka hawlgalka Ubuntu 18.04.

1. Geli serverka

Si aan si fudud ugu maamulno mashruucayaga oo aan u helno dhammaan macluumaadka lagama maarmaanka ah, waxaan geyn doonaa server. Adeegga raadraaca MLflow wuxuu leeyahay laba qaybood oo waaweyn:

dukaanka dhabarka - mas'uul ka ah kaydinta macluumaadka ku saabsan moodooyinka diiwaangashan (taageera 4 DBMSs: mysql, mssql, sqlite, iyo postgresql);
dukaanka artifact - mas'uul ka ah kaydinta artifacts ( taageertaa 7 fursadaha kaydinta: Amazon S3, Azure Blob Kaydinta, Google Cloud Kaydinta, server FTP, SFTP Server, NFS, HDFS).

Sida dukaanka artifact Si ay u fududaato, aynu soo qaadano server-ka sftp.

koox abuur
```
$ sudo groupadd sftpg
```

ku dar isticmaale oo u samee furaha sirta ah

$ sudo useradd -g sftpg mlflowsftp
$ sudo passwd mlflowsftp

hagaajinta dhowr goobood oo gelitaan

$ sudo mkdir -p /data/mlflowsftp/upload
$ sudo chown -R root.sftpg /data/mlflowsftp
$ sudo chown -R mlflowsftp.sftpg /data/mlflowsftp/upload

ku dar dhowr xariiq /etc/ssh/sshd_config

Match Group sftpg
 ChrootDirectory /data/%u
 ForceCommand internal-sftp

dib u bilow adeegga
```
$ sudo systemctl restart sshd
```

Sida dukaanka dhabarka Aynu qaadano postgresql.

$ sudo apt update
$ sudo apt-get install -y postgresql postgresql-contrib postgresql-server-dev-all
$ sudo apt install gcc
$ pip install psycopg2
$ sudo -u postgres -i
# Create new user: mlflow_user
[postgres@user_name~]$ createuser --interactive -P
Enter name of role to add: mlflow_user
Enter password for new role: mlflow
Enter it again: mlflow
Shall the new role be a superuser? (y/n) n
Shall the new role be allowed to create databases? (y/n) n
Shall the new role be allowed to create more new roles? (y/n) n
# Create database mlflow_bd owned by mlflow_user
$ createdb -O mlflow_user mlflow_db

Si aad u bilowdo server-ka, waxaad u baahan tahay inaad ku rakibto xirmooyinka Python ee soo socda (Waxaan ku talinayaa in la abuuro jawi macmal ah oo gooni ah):

pip install mlflow
pip install pysftp

Aan bilowno server-keena

$ mlflow server  
                 --backend-store-uri postgresql://mlflow_user:mlflow@localhost/mlflow_db 
                 --default-artifact-root sftp://mlflowsftp:mlflow@sftp_host/upload  
                --host server_host 
                --port server_port

2. Ku dar raad raac

Si aan natiijada tababarkeena u lumin, jiilalka mustaqbalka ee horumariyayaashu si ay u fahmaan waxa dhacaya, iyo saaxiibada da'da weyn iyo adiga oo awood u leh in aad si degan u falanqeeyaan habka waxbarashada, waxaan u baahanahay inaan ku darno raadraaca. Dabagalku macnihiisu waa kaydinta cabbirada, cabbirada, farshaxanada iyo macluumaad kasta oo dheeraad ah oo ku saabsan bilowga tababarka, xaaladeena, ee serverka.

Tusaale ahaan, waxaan sameeyay mid yar mashruuca github on Keras si loo kala saaro wax kasta oo ku jira Xogta COCO. Si aan ugu daro raadraaca, waxaan sameeyay fayl mlflow_training.py.

Waa kuwan khadadka ay ka dhacaan waxyaabaha ugu xiisaha badan:

def run(self, epochs, lr, experiment_name):
        # getting the id of the experiment, creating an experiment in its absence
        remote_experiment_id = self.remote_server.get_experiment_id(name=experiment_name)
        # creating a "run" and getting its id
        remote_run_id = self.remote_server.get_run_id(remote_experiment_id)

        # indicate that we want to save the results on a remote server
        mlflow.set_tracking_uri(self.tracking_uri)
        mlflow.set_experiment(experiment_name)

        with mlflow.start_run(run_id=remote_run_id, nested=False):
            mlflow.keras.autolog()
            self.train_pipeline.train(lr=lr, epochs=epochs)

        try:
            self.log_tags_and_params(remote_run_id)
        except mlflow.exceptions.RestException as e:
            print(e)

Halkan self.remote_server waa duub yar oo ka sarreeya hababka mlflow.tracking. MlflowClient (Waxaan u sameeyay si ku habboon), iyadoo la kaashanayo taas oo aan abuuro tijaabo oo aan ku socodsiiyo server-ka. Marka xigta, waxaan tilmaamayaa halka natiijooyinka bilaabista la isku darayo (mlflow.set_tracking_uri(self.tracking_uri)). Waxaan awood u geliyaa si toos ah mlflow.keras.autolog(). Hadda MLflow Tracking waxay taageertaa galitaanka tooska ah ee TensorFlow, Keras, Gluon XGBoost, LightGBM, Spark. Haddii aadan helin qaab-dhismeedkaaga ama maktabaddaada, markaa had iyo jeer si cad ayaad u gali kartaa. Waxaan bilaabaynaa tababar. Diiwaangeli calaamadaha iyo cabbiraadaha gelinta serverka fog.

Dhowr sadar oo adiga, sida dadka kale, ayaa heli kara macluumaadka ku saabsan dhammaan bilaabista. Qabow?

3. Waxaan diyaarineynaa mashruuca

Hadda aynu fududeyno bilaabida mashruuca. Si tan loo sameeyo, ku dar faylka MLproject iyo conda.yaml xididka mashruuca.
Mashruuca ML

name: flow_segmentation
conda_env: conda.yaml

entry_points:
  main:
    parameters:
        categories: {help: 'list of categories from coco dataset'}
        epochs: {type: int, help: 'number of epochs in training'}

        lr: {type: float, default: 0.001, help: 'learning rate'}
        batch_size: {type: int, default: 8}
        model_name: {type: str, default: 'Unet', help: 'Unet, PSPNet, Linknet, FPN'}
        backbone_name: {type: str, default: 'resnet18', help: 'exampe resnet18, resnet50, mobilenetv2 ...'}

        tracking_uri: {type: str, help: 'the server address'}
        experiment_name: {type: str, default: 'My_experiment', help: 'remote and local experiment name'}
    command: "python mlflow_training.py 
            --epochs={epochs}
            --categories={categories}
            --lr={lr}
            --tracking_uri={tracking_uri}
            --model_name={model_name}
            --backbone_name={backbone_name}
            --batch_size={batch_size}
            --experiment_name={experiment_name}"

Mashruuca MLflow wuxuu leeyahay sifooyin dhowr ah:

Magaca - magaca mashruucaaga;
Deegaanka - xaaladdeyda, conda_env waxay muujineysaa in Anaconda loo isticmaalo in lagu shaqeeyo oo sharaxaadda ku tiirsanaanta ay ku jirto faylka conda.yaml;
Dhibcaha Gelitaanka - waxay muujisaa faylalka iyo cabbiraadaha aan ku socon karno (dhammaan xuduudaha si toos ah ayaa loo diiwaangeliyaa marka tababarka la bilaabayo)

conda.yaml

name: flow_segmentation
channels:
  - defaults
  - anaconda
dependencies:
  - python==3.7
  - pip:
    - mlflow==1.8.0
    - pysftp==0.2.9
    - Cython==0.29.19
    - numpy==1.18.4
    - pycocotools==2.0.0
    - requests==2.23.0
    - matplotlib==3.2.1
    - segmentation-models==1.0.1
    - Keras==2.3.1
    - imgaug==0.4.0
    - tqdm==4.46.0
    - tensorflow-gpu==1.14.0

Waxaad u isticmaali kartaa docker sida jawiga runtime, wixii faahfaahin dheeraad ah fadlan tixraac dukumentiyo.

4. Aan bilowno tababarka

Waxaanu xidhnay mashruuca oo aanu tagnaa hagaha mashruuca:

git clone https://github.com/simbakot/mlflow_example.git
cd mlflow_example/

Si aad u shaqeyso waxaad u baahan tahay inaad ku rakibto maktabado

pip install mlflow
pip install pysftp

Sababtoo ah Tusaalaha aan isticmaalo conda_env, Anaconda waa in lagu rakibaa kombuyuutarkaaga (laakiin waxaad ka heli kartaa tan adigoo ku rakibaya dhammaan baakadaha lagama maarmaanka ah naftaada oo aad la ciyaareyso xuduudaha bilowga).

Dhammaan tillaabooyinka diyaarinta waa la dhammaystiray waxaanan bilaabi karnaa inaan bilowno tababarka. Laga soo bilaabo xididka mashruuca:

$ mlflow run -P epochs=10 -P categories=cat,dog -P tracking_uri=http://server_host:server_port .

Kadib gelitaanka taliska, jawiga konda ayaa si toos ah loo abuuri doonaa tababarkuna wuu bilaaban doonaa.
Tusaalaha kore, waxaan ka gudbay tirada xilliyada tababarka, qaybaha aan rabno inaan u kala saarno (waxaad arki kartaa liiska buuxa halkan) iyo ciwaanka seerfarkeena fog.
Liis dhammaystiran oo cabbirro macquul ah ayaa laga heli karaa faylka MLproject.

5. Qiimee natiijada waxbarashada

Kadib marka aan dhammeyno tababarka, waxaan gali karnaa browserka ciwaanka serverkayaga http://server_host:server_port

Halkan waxaan ku aragnaa liiska dhammaan tijaabooyinka (bidix sare), iyo sidoo kale macluumaadka ku saabsan orodka (dhexe). Waxaan arki karnaa macluumaad aad u faahfaahsan (halbeegyada, cabbirada, farshaxanimada iyo macluumaad dheeraad ah) ee bilawga kasta.

Halbeeg kasta waxaan u ilaalin karnaa taariikhda isbeddelka

Kuwaas. Waqtigan xaadirka ah waxaan ku falanqeyn karnaa natiijooyinka qaab "Manual", waxaad sidoo kale dejin kartaa ansaxinta tooska ah adoo isticmaalaya MLflow API.

6. Diiwaangeli qaabka

Ka dib markii aan falanqeynay qaabkayaga oo aan go'aansanay in ay diyaar u tahay dagaalka, waxaan sii wadnaa inaan diiwaan geliyo, tan waxaan dooraneynaa bilawga aan u baahanahay (sida ku cad cutubka hore) oo hoos u dhig.

Ka dib markii aan siin model our magac, waxa ay leedahay version. Haddii aad kaydsato nooc kale oo isla magac ah, nooca si toos ah ayaa loo cusboonaysiin doonaa.

Nooc kasta, waxaan ku dari karnaa sharaxaad oo waxaan dooran karnaa mid ka mid ah saddexda gobol (Staging, Production, Archived); ka dibna, adoo isticmaalaya API-ga, waxaan geli karnaa goboladan, kaas oo, oo ay weheliso nuqul ka mid ah, waxay bixisaa dabacsanaan dheeraad ah.

Waxaan sidoo kale si fudud u heli karnaa dhammaan moodooyinka

iyo noocyadooda

Sida cutubkii hore, dhammaan hawlgallada waxaa lagu samayn karaa API-ga.

7. Geli qaabka

Marxaladdan, waxaan horay u haysanay moodal tababbaran (kera). Tusaale ahaan sida aad u isticmaali karto:

class SegmentationModel:
    def __init__(self, tracking_uri, model_name):

        self.registry = RemoteRegistry(tracking_uri=tracking_uri)
        self.model_name = model_name
        self.model = self.build_model(model_name)

    def get_latest_model(self, model_name):
        registered_models = self.registry.get_registered_model(model_name)
        last_model = self.registry.get_last_model(registered_models)
        local_path = self.registry.download_artifact(last_model.run_id, 'model', './')
        return local_path

    def build_model(self, model_name):
        local_path = self.get_latest_model(model_name)

        return mlflow.keras.load_model(local_path)

    def predict(self, image):
        image = self.preprocess(image)
        result = self.model.predict(image)
        return self.postprocess(result)

    def preprocess(self, image):
        image = cv2.resize(image, (256, 256))
        image = image / 255.
        image = np.expand_dims(image, 0)
        return image

    def postprocess(self, result):
        return result

Halkan self.registry waa mar kale duub yar oo dulmarsan mlflow.tracking.MlflowClient, si ay ugu sahlanaato. Ujeedadu waa in aan galo server fog oo aan ka raadiyo moodal leh magaca la cayimay, iyo nooca ugu dambeeyay ee wax soo saarka. Marka xigta, waxaan u soo dejiyaa artifact gudaha galka ./model oo waxaan ka dhisay qaabka buuggan mlflow.keras.load_model(local_path). Hadda waxaan isticmaali karnaa qaabkayaga. Horumarinta CV (ML) waxay si fudud u wanaajin karaan moodalka oo daabici karaan noocyo cusub.

Gabagabada

Waxaan soo bandhigay nidaam ogolaanaya:

meel dhexe u kaydi macluumaadka ku saabsan moodooyinka ML, horumarka tababarka iyo natiijooyinka;
si deg deg ah loo geeyo deegaan horumarineed;
la socdo oo lafo-gureeyo horumarka shaqada ee moodooyinka;
way ku habboon tahay in la daabaco oo la maareeyo xaaladda moodooyinka;
Way fududahay in la geeyo moodooyinka soo baxay.

Tusaalahani waa toy wuxuuna u adeegaa sidii barta bilowga ah ee lagu dhisayo nidaamkaaga, kaas oo laga yaabo inay ku jiraan otomaatig ah qiimeynta natiijooyinka iyo diiwaangelinta moodooyinka (dhibcaha 5 iyo 6, siday u kala horreeyaan) ama waxaad ku dari doontaa nuqulka xogta, ama laga yaabee wax kale? Qodobka aan isku dayay inaan sameeyo ayaa ah inaad u baahan tahay MLOps guud ahaan, MLflow waa uun hab lagu dhammeeyo.

Qor dhibaatooyinka aad la kulantay ee aanan soo bandhigin?
Maxaad ku dari lahayd nidaamka si uu u daboolo baahiyahaaga?
Qalabkee iyo dariiqooyinkee ayaad isticmaashaa si aad u xalliso dhammaan ama qayb ka mid ah dhibaatooyinka?

PS Waxaan uga tagayaa dhowr xiriiriye:
mashruuca github - https://github.com/simbakot/mlflow_example
MLflow - https://mlflow.org/
Iimaylkayga shaqada ee su'aalaha - [emailka waa la ilaaliyay]

Shirkadeena waxay si xilliyo ah u qabataa munaasabado kala duwan oo loogu talagalay khabiirada IT-ga, tusaale ahaan: Luulyo 8 markay tahay 19:00 wakhtiga Moscow waxaa jiri doona kulan CV oo qaab online ah, haddii aad xiisaynayso, waad ka qayb qaadan kartaa, isdiiwaangelinta halkan .

Source: www.habr.com

MLOps - Buuga karinta, cutubka 1