ProHoster > Π‘Π»ΠΎΠ³ > administratio > Extendit scintilla cum MLflow

Extendit scintilla cum MLflow

Salve, Khabrovsk habitant. Ut iam scripsimus, hic mensis OTUS duas machinas discendi cursus simul emittit, scilicet basis ΠΈ provectus. Qua in re utilem materiam communicare pergimus.

Propositum huius articuli de prima experientia loqui est usus MLflow.

Nos incipiemus a review MLflow a servo suo vestigando et aperi omnes iterationes studiorum. Experientiam nostram connectendi Scintillam cum MLflow utendi UDF erimus.

context

Nos sumus in Alpha Health Machina discendi et intellegentiae artificialis utimur ad talem homines ut suae valetudinis et bene esse curam habeant. Ideo apparatus discendi exempla in corde notitiarum scientiarum productorum enucleatur, et ideo ad MLflow trahimur, fons apertus suggestus qui omnes rationes machinae discendi lifecycli tegit.

MLflow

Praecipuum propositum MLflow est praebere additam iacum super machinae discendi quae notitias scientiarum patiatur laborare cum nulla fere bibliotheca discendi apparatus (h2o, Keras, mleap, pytorch, sklearn ΠΈ tensorflow) Opus suum ad gradum capiens.

MLflow tria praebet partes:

  • semita - notare et petitiones experimentorum: codicem, datam, configurationem et eventum. Cras processus exemplar creandi magni momenti est.
  • projects - Packaging forma ut run in quolibet suggestu (exempli gratia. SageMaker)
  • exemplum – forma communis ad exempla exhibenda variis instrumentis instruere.

MLflow (in alpha tempore scriptionis) suggestum apertum est quod sinit te machinam discendi vivendi, inter experimenta, reuse et instruere.

Occasus sursum MLflow

MLflow uti, debes primum totam Python ambitum tuam constituere, hoc enim utemur PyEnv (Pythonem in Mac installare, reprehendo sicco hic). Hoc modo virtualem ambitum creare possumus quo instituemus omnes bibliothecas necessarias ad eam currendam.

```
pyenv install 3.7.0
pyenv global 3.7.0 # Use Python 3.7
mkvirtualenv mlflow # Create a Virtual Env with Python 3.7
workon mlflow
```

Let's install the required libraries.

```
pip install mlflow==0.7.0 
            Cython==0.29  
            numpy==1.14.5 
            pandas==0.23.4 
            pyarrow==0.11.0
```

Nota: PyArrow utimur ad exempla currendi ut UDF. versiones PyArrow et Numpy fixae sunt quia hae versiones inter se dissidebant.

Lorem mauris UI

MLflow Semita nobis permittit ut experimenta et interrogationes aperias utentes Pythone et OTIUM API. Praeterea determinare potes ubi exemplar artificialia reponunt (localhost; Amazon S3, Caeruleus Blob PRAECLUSIO, Google Cloud PRAECLUSIO aut SFTP servo). Cum utimur AWS in Alpha Health, artificium repositum nostrum erit S3.

# Running a Tracking Server
mlflow server 
    --file-store /tmp/mlflow/fileStore 
    --default-artifact-root s3://<bucket>/mlflow/artifacts/ 
    --host localhost
    --port 5000

MLflow suadet utens lima repono persistentes. Tabularium tabulariorum est ubi servo thesaurizabit currere et experimentum metadata. Incipiens a servo, fac eam demonstrat perseverantiorum tabellariorum copia. Hic ad experimentum simpliciter utimur /tmp.

Memento quod si velimus ut servo mlflow uti ad antiqua experimenta currendam, in tabula repositionis adesse debent. Nihilominus etiam sine hac in UDF uti potuimus, quandoquidem ad exemplar solum opus est.

Nota: Meminerint sequi vestigia UI et clientem exemplarium accessum habere ad artificii locum. Id est, pro eo quod Semita UI in casu EC2 residet, cum localiter MLflow currit, apparatus directam accessum habere debet ad S3 ad exempla artificii scribenda.

Extendit scintilla cum MLflow
Tracking III stores artificialibus in S3 situla

Donec cursus

Statim ac cum servo mauris currit, exempla monstrare potes.

Exempli causa, modificatione vini utemur ex exemplo MLflow in . Sklearn.

MLFLOW_TRACKING_URI=http://localhost:5000 python wine_quality.py 
  --alpha 0.9
  --l1_ration 0.5
  --wine_file ./data/winequality-red.csv

Ut iam disputavimus, MLflow permittit ut exemplar parametri, metrics, et artificia aperias ut indagare potes quomodo per iterationes evolvere potes. Haec factura perquam utilis est quia hoc modo exemplar optimum effingere possumus per contactum servo vel intellectui Semita, quae codicem explevit, utens git Nullam tigna committit. 

with mlflow.start_run():

    ... model ...

    mlflow.log_param("source", wine_path)
    mlflow.log_param("alpha", alpha)
    mlflow.log_param("l1_ratio", l1_ratio)

    mlflow.log_metric("rmse", rmse)
    mlflow.log_metric("r2", r2)
    mlflow.log_metric("mae", mae)

    mlflow.set_tag('domain', 'wine')
    mlflow.set_tag('predict', 'quality')
    mlflow.sklearn.log_model(lr, "model")

Extendit scintilla cum MLflow
Vinum iterations

Servo partem ad exemplar

MLflow servitor sequi, utens "mlflow servo" mandatum, requiem API habet ad vestigia currentium et scribens notitias ad systema fasciculi localis. Servum sequi electronicum notare potes utens ambitus variabilis "MLFLOW_TRACKING_URI" et MLflow semita API sponte contactum servo tracking hac inscriptione ad creandum/accipe informationes launch, metrics stipes, etc.

Source: Docs // Thronus tracking server

Ad exemplar cum servo praebendum, servo currenti vestibulo (vide interface launch) et ID currite exemplaris.

Extendit scintilla cum MLflow
Curre ID

# Serve a sklearn model through 127.0.0.0:5005
MLFLOW_TRACKING_URI=http://0.0.0.0:5000 mlflow sklearn serve 
  --port 5005  
  --run_id 0f8691808e914d1087cf097a08730f17 
  --model-path model

Ad exempla serviendi utentes MLflow serviant officiativi, accessum ad Semita UI indigebimus ut informationes accipias de exemplari simpliciter definiendo. --run_id.

Cum exemplum contactus cum servo semitario, novum exemplar punctum attingere possumus.

# Query Tracking Server Endpoint
curl -X POST 
  http://127.0.0.1:5005/invocations 
  -H 'Content-Type: application/json' 
  -d '[
	{
		"fixed acidity": 3.42, 
		"volatile acidity": 1.66, 
		"citric acid": 0.48, 
		"residual sugar": 4.2, 
		"chloridessssss": 0.229, 
		"free sulfur dsioxide": 19, 
		"total sulfur dioxide": 25, 
		"density": 1.98, 
		"pH": 5.33, 
		"sulphates": 4.39, 
		"alcohol": 10.8
	}
]'

> {"predictions": [5.825055635303461]}

Donec a currit scintilla

Non obstante quod Servus Semita potens est ad exempla in reali tempore ministrare, ea exercere et ministris functionality uti (source: mlflow // soUicitudo // exempla # locale) Scintillae (batch vel effusis) usus est solutio magis validior propter distributionem.

Finge te simpliciter fecisse online disciplina et deinde exemplar output ad omnia tua data applicasse. Hoc est ubi scintilla et MLflow lucent.

Install PySpark + Iuppiter + Spark

Source: Incipias PySpark - Jupyter

Ut ostenderemus quomodo exempla MLflow ad Spark dataframes applicamus, opus est ut codicillos Iupyter erigat qui una cum PySpark cooperantur.

Satus installing tardus poema firmum Apache Spark:

cd ~/Downloads/
tar -xzf spark-2.4.3-bin-hadoop2.7.tgz
mv ~/Downloads/spark-2.4.3-bin-hadoop2.7 ~/
ln -s ~/spark-2.4.3-bin-hadoop2.7 ~/sparkΜ€

Instrue PySpark et Iuppiter in ambitu virtuali:

pip install pyspark jupyter

Elit variabiles extruxerat:

export SPARK_HOME=~/spark
export PATH=$SPARK_HOME/bin:$PATH
export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPARK_DRIVER_PYTHON_OPTS="notebook --notebook-dir=${HOME}/Projects/notebooks"

Constitutis notebook-dir, codicillos nostros in folder optato possumus reponere.

Deductis Jupyter ex PySpark

Cum possimus Iovem configurare tamquam aurigam PySpark, nunc libellum Iupyter in contextu PySpark currere possumus.

(mlflow) afranzi:~$ pyspark
[I 19:05:01.572 NotebookApp] sparkmagic extension enabled!
[I 19:05:01.573 NotebookApp] Serving notebooks from local directory: /Users/afranzi/Projects/notebooks
[I 19:05:01.573 NotebookApp] The Jupyter Notebook is running at:
[I 19:05:01.573 NotebookApp] http://localhost:8888/?token=c06252daa6a12cfdd33c1d2e96c8d3b19d90e9f6fc171745
[I 19:05:01.573 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 19:05:01.574 NotebookApp]

    Copy/paste this URL into your browser when you connect for the first time,
    to login with a token:
        http://localhost:8888/?token=c06252daa6a12cfdd33c1d2e96c8d3b19d90e9f6fc171745

Extendit scintilla cum MLflow

Ut supra, MLflow praebet plumam pro logis artificialibus in S3. Quamprimum exemplar electum in manibus habemus, opportunitatem habemus eam importare ut UDF moduli utendi mlflow.pyfunc.

import mlflow.pyfunc

model_path = 's3://<bucket>/mlflow/artifacts/1/0f8691808e914d1087cf097a08730f17/artifacts/model'
wine_path = '/Users/afranzi/Projects/data/winequality-red.csv'
wine_udf = mlflow.pyfunc.spark_udf(spark, model_path)

df = spark.read.format("csv").option("header", "true").option('delimiter', ';').load(wine_path)
columns = [ "fixed acidity", "volatile acidity", "citric acid",
            "residual sugar", "chlorides", "free sulfur dioxide",
            "total sulfur dioxide", "density", "pH",
            "sulphates", "alcohol"
          ]
          
df.withColumn('prediction', wine_udf(*columns)).show(100, False)

Extendit scintilla cum MLflow
PySpark - Outputting vini qualitas praedictiones

Huc usque locuti sumus quomodo usus PySpark cum MLflowo uteretur, praedictiones qualis vinum in toto vino dataset. Sed quid si Pythone MLflow modulis ex Scala Scintilla utere debes?

Hoc quoque probavimus scindendo contextum Scintillae inter Scalam et Pythonem. Hoc est, MLflow UDF in Pythone descripsimus, eoque usi sumus a Scala (sic, fortasse non optima solutione, sed quod habemus).

Scala scintilla + MLflow

Ad hoc exemplum addemus Toree Kernel in exsistens lovis.

Install Scintilla + Toree + Jupyter

pip install toree
jupyter toree install --spark_home=${SPARK_HOME} --sys-prefix
jupyter kernelspec list
```
```
Available kernels:
  apache_toree_scala    /Users/afranzi/.virtualenvs/mlflow/share/jupyter/kernels/apache_toree_scala
  python3               /Users/afranzi/.virtualenvs/mlflow/share/jupyter/kernels/python3
```

Ut videre potes e libello apposito, UDF inter Scintillam et PySpark communicatur. Speramus hanc partem utilem fore iis qui Scalam amant et apparatus discendi exempla in productione explicandi cupiunt. 

import org.apache.spark.sql.functions.col
import org.apache.spark.sql.types.StructType
import org.apache.spark.sql.{Column, DataFrame}
import scala.util.matching.Regex

val FirstAtRe: Regex = "^_".r
val AliasRe: Regex = "[\s_.:@]+".r

def getFieldAlias(field_name: String): String = {
    FirstAtRe.replaceAllIn(AliasRe.replaceAllIn(field_name, "_"), "")
}

def selectFieldsNormalized(columns: List[String])(df: DataFrame): DataFrame = {
    val fieldsToSelect: List[Column] = columns.map(field =>
        col(field).as(getFieldAlias(field))
    )
    df.select(fieldsToSelect: _*)
}

def normalizeSchema(df: DataFrame): DataFrame = {
    val schema = df.columns.toList
    df.transform(selectFieldsNormalized(schema))
}

FirstAtRe = ^_
AliasRe = [s_.:@]+

getFieldAlias: (field_name: String)String
selectFieldsNormalized: (columns: List[String])(df: org.apache.spark.sql.DataFrame)org.apache.spark.sql.DataFrame
normalizeSchema: (df: org.apache.spark.sql.DataFrame)org.apache.spark.sql.DataFrame
Out[1]:
[s_.:@]+
In [2]:
val winePath = "~/Research/mlflow-workshop/examples/wine_quality/data/winequality-red.csv"
val modelPath = "/tmp/mlflow/artifactStore/0/96cba14c6e4b452e937eb5072467bf79/artifacts/model"

winePath = ~/Research/mlflow-workshop/examples/wine_quality/data/winequality-red.csv
modelPath = /tmp/mlflow/artifactStore/0/96cba14c6e4b452e937eb5072467bf79/artifacts/model
Out[2]:
/tmp/mlflow/artifactStore/0/96cba14c6e4b452e937eb5072467bf79/artifacts/model
In [3]:
val df = spark.read
              .format("csv")
              .option("header", "true")
              .option("delimiter", ";")
              .load(winePath)
              .transform(normalizeSchema)

df = [fixed_acidity: string, volatile_acidity: string ... 10 more fields]
Out[3]:
[fixed_acidity: string, volatile_acidity: string ... 10 more fields]
In [4]:
%%PySpark
import mlflow
from mlflow import pyfunc

model_path = "/tmp/mlflow/artifactStore/0/96cba14c6e4b452e937eb5072467bf79/artifacts/model"
wine_quality_udf = mlflow.pyfunc.spark_udf(spark, model_path)

spark.udf.register("wineQuality", wine_quality_udf)
Out[4]:
<function spark_udf.<locals>.predict at 0x1116a98c8>
In [6]:
df.createOrReplaceTempView("wines")
In [10]:
%%SQL
SELECT 
    quality,
    wineQuality(
        fixed_acidity,
        volatile_acidity,
        citric_acid,
        residual_sugar,
        chlorides,
        free_sulfur_dioxide,
        total_sulfur_dioxide,
        density,
        pH,
        sulphates,
        alcohol
    ) AS prediction
FROM wines
LIMIT 10
Out[10]:
+-------+------------------+
|quality|        prediction|
+-------+------------------+
|      5| 5.576883967129615|
|      5|  5.50664776916154|
|      5| 5.525504822954496|
|      6| 5.504311247097457|
|      5| 5.576883967129615|
|      5|5.5556903912725755|
|      5| 5.467882654744997|
|      7| 5.710602976324739|
|      7| 5.657319539336507|
|      5| 5.345098606538708|
+-------+------------------+

In [17]:
spark.catalog.listFunctions.filter('name like "%wineQuality%").show(20, false)

+-----------+--------+-----------+---------+-----------+
|name       |database|description|className|isTemporary|
+-----------+--------+-----------+---------+-----------+
|wineQuality|null    |null       |null     |true       |
+-----------+--------+-----------+---------+-----------+

Deinde gradibus

Etsi MLflow in versione Alpha est tempore scribendi, satis promissum spectat. Iusta facultas discurrendi multiplex apparatus discendi compages et eas consumendi ab uno termino systemata suadet ad gradum proximum sumit.

Praeterea MLflow addat Engineers Data et Arti- tores Data Scientiae propius in unum, stratum commune inter eos ponendo.

Post hanc explorationem MLflow, confidimus nos progredi et uti ad systema nostra scintilla fistulalia et commendator.

Nice synchronizare tabulam repositionis cum datorum loco tabellae. Hoc nobis debent plures terminos dare, qui eadem tabula repono uti possunt. Exempli gratia: Instantiae plures uti Gloria ΠΈ Athens cum eodem Glu metastore.

Ut summatim dicam, gratias ago tibi MLFlow communitati ut opus nostrum cum notitia magis iucunda reddas.

Si cum MLflow ludit, noli dubitare scribere nobis et narra nobis quomodo ea uteris, ac magis si in productione uteris.

Plura reperies de divisionibus;
Apparatus eruditionis. Basic cursum
Apparatus eruditionis. Provectus cursum

Lege plus:

Source: www.habr.com