Kev Kawm Tshuab Kev Lag Luam: 10 Tus Qauv Tsim Qauv
Niaj hnub no, cov kev pabcuam tshiab, cov ntawv thov thiab lwm cov kev pabcuam tseem ceeb tau tsim txhua hnub uas ua rau nws muaj peev xwm tsim tau yam tsis txaus ntseeg: los ntawm software rau tswj lub foob pob hluav taws SpaceX mus rau kev cuam tshuam nrog lub hwj hauv chav tom ntej ntawm lub xov tooj smartphone.
Thiab, qee zaum, txhua tus neeg ua haujlwm tshiab tshiab, txawm tias nws yog tus pib ua haujlwm siab lossis ib qho zoo tib yam Full Stack lossis Data Scientist, sai lossis tom qab los txog qhov pom tau tias muaj qee yam kev cai rau programming thiab tsim software uas yooj yim rau lub neej.
Hauv tsab xov xwm no, kuv yuav piav qhia luv luv txog 10 lub hauv paus ntsiab lus ntawm yuav ua li cas los tsim kev lag luam tshuab kev kawm kom nws tuaj yeem yooj yim ua ke rau hauv daim ntawv thov / kev pabcuam, raws li 12-factor App methodology. . Kuv lub hom phiaj yog txhawm rau ua kom paub txog cov txheej txheem no, uas tuaj yeem pab ntau tus neeg tsim khoom thiab cov ntaub ntawv tshawb fawb cov neeg.
Kab lus no yog ib qho lus tshaj tawm rau ntau cov kab lus hais txog kev lag luam Machine Learning. Hauv lawv kuv yuav tham ntxiv txog yuav ua li cas thiaj ua tau ib tus qauv thiab tso tawm rau hauv kev tsim khoom, tsim API rau nws, nrog rau cov piv txwv los ntawm ntau qhov chaw thiab cov tuam txhab uas tau tsim-hauv ML hauv lawv lub tshuab.
Txoj Cai 1: Ib lub hauv paus code
Qee tus programmers ntawm thawj theem, tawm ntawm kev tub nkeeg los txiav txim siab nws tawm (lossis rau qee yam ntawm lawv tus kheej), tsis nco qab txog Git. Lawv ib txwm tsis nco qab lo lus, uas yog, lawv pov cov ntaub ntawv rau ib leeg hauv lub tsav / tsuas yog pov ntawv / xa los ntawm nquab, lossis lawv tsis xav txog lawv txoj haujlwm, thiab cog lus rau txhua tus rau lawv tus kheej ceg, thiab tom qab ntawd mus rau lub tus tswv.
Txoj ntsiab cai no hais tias: muaj ib tug codebase thiab ntau deployments.
Git tuaj yeem siv ob qho tib si hauv kev tsim khoom thiab hauv kev tshawb fawb thiab kev tsim kho (R&D), uas nws tsis siv ntau zaus.
Piv txwv li, nyob rau hauv R & D theem koj tuaj yeem tawm hauv kev cog lus nrog cov ntaub ntawv sib txawv thiab cov qauv, thiaj li xaiv qhov zoo tshaj plaws thiab yooj yim ua haujlwm nrog nws ntxiv.
Qhov thib ob, hauv kev tsim khoom no yog qhov tsis tuaj yeem hloov pauv - koj yuav tsum tau saib tas li yuav ua li cas koj cov cai hloov pauv thiab paub tias tus qauv twg tsim cov txiaj ntsig zoo tshaj plaws, tus lej twg ua haujlwm thaum kawg thiab qhov tshwm sim uas ua rau nws tsis ua haujlwm lossis pib tsim cov txiaj ntsig tsis raug. . Qhov ntawd yog dab tsi commits rau!
Koj tuaj yeem tsim ib pob ntawm koj qhov project, tso nws, piv txwv li, ntawm Gemfury, thiab tom qab ntawd tsuas yog xa cov haujlwm los ntawm nws rau lwm qhov haujlwm, thiaj li tsis rov sau dua 1000 zaug, tab sis ntau ntxiv rau tom qab ntawd.
Txoj Cai 2: Qhia meej meej thiab cais cov kev vam meej
Txhua qhov project muaj cov tsev qiv ntawv sib txawv uas koj tuaj yeem los ntawm sab nraud thiaj li siv tau rau qhov chaw. Txawm hais tias nws yog Python cov tsev qiv ntawv, lossis cov tsev qiv ntawv ntawm lwm yam lus rau ntau lub hom phiaj, lossis cov cuab yeej siv - koj txoj haujlwm yog:
- Qhia meej meej txog kev vam khom, uas yog, cov ntaub ntawv uas yuav muaj tag nrho cov tsev qiv ntawv, cov cuab yeej, thiab lawv cov qauv siv hauv koj qhov project thiab yuav tsum tau nruab (piv txwv li, hauv Python qhov no tuaj yeem ua tiav siv Pipfile lossis requirements.txt. txuas uas tso cai rau kom nkag siab zoo: )
- cais cov kev vam meej tshwj xeeb rau koj qhov kev zov me nyuam thaum lub sij hawm txhim kho. Koj tsis xav hloov pauv hloov pauv thiab rov nruab dua, piv txwv li, Tensorflow?
Txoj kev no, cov neeg tsim khoom uas yuav koom nrog koj pab neeg yav tom ntej yuav tuaj yeem paub sai sai nrog cov tsev qiv ntawv thiab lawv cov versions uas tau siv hauv koj qhov project, thiab koj tseem yuav muaj sijhawm los tswj cov versions thiab cov tsev qiv ntawv lawv tus kheej tau teeb tsa rau ib qho tshwj xeeb. project, uas yuav pab tau koj kom tsis txhob incompatibility ntawm cov tsev qiv ntawv los yog lawv versions.
Koj daim ntawv thov yuav tsum tsis txhob cia siab rau cov cuab yeej siv uas yuav raug ntsia rau ntawm OS tshwj xeeb. Cov cuab yeej no kuj yuav tsum tau tshaj tawm nyob rau hauv cov dependencies manifest. Qhov no yog qhov tsim nyog txhawm rau txhawm rau zam cov xwm txheej uas cov cuab yeej version (nrog rau lawv qhov muaj) tsis phim cov cuab yeej siv ntawm ib qho OS.
Yog li, txawm tias curl tuaj yeem siv rau yuav luag txhua lub khoos phis tawj, koj yuav tsum tseem tshaj tawm nws hauv kev vam khom, txij li thaum tsiv mus rau lwm lub platform nws yuav tsis nyob ntawd lossis version yuav tsis yog qhov koj xav tau thaum xub thawj.
Piv txwv li, koj cov tseev kom muaj.txt yuav zoo li no:
# Model Building Requirements
numpy>=1.18.1,<1.19.0
pandas>=0.25.3,<0.26.0
scikit-learn>=0.22.1,<0.23.0
joblib>=0.14.1,<0.15.0
# testing requirements
pytest>=5.3.2,<6.0.0
# packaging
setuptools>=41.4.0,<42.0.0
wheel>=0.33.6,<0.34.0
# fetching datasets
kaggle>=1.5.6,<1.6.0
Txoj Cai 3: Configurations
Ntau tus tau hnov ββββcov dab neeg ntawm ntau tus neeg tsim tawm yuam kev upload code rau GitHub rau hauv cov chaw khaws ntaub ntawv pej xeem nrog cov passwords thiab lwm yam yuam sij los ntawm AWS, sawv ntxov hnub tom qab nrog tus nqi ntawm $ 6000, lossis txawm tias $ 50000.
Tau kawg, cov xwm txheej no yog qhov phem heev, tab sis tseem ceeb heev. Yog tias koj khaws koj cov ntawv pov thawj lossis lwm cov ntaub ntawv xav tau rau kev teeb tsa hauv cov lej, koj ua yuam kev, thiab kuv xav tias tsis tas yuav piav qhia vim li cas.
Ib qho ntxiv rau qhov no yog khaws cov teeb tsa hauv ib puag ncig hloov pauv. Koj tuaj yeem nyeem ntxiv txog qhov hloov pauv ib puag ncig .
Piv txwv ntawm cov ntaub ntawv uas feem ntau khaws cia hauv ib puag ncig hloov pauv:
- Cov npe
- API URLs/URIs
- Public thiab private keys
- Tiv tauj (mail, xov tooj, thiab lwm yam)
Txoj kev no koj tsis tas yuav hloov cov cai tas li yog tias koj qhov kev hloov pauv hloov pauv. Qhov no yuav pab txuag koj lub sijhawm, kev siv zog thiab nyiaj txiag.
Piv txwv li, yog tias koj siv Kaggle API los ua cov kev xeem (piv txwv li, rub tawm cov software thiab khiav cov qauv los ntawm nws los ntsuas thaum khiav tias tus qauv ua haujlwm zoo), ces cov yuam sij ntiag tug los ntawm Kaggle, xws li KAGGLE_USERNAME thiab KAGGLE_KEY, yuav tsum yog khaws cia rau hauv ib puag ncig hloov pauv.
Txoj Cai 4: Kev Pabcuam Thib Peb
Lub tswv yim ntawm no yog los tsim cov kev pab cuam nyob rau hauv xws li ib txoj kev uas tsis muaj qhov sib txawv ntawm cov khoom siv hauv zos thiab cov neeg thib peb ntawm cov cai. Piv txwv li, koj tuaj yeem txuas ob lub zos MySQL thiab cov neeg thib peb. Tib yam mus rau ntau yam APIs xws li Google Maps lossis Twitter API.
Txhawm rau kom tsis txhob muaj kev pabcuam thib peb lossis txuas rau lwm tus, koj tsuas yog yuav tsum hloov cov yuam sij hauv kev teeb tsa hauv ib puag ncig hloov pauv, uas kuv tau tham txog hauv kab lus saum toj no.
Yog li, piv txwv li, es tsis txhob qhia txoj hauv kev rau cov ntaub ntawv nrog cov ntaub ntawv hauv cov lej txhua zaus, nws yog qhov zoo dua los siv lub tsev qiv ntawv pathlib thiab tshaj tawm txoj hauv kev rau cov ntaub ntawv hauv config.py, yog li tsis muaj teeb meem dab tsi koj siv (rau Piv txwv li, CircleCI), qhov kev zov me nyuam muaj peev xwm nrhiav tau txoj hauv kev mus rau cov ntaub ntawv khaws cia rau hauv tus account tus qauv ntawm cov ntaub ntawv tshiab hauv cov kev pabcuam tshiab.
Txoj Cai 5. Tsim, tso tawm, khiav sijhawm
Ntau tus neeg hauv Data Science pom tias nws muaj txiaj ntsig los txhim kho lawv cov kev txawj sau ntawv software. Yog tias peb xav kom peb qhov kev pab cuam poob qis npaum li qhov ua tau thiab ua haujlwm yam tsis muaj kev ua haujlwm ntev li ntev tau, peb yuav tsum tau faib cov txheej txheem ntawm kev tso tawm tshiab rau hauv 3 theem:
- Theem cov rooj sib txoos. Koj hloov koj cov lej liab qab nrog cov peev txheej ntawm tus kheej rau hauv lub npe hu ua pob uas muaj tag nrho cov lej tsim nyog thiab cov ntaub ntawv. Lub pob no hu ua kev sib dhos.
- Theem tso tawm - ntawm no peb txuas peb config rau lub rooj sib txoos, yam tsis muaj qhov uas peb yuav tsis muaj peev xwm tso tawm peb qhov kev pab cuam. Tam sim no qhov no yog ib qho kev npaj txhij-rau-launch tso tawm.
- Tom ntej no los ntawm theem ua kom tiav. Ntawm no peb tso daim ntawv thov los ntawm kev khiav cov txheej txheem tsim nyog los ntawm peb qhov kev tso tawm.
Xws li cov txheej txheem rau kev tso tawm cov qauv tshiab los yog tag nrho cov kav dej tso cai rau koj cais lub luag haujlwm ntawm cov thawj coj thiab cov tsim tawm, tso cai rau koj taug qab cov qauv thiab tiv thaiv qhov tsis xav tau ntawm qhov kev zov me nyuam.
Rau txoj haujlwm tso tawm, ntau cov kev pabcuam sib txawv tau tsim nyob rau hauv uas koj tuaj yeem sau cov txheej txheem los khiav koj tus kheej hauv cov ntaub ntawv .yml (piv txwv li, hauv CircleCI qhov no yog config.yml los txhawb cov txheej txheem nws tus kheej). Wheely yog qhov zoo ntawm kev tsim cov pob khoom rau cov haujlwm.
Koj tuaj yeem tsim cov pob khoom sib txawv ntawm koj lub tshuab kev kawm qauv, thiab tom qab ntawd ntim lawv thiab xa mus rau cov pob tsim nyog thiab lawv cov qauv siv cov haujlwm uas koj tau sau los ntawm qhov ntawd. Qhov no yuav pab koj tsim API rau koj tus qauv, thiab koj lub pob tuaj yeem tuav ntawm Gemfury, piv txwv li.
Txoj Cai 6. Khiav koj tus qauv ua ib lossis ntau txheej txheem
Ntxiv mus, cov txheej txheem yuav tsum tsis txhob muaj cov ntaub ntawv sib koom. Ntawd yog, cov txheej txheem yuav tsum muaj nyob nyias, thiab txhua yam ntaub ntawv yuav tsum muaj nyob nyias, piv txwv li, ntawm cov kev pabcuam thib peb xws li MySQL lossis lwm tus, nyob ntawm seb koj xav tau dab tsi.
Ntawd yog, nws tsis tsim nyog khaws cia cov ntaub ntawv hauv cov txheej txheem cov ntaub ntawv, txwv tsis pub qhov no tuaj yeem ua rau tshem tawm cov ntaub ntawv no thaum lub sijhawm tso tawm tom ntej / hloov pauv ntawm kev teeb tsa lossis hloov pauv ntawm lub kaw lus uas lub program khiav.
Tab sis muaj ib qho kev zam: rau cov kev kawm tshuab, koj tuaj yeem khaws lub cache ntawm cov tsev qiv ntawv kom tsis txhob rov nruab lawv txhua zaus koj tso tawm cov ntawv tshiab, yog tias tsis muaj cov tsev qiv ntawv ntxiv lossis hloov pauv rau lawv cov versions. Txoj kev no, koj yuav txo tau lub sijhawm nws yuav siv los tsim koj tus qauv hauv kev lag luam.
Txhawm rau khiav tus qauv raws li ob peb txheej txheem, koj tuaj yeem tsim cov ntaub ntawv .yml uas koj qhia meej txog cov txheej txheem tsim nyog thiab lawv cov kab ke.
Txoj Cai 7: Recyclability
Cov txheej txheem uas khiav hauv koj daim ntawv thov qauv yuav tsum yooj yim pib thiab nres. Yog li, qhov no yuav tso cai rau koj kom xa cov lej hloov pauv sai, hloov pauv kev teeb tsa, nrawm nrawm thiab hloov pauv tau yooj yim, thiab tiv thaiv kev ua txhaum ntawm cov haujlwm ua haujlwm.
Ntawd yog, koj cov txheej txheem nrog tus qauv yuav tsum:
- Minimize lub sij hawm pib. Qhov zoo tshaj plaws, lub sijhawm pib (txij li lub sijhawm pib cov lus txib tau muab rau lub sijhawm cov txheej txheem los ua haujlwm) yuav tsum tsis pub ntau tshaj li ob peb feeb. Cov tsev qiv ntawv caching, piav qhia saum toj no, yog ib txoj hauv kev los txo lub sijhawm pib.
- Xaus kom raug. Ntawd yog, kev mloog ntawm qhov chaw nres nkoj qhov kev pabcuam raug tshem tawm tiag tiag, thiab cov ntawv thov tshiab xa mus rau qhov chaw nres nkoj no yuav tsis ua tiav. Ntawm no koj yuav tsum teeb tsa kev sib txuas lus zoo nrog DevOps engineers, lossis nkag siab tias nws ua haujlwm li cas koj tus kheej (zoo dua, tau kawg, tom kawg, tab sis kev sib txuas lus yuav tsum tau khaws cia, hauv txhua qhov project!)
Txoj Cai 8: Kev xa mus tas li / kev sib koom ua ke
Ntau lub tuam txhab siv kev sib cais ntawm kev txhim kho daim ntawv thov thiab pab pawg xa tawm (ua rau daim ntawv thov muaj rau cov neeg siv kawg). Qhov no tuaj yeem ua rau qeeb qeeb software txhim kho thiab nce qib hauv kev txhim kho nws. Nws kuj ua rau DevOps kab lis kev cai, qhov twg kev txhim kho thiab kev sib koom ua ke, kwv yees hais lus, ua ke.
Yog li ntawd, lub hauv paus ntsiab lus no hais tias koj ib puag ncig kev loj hlob yuav tsum nyob ze li sai tau rau koj qhov chaw tsim khoom.
Qhov no yuav tso cai rau:
- Txo lub sij hawm tso tawm los ntawm kaum tawm zaus
- Txo tus naj npawb ntawm qhov yuam kev vim yog cov lej tsis sib xws.
- Qhov no kuj txo cov haujlwm ntawm cov neeg ua haujlwm, txij li cov neeg tsim khoom thiab cov neeg siv daim ntawv thov tam sim no yog ib pab neeg.
Cov cuab yeej uas tso cai rau koj ua haujlwm nrog qhov no yog CircleCI, Travis CI, GitLab CI thiab lwm yam.
Koj tuaj yeem ua ntxiv rau tus qauv sai sai, hloov kho nws, thiab tso tawm tam sim ntawd, thaum nws yuav yooj yim, nyob rau hauv cov ntaub ntawv ntawm kev ua tsis tiav, rov qab sai heev rau qhov ua haujlwm version, kom cov neeg siv kawg tsis txawm pom nws. Qhov no tuaj yeem ua tau tshwj xeeb yog yooj yim thiab sai yog tias koj muaj kev sim zoo.
Txo qhov sib txawv !!!
Txoj Cai 9. Koj cov ntaub ntawv
Logs (lossis "Log") yog cov xwm txheej, feem ntau sau rau hauv cov ntawv nyeem, uas tshwm sim hauv daim ntawv thov (kev tshwm sim kwj). Ib qho piv txwv yooj yim: "2020-02-02 - system theem - txheej txheem npe." Lawv tau tsim los kom tus tsim tawm tuaj yeem pom qhov tshwm sim thaum qhov kev zov me nyuam tab tom khiav. Nws pom kev nce qib ntawm cov txheej txheem thiab nkag siab seb nws puas yog raws li tus tsim tawm nws tus kheej npaj.
Lub hauv paus ntsiab lus no hais tias koj yuav tsum tsis txhob khaws koj cov cav hauv koj cov ntaub ntawv kaw lus - koj yuav tsum tsuas yog "tso tawm" lawv mus rau qhov screen, piv txwv li, ua qhov no ntawm qhov system tus qauv tso tawm. Thiab nyob rau hauv txoj kev no nws yuav muaj peev xwm saib xyuas qhov ntws hauv lub davhlau ya nyob twg thaum lub sijhawm txhim kho.
Qhov no puas txhais tau tias tsis tas yuav txuag cov cav txhua? Tau kawg tsis. Koj daim ntawv thov tsuas yog yuav tsum tsis txhob ua qhov no - cia nws mus rau cov kev pabcuam thib peb. Koj daim ntawv thov tsuas tuaj yeem xa mus rau cov ntaub ntawv tshwj xeeb lossis lub davhlau ya nyob twg rau kev saib lub sijhawm tiag tiag, lossis xa mus rau lub hom phiaj tseem ceeb ntawm cov ntaub ntawv khaws cia (xws li Hadoop). Koj daim ntawv thov nws tus kheej yuav tsum tsis txhob khaws lossis cuam tshuam nrog cov cav.
Txoj Cai 10. Kuaj!
Rau kev kawm tshuab kev lag luam, theem no tseem ceeb heev, vim koj yuav tsum nkag siab tias tus qauv ua haujlwm kom raug thiab ua qhov koj xav tau.
Kev ntsuam xyuas tuaj yeem tsim tau siv pytest, thiab sim siv cov ntaub ntawv me me yog tias koj muaj kev rov ua haujlwm / kev faib tawm.
Tsis txhob hnov ββββqab teeb tsa tib lub noob rau cov qauv kev kawm tob kom lawv tsis tas yuav tsim cov txiaj ntsig sib txawv.
Qhov no yog ib qho kev piav qhia luv luv ntawm 10 lub hauv paus ntsiab lus, thiab, ntawm chav kawm, nws yog ib qho nyuaj rau siv lawv yam tsis tau sim thiab pom lawv ua haujlwm li cas, yog li tsab xov xwm no tsuas yog cov lus tshaj tawm rau cov kab lus nthuav dav uas kuv yuav qhia yuav ua li cas tsim. industrial machine learning qauv, yuav ua li cas koom ua ke rau hauv systems, thiab yuav ua li cas cov ntsiab cai no yuav ua rau lub neej yooj yim dua rau peb txhua tus.
Kuv tseem yuav sim siv cov ntsiab cai txias uas leej twg tuaj yeem tawm hauv cov lus yog tias lawv xav tau.
Tau qhov twg los: www.hab.com