Lero tikusindikiza gawo lachiwiri la kumasulira kwazinthu za momwe Dropbox adakonzera kuwongolera kwamtundu wa mizere mamiliyoni angapo a Python code.
β
Thandizo lamtundu wovomerezeka (PEP 484)
Tidachita zoyeserera zathu zazikulu ndi mypy pa Dropbox pa Hack Week 2014. Sabata ya Hack ndi chochitika cha sabata limodzi chochitidwa ndi Dropbox. Panthawi imeneyi, ogwira ntchito akhoza kugwira ntchito iliyonse yomwe akufuna! Zina mwazinthu zodziwika bwino zaukadaulo za Dropbox zidayamba pazochitika ngati izi. Chifukwa cha kuyesaku, tatsimikiza kuti mypy ikuwoneka yodalirika, ngakhale kuti polojekitiyi sinakonzekere kugwiritsidwa ntchito ponseponse.
Panthawiyo, lingaliro la kuyimilira machitidwe amtundu wa Python linali mlengalenga. Monga ndanenera, popeza Python 3.0 zinali zotheka kugwiritsa ntchito zofotokozera zamtundu wa ntchito, koma awa anali mawu osamveka, opanda mawu ofotokozera ndi semantics. Pakukhazikitsa pulogalamu, zofotokozera izi, makamaka, zidangonyalanyazidwa. Pambuyo kuthyolako Sabata, tinayamba ntchito yokhazikika semantics. Ntchito imeneyi inachititsa kuti ziwonekere (Guido van Rossum, Εukasz Langa ndi ine tinagwirizana nawo pachikalatachi).
Zolinga zathu tingazione mbali ziwiri. Choyamba, tinkayembekezera kuti chilengedwe chonse cha Python chikhoza kutengera njira yofanana yogwiritsira ntchito zizindikiro zamtundu (mawu ogwiritsidwa ntchito mu Python monga ofanana ndi "mawu ofotokozera"). Izi, poganizira zoopsa zomwe zingatheke, zingakhale bwino kusiyana ndi kugwiritsa ntchito njira zambiri zosagwirizana. Chachiwiri, tinkafuna kukambirana momasuka njira zofotokozera ndi anthu ambiri amgulu la Python. Chikhumbo ichi chinatsimikiziridwa ndi mfundo yakuti sitingafune kuoneka ngati "ampatuko" kuchokera ku malingaliro oyambirira a chinenero pamaso pa anthu ambiri a Python programmers. Ndichilankhulidwe chodziwika bwino chomwe chimatchedwa "kulemba bakha". M'deralo, pachiyambi penipeni, malingaliro okayikitsa pa lingaliro la typing typing sangathandize koma kuwuka. Koma malingaliro amenewo adazilala pambuyo pake zitadziwika kuti kuyimba mokhazikika sikukhala kokakamiza (ndipo anthu atazindikira kuti kunali kothandiza).
Kalankhulidwe ka mtundu womwe udakhazikitsidwa pambuyo pake unali wofanana kwambiri ndi zomwe mypy idathandizira panthawiyo. PEP 484 idatulutsidwa ndi Python 3.5 mu 2015. Python sinalinso chilankhulo chojambulidwa mwamphamvu. Ndimakonda kuganiza za chochitikachi ngati chochitika chofunikira kwambiri m'mbiri ya Python.
Kuyamba kwa kusamuka
Kumapeto kwa 2015, Dropbox adapanga gulu la anthu atatu kuti agwire ntchito pa mypy. Anaphatikizapo Guido van Rossum, Greg Price ndi David Fisher. Kuyambira nthawi imeneyo, zinthu zinayamba kukula mofulumira kwambiri. Cholepheretsa choyamba kukula kwa mypy chinali kuchita. Monga ndanenera pamwambapa, m'masiku oyambirira a polojekitiyi ndinaganiza zomasulira kukhazikitsidwa kwa mypy kukhala C, koma lingaliro ili linadutsa pamndandanda pano. Tinakakamira ndikuyendetsa dongosololi pogwiritsa ntchito womasulira wa CPython, yemwe sali wothamanga mokwanira pazida ngati mypy. (Pulojekiti ya PyPy, kukhazikitsa kwina kwa Python ndi JIT compiler, sikunatithandizenso.)
Mwamwayi, zosintha zina za algorithmic zabwera kutithandiza pano. "Accelerator" yoyamba yamphamvu inali kukhazikitsa kuwunika kowonjezereka. Lingaliro lakuwongolera uku linali losavuta: ngati kudalira kwa module sikunasinthe kuyambira pomwe mypy idayamba, ndiye kuti titha kugwiritsa ntchito zomwe zidasungidwa panthawi yapitayi tikugwira ntchito ndi zodalira. Tinkangofunikira kuyang'ana mtundu pamafayilo osinthidwa komanso pamafayilo omwe amadalira iwo. Mypy adapitanso patsogolo pang'ono: ngati mawonekedwe akunja a module sanasinthe, mypy adaganiza kuti ma module ena omwe adatumiza gawoli sanafunikire kufufuzidwanso.
Kuwunika kowonjezereka kwatithandiza kwambiri pofotokozera ma code ambiri omwe alipo. Chowonadi ndi chakuti njirayi nthawi zambiri imaphatikizapo maulendo ambiri obwerezabwereza a mypy monga ndemanga zimawonjezeredwa pang'onopang'ono ku code ndikuwongolera pang'onopang'ono. Kuthamanga koyamba kwa mypy kunali kochedwa kwambiri chifukwa kunali ndi zodalira zambiri kuti muwone. Kenako, kuti zinthu ziyende bwino, tidagwiritsa ntchito makina osungira akutali. Ngati mypy iwona kuti cache yakomweko ikuyenera kukhala yachikale, imatsitsa chithunzithunzi chaposachedwa cha codebase yonse kuchokera kunkhokwe yapakati. Kenako imachita cheke chowonjezera pogwiritsa ntchito chithunzithunzi ichi. Izi zatitengera gawo limodzi lalikulu pakukulitsa magwiridwe antchito a mypy.
Iyi inali nthawi yotengera mwachangu komanso mwachilengedwe kuyang'ana mtundu ku Dropbox. Pofika kumapeto kwa 2016, tinali kale ndi mizere pafupifupi 420000 ya Python code yokhala ndi zolemba zamtundu. Ogwiritsa ntchito ambiri anali ndi chidwi ndi kufufuza kwamtundu. Magulu ochulukirachulukira adagwiritsa ntchito Dropbox mypy.
Zonse zinkaoneka bwino panthawiyo, komabe tinali ndi zambiri zoti tichite. Tinayamba kuchita kafukufuku wamkati mkati mwanthawi ndi nthawi kuti tidziwe madera omwe ali ndi vuto la polojekitiyi ndikumvetsetsa zomwe ziyenera kuthetsedwa kaye (mchitidwewu ukugwiritsidwabe ntchito pakampani lero). Zofunika kwambiri, monga zinawonekera, zinali ntchito ziwiri. Choyamba, tinkafunika kufalitsa mtundu wina wa code, chachiwiri, timafunikira mypy kuti igwire ntchito mofulumira. Zinali zoonekeratu kuti ntchito yathu yofulumizitsa mypy ndikuigwiritsa ntchito m'mapulojekiti akampani inali isanathe. Ife, podziwa bwino za kufunika kwa ntchito ziwirizi, tinayamba kuzithetsa.
Zochuluka!
Macheke owonjezera adapanga mypy mwachangu, koma chida sichinali chachangu mokwanira. Macheke ambiri adatenga pafupifupi miniti imodzi. Chifukwa cha ichi chinali cyclical imports. Izi mwina sizingadabwitse aliyense yemwe wagwirapo ntchito ndi ma codebases akulu olembedwa ku Python. Tinali ndi mazana a ma modules, iliyonse yomwe inkatumiza kunja kwina kulikonse. Ngati fayilo iliyonse muzolowera zolowetsa idasinthidwa, mypy idayenera kukonza mafayilo onse mu loop, ndipo nthawi zambiri ma module omwe amalowetsamo ma module kuchokera ku loop. Kuzungulira kumodzi kotereku kunali "kudalirana" koyipa komwe kudayambitsa mavuto ambiri pa Dropbox. Kapangidwe kameneka kamakhala ndi ma module mazana angapo, pomwe adatumizidwa kunja, mwachindunji kapena mwanjira ina, mayeso ambiri, adagwiritsidwanso ntchito pamakhodi opanga.
Tinalingalira za kuthekera kwa "kumasula" kudalira kozungulira, koma tinalibe zinthu zochitira izo. Panali ma code ochuluka kwambiri omwe sitinkawadziwa. Zotsatira zake, tinabwera ndi njira ina. Tinasankha kupanga mypy ntchito mwamsanga ngakhale pamaso pa "kudalira tangles". Tidakwaniritsa cholinga ichi pogwiritsa ntchito mypy daemon. Daemon ndi njira ya seva yomwe imagwiritsa ntchito zinthu ziwiri zosangalatsa. Choyamba, imasunga zambiri za codebase yonse mu kukumbukira. Izi zikutanthauza kuti nthawi iliyonse mukamayendetsa mypy, simuyenera kuyika deta yosungidwa yokhudzana ndi zikwizikwi zomwe zimatumizidwa kunja. Kachiwiri, iye mosamala, pamlingo wa mayunitsi ang'onoang'ono, amasanthula kudalira pakati pa ntchito ndi mabungwe ena. Mwachitsanzo, ngati ntchito foo kuyitana ntchito bar, ndiye pali kudalira foo ΠΎΡ bar. Fayilo ikasintha, daemon poyamba, payokha, imangosintha fayilo yosinthidwa. Kenako imayang'ana zosintha zowonekera kunja kwa fayiloyo, monga masiginecha osinthidwa. Daemon imagwiritsa ntchito zambiri zazomwe zimatumizidwa kunja kuti ziwonenso kawiri ntchito zomwe zimagwiritsa ntchito zomwe zasinthidwa. Kawirikawiri, ndi njirayi, muyenera kuyang'ana ntchito zochepa kwambiri.
Kukwaniritsa zonsezi sikunali kophweka, popeza kukhazikitsa koyambirira kwa mypy kunali koyang'ana kwambiri pakukonza fayilo imodzi panthawi imodzi. Tinayenera kuthana ndi zochitika zambiri zam'malire, zomwe zimachitika zomwe zimafuna kufufuza mobwerezabwereza pamene chinachake chinasintha mu code. Mwachitsanzo, izi zimachitika pamene kalasi yapatsidwa kalasi yatsopano yoyambira. Titachita zomwe timafuna, tinatha kuchepetsa nthawi yopangira macheke owonjezera kukhala masekondi ochepa chabe. Zimenezi zinkaoneka ngati chipambano chachikulu kwa ife.
Kuchulukirachulukira!
Pamodzi ndi caching yakutali yomwe ndidakambirana pamwambapa, mypy daemon pafupifupi idathetsa mavuto omwe amayamba pomwe wopanga mapulogalamu nthawi zambiri amayendetsa kuwunika kwamtundu, kupanga zosintha pamafayilo ochepa. Komabe, kachitidwe kachitidwe pakagwiritsidwe kocheperako kamene kanali kocheperako. Kuyambitsa koyera kwa mypy kumatha kutenga mphindi 15. Ndipo izi zinali zochuluka kuposa momwe tikadakondwera nazo. Sabata iliyonse zinthu zidayamba kuipiraipira pomwe opanga mapulogalamu adapitilizabe kulemba ma code atsopano ndikuwonjezera ndemanga pama code omwe analipo kale. Ogwiritsa ntchito anali adakali ndi njala yowonjezereka, koma tinali okondwa kukumana nawo theka.
Tinaganiza zobwereranso ku imodzi mwamalingaliro akale okhudza mypy. Momwemo, kusinthira Python code kukhala C code. Kuyesa Cython (kachitidwe kamene kamakulolani kumasulira kachidindo kolembedwa mu Python kukhala C code) sikunatipatse liwiro lowoneka, kotero tinaganiza zotsitsimutsa lingaliro lolemba wolemba wathu tokha. Popeza codebase ya mypy (yolembedwa mu Python) inali kale ndi zolemba zonse zofunika, tinaganiza kuti zingakhale zopindulitsa kuyesa kugwiritsa ntchito mawuwa kuti mufulumizitse dongosolo. Mwamsanga ndinapanga prototype kuyesa lingaliro ili. Idawonetsa kuwonjezeka kopitilira 10 pakuchita ma benchmarks osiyanasiyana. Lingaliro lathu linali loti tipange ma module a Python kukhala ma module a C pogwiritsa ntchito Cython, ndikusintha zolemba zamtundu kukhala macheke amtundu wa nthawi (nthawi zambiri zofotokozera sizimaganiziridwa panthawi yothamanga ndipo zimagwiritsidwa ntchito ndi makina owunika). Tidakonzadi kumasulira kukhazikitsidwa kwa mypy kuchokera ku Python kupita kuchilankhulo chomwe chidapangidwa kuti chilembedwe mokhazikika, chomwe chimawoneka (ndipo, makamaka, chimagwira ntchito) chimodzimodzi ngati Python. (Kusamuka kwamtundu uwu kwakhala chikhalidwe cha ntchito ya mypy. Kukhazikitsa koyambirira kwa mypy kunalembedwa mu Alore, ndiye panali wosakanizidwa wa Java ndi Python).
Kuyang'ana pa API yowonjezera ya CPython kunali kofunika kwambiri kuti musataye luso la kayendetsedwe ka polojekiti. Sitinafunikire kugwiritsa ntchito makina enieni kapena malaibulale aliwonse omwe mypy amafunikira. Kuphatikiza apo, tikadakhalabe ndi mwayi wopeza chilengedwe chonse cha Python ndi zida zonse (monga pytest). Izi zikutanthauza kuti tikhoza kupitiriza kugwiritsa ntchito kachidindo ka Python yotanthauziridwa panthawi yachitukuko, kutilola kuti tipitirize kugwira ntchito ndi ndondomeko yofulumira kwambiri yopangira ma code ndi kuyesa, m'malo modikirira kuti codeyo ipangidwe. Zinkawoneka ngati tikuchita ntchito yaikulu yokhala pamipando iwiri, kunena kwake titero, ndipo tinaikonda.
Wopangayo, yemwe tidatcha kuti mypyc (popeza amagwiritsa ntchito mypy ngati kumapeto kwa kusanthula mitundu), idakhala projekiti yopambana kwambiri. Ponseponse, tapeza pafupifupi 4x liwiro la kuthamanga kwa mypy pafupipafupi popanda caching. Kupanga maziko a polojekiti ya mypyc kunatenga gulu laling'ono la Michael Sullivan, Ivan Levkivsky, Hugh Hahn, ndi ineyo pafupifupi miyezi inayi ya kalendala. Ntchitoyi inali yaying'ono kwambiri kuposa zomwe zikanafunika kuti mulembenso mypy, mwachitsanzo, mu C ++ kapena Go. Ndipo tinafunika kusintha zinthu zocepa kuposa mmene tikanafunika kupanga polembanso mβcinenelo cina. Tinkayembekezanso kuti titha kubweretsa mypyc pamlingo woti okonza mapulogalamu ena a Dropbox atha kuugwiritsa ntchito popanga ndikufulumizitsa ma code awo.
Kuti tikwaniritse ntchitoyi, tidayenera kugwiritsa ntchito mayankho osangalatsa a uinjiniya. Choncho, wosonkhanitsa akhoza kufulumizitsa ntchito zambiri pogwiritsa ntchito zomangamanga zofulumira, zotsika kwambiri za C. Mwachitsanzo, kuyitana kophatikizana kumatanthauziridwa kukhala C. Ndipo kuyitana koteroko kumathamanga kwambiri kuposa kuyitana ntchito yotanthauziridwa. Ntchito zina, monga kusanthula mtanthauzira mawu, zimakhudzidwabe kugwiritsa ntchito mafoni anthawi zonse a C-API ochokera ku CPython, omwe anali othamanga pang'ono akapangidwa. Tinatha kuthetsa katundu wowonjezera pa dongosolo lopangidwa ndi kutanthauzira, koma izi zinangopereka phindu laling'ono pochita ntchito.
Kuti tidziwe zomwe zimachitika kwambiri "zapang'onopang'ono", tidalemba mbiri yakale. Pokhala ndi izi, tidayesa kuyika mypyc kuti ipange kachidindo ka C mwachangu pazochita zotere, kapena kulembanso nambala yofananira ya Python pogwiritsa ntchito magwiridwe antchito mwachangu (ndipo nthawi zina tinalibe yankho losavuta la izi kapena vuto lina) . Kulembanso kachidindo ka Python nthawi zambiri kunali njira yosavuta yothetsera vutoli kusiyana ndi kukhala ndi compiler yokha kusintha komweko. M'kupita kwa nthawi, tinkafuna kupanga masinthidwe ambiri, koma panthawiyo tinkangoyang'ana kwambiri kufulumizitsa mypy ndi khama lochepa. Ndipo popita ku cholinga ichi, timadula ngodya zingapo.
Zipitilizidwaβ¦
Wokondedwa owerenga! Kodi mumawona bwanji za polojekiti ya mypy mutamva kuti ilipo?
Source: www.habr.com