Raws li , cov ntaub ntawv engineer tam sim no yog ib txoj haujlwm uas nws qhov kev thov tau loj hlob sai dua li lwm yam. Tus kws tshaj lij cov ntaub ntawv ua lub luag haujlwm tseem ceeb hauv lub koom haum - tsim thiab tswj cov kav dej thiab cov ntaub ntawv uas siv los ua, hloov pauv thiab khaws cov ntaub ntawv. Cov txuj ci dab tsi uas cov neeg sawv cev ntawm txoj haujlwm no xav tau ua ntej? Daim ntawv twg yog qhov txawv ntawm qhov yuav tsum tau ntawm cov ntaub ntawv kws tshawb fawb? Koj yuav kawm txog txhua yam no los ntawm kuv tsab xov xwm.
Kuv tau txheeb xyuas qhov chaw ua haujlwm rau txoj haujlwm ntawm cov ntaub ntawv engineer raws li lawv nyob rau lub Ib Hlis 2020 kom nkag siab txog cov txuj ci thev naus laus zis uas nyiam tshaj plaws. Tom qab ntawd kuv piv cov txiaj ntsig nrog cov txheeb cais ntawm cov neeg ua haujlwm rau cov ntaub ntawv tus kws tshawb fawb txoj haujlwm - thiab qee qhov sib txawv nthuav tawm.
Tsis muaj ntau preamble, ntawm no yog kaum sab saum toj technologies uas tau hais feem ntau hauv kev tshaj tawm txoj haujlwm:
Hais txog cov thev naus laus zis hauv cov neeg ua haujlwm rau txoj haujlwm ntawm cov ntaub ntawv engineer hauv 2020
Cia peb paub tseeb.
Lub luag hauj lwm ntawm ib tug data engineer
Niaj hnub no, cov hauj lwm uas cov ntaub ntawv engineers ua yog ib qho tseem ceeb heev rau cov koom haum - cov no yog cov neeg uas muaj lub luag hauj lwm rau khaws cov ntaub ntawv thiab coj mus rau hauv xws li ib tug daim ntawv uas lwm tus neeg ua hauj lwm ua hauj lwm nrog nws. Cov ntaub ntawv engineers tsim cov raj xa dej los yog cov ntaub ntawv batch los ntawm ntau qhov chaw. Cov kav dej tom qab ntawd ua cov kev rho tawm, hloov pauv, thiab kev thauj khoom (hauv lwm lo lus, cov txheej txheem ETL), ua cov ntaub ntawv tsim nyog rau kev siv ntxiv. Tom qab ntawd, cov ntaub ntawv raug xa mus rau cov kws tshuaj ntsuam thiab cov kws tshawb fawb cov ntaub ntawv rau kev ua kom tob. Thaum kawg, cov ntaub ntawv xaus nws txoj kev taug hauv dashboards, cov ntawv ceeb toom, thiab cov qauv kev kawm tshuab.
Kuv tab tom nrhiav rau cov ntaub ntawv uas yuav tso cai rau kuv los txiav txim siab seb cov thev naus laus zis twg yog qhov xav tau tshaj plaws hauv kev ua haujlwm ntawm cov ntaub ntawv engineer tam sim no.
Txoj kev
Kuv tau sau cov ntaub ntawv los ntawm peb qhov chaw nrhiav haujlwm β , ΠΈ thiab saib cov lus tseem ceeb twg los ua ke nrog "tus kws tshaj lij cov ntaub ntawv" hauv cov ntawv ntawm cov neeg ua haujlwm tsis txaus siab rau cov neeg nyob hauv Asmeskas. Rau txoj haujlwm no kuv siv ob lub tsev qiv ntawv Python β ΠΈ . Ntawm cov ntsiab lus tseem ceeb, kuv suav nrog ob qho tib si uas tau suav nrog hauv daim ntawv teev npe dhau los rau kev tshuaj xyuas qhov chaw ua haujlwm rau txoj haujlwm ntawm tus kws tshawb fawb cov ntaub ntawv, thiab cov uas kuv xaiv los ntawm tus kheej thaum nyeem cov haujlwm muab rau cov ntaub ntawv engineers. LinkedIn tsis suav nrog hauv cov npe ntawm cov peev txheej, txij li kuv tau raug txwv nyob ntawd tom qab kuv sim sau cov ntaub ntawv kawg.
Rau txhua lo lus tseem ceeb, kuv suav qhov feem pua ββββntawm cov hits los ntawm tag nrho cov ntawv ntawm txhua qhov chaw sib cais, thiab tom qab ntawd suav qhov nruab nrab rau peb qhov chaw.
Π Π΅Π·ΡΠ»ΡΡΠ°ΡΡ
Hauv qab no yog peb caug technical data engineering cov ntsiab lus nrog cov qhab nia siab tshaj plaws thoob plaws peb qhov chaw ua haujlwm.
Thiab ntawm no yog tib cov lej, tab sis nthuav tawm hauv daim ntawv:
Wb mus nyob rau hauv kev txiav txim.
Kev tshuaj xyuas cov txiaj ntsig
Ob leeg SQL thiab Python tshwm nyob rau hauv ntau tshaj li ob feem peb ntawm cov haujlwm qhib tau tshuaj xyuas. Nws yog ob lub thev naus laus zis no uas ua rau kev nkag siab los kawm ua ntej. yog ib hom lus programming nrov heev siv rau kev ua haujlwm nrog cov ntaub ntawv, tsim cov vev xaib, thiab sau ntawv. stands rau Structured Query Language; nws suav nrog tus qauv siv los ntawm ib pawg lus thiab siv los muab cov ntaub ntawv los ntawm cov ntaub ntawv sib raug zoo. Nws tshwm sim ntev dhau los thiab tau ua pov thawj nws tus kheej kom tiv taus siab heev.
Spark tau hais nyob rau hauv ib nrab ntawm qhov chaw ua haujlwm. yog "kev koom ua ke cov ntaub ntawv loj analytics engine nrog built-in modules rau streaming, SQL, machine learning, thiab graph processing." Nws yog tshwj xeeb tshaj yog nrov ntawm cov neeg uas ua hauj lwm nrog loj databases.
AWS tshwm hauv kwv yees li 45% ntawm kev tshaj tawm txoj haujlwm. Nws yog huab xam platform tsim los ntawm Amazon; nws muaj kev lag luam loj tshaj plaws ntawm tag nrho cov huab platforms.
Tom ntej no tuaj Java thiab Hadoop - me ntsis ntau dua 40% rau lawv cov kwv tij. yog ib hom lus dav dav, sib ntaus sib tua-tested lus uas tau txais txiaj ntsig thib kaum ntawm cov lus uas ua rau ntshai heev ntawm cov programmers. Hauv qhov sib piv, Python yog ob hom lus nyiam tshaj plaws. Cov lus Java yog khiav los ntawm Oracle, thiab txhua yam koj xav paub txog nws tuaj yeem nkag siab los ntawm qhov screenshot ntawm nplooj ntawv nom tswv txij lub Ib Hlis 2020.
Nws zoo li caij tsheb hauv lub sijhawm
siv MapReduce programming qauv nrog server pawg rau cov ntaub ntawv loj. Tam sim no tus qauv no tau raug tso tseg.
Tom qab ntawd peb pom Hive, Scala, Kafka thiab NoSQL - txhua yam ntawm cov thev naus laus zis no tau hais nyob rau hauv ib lub hlis twg ntawm cov neeg ua haujlwm xa tawm. Apache Hive yog cov ntaub ntawv khaws cia software uas "ua kom yooj yim nyeem, sau, thiab tswj cov ntaub ntawv loj nyob hauv khw muag khoom siv SQL." - ib hom lus programming uas nquag siv thaum ua haujlwm nrog cov ntaub ntawv loj. Tshwj xeeb, Spark tau tsim hauv Scala. Nyob rau hauv qhov uas twb tau hais los qeb duas ntawm cov lus ntshai, Scala qeb thib kaum ib. - lub platform faib rau kev ua cov lus streaming. Nrov heev raws li ib tug txhais tau tias ntawm streaming cov ntaub ntawv.
sib piv lawv tus kheej nrog SQL. Lawv txawv ntawm qhov uas lawv tsis muaj kev sib raug zoo, tsis muaj qauv, thiab kab rov tav scalable. NoSQL tau txais qee qhov muaj koob meej, tab sis qhov craze rau txoj hauv kev, txawm tias mus rau lub ntsiab lus ntawm cov lus faj lem uas nws yuav hloov SQL raws li qhov tseem ceeb cia paradigm, zoo li dhau lawm.
Kev sib piv nrog cov ntsiab lus hauv cov ntaub ntawv kws tshawb fawb tsis txaus
Nov yog peb caug lub ntsiab lus siv tshuab feem ntau ntawm cov ntaub ntawv tshawb fawb cov tswv ntiav haujlwm. Kuv tau txais cov npe no tib yam li tau piav qhia saum toj no rau cov ntaub ntawv engineering.
Kev hais txog kev siv thev naus laus zis hauv cov neeg ua haujlwm rau txoj haujlwm ntawm tus kws tshawb fawb cov ntaub ntawv hauv 2020
Yog hais tias peb tham txog tag nrho cov naj npawb, piv rau yav dhau los xav txog kev nrhiav neeg ua haujlwm, muaj 28% ntau qhov chaw ua haujlwm (12 piv rau 013). Cia peb pom cov thev naus laus zis twg tsawg dua hauv cov chaw haujlwm rau cov kws tshawb fawb cov ntaub ntawv dua li cov ntaub ntawv engineers.
Nrov tshaj nyob rau hauv cov ntaub ntawv engineering
Daim duab hauv qab no qhia cov ntsiab lus nrog qhov sib txawv nruab nrab ntawm ntau dua 10% lossis tsawg dua -10%.
Qhov sib txawv loj tshaj plaws hauv lo lus tseem ceeb ntawm cov ntaub ntawv engineer thiab cov kws tshawb fawb cov ntaub ntawv
AWS qhia qhov tseem ceeb tshaj plaws: hauv cov ntaub ntawv engineering nws zoo li 25% tsis tu ncua ntau dua li hauv cov ntaub ntawv tshawb fawb (kwv yees li 45% thiab 20% ntawm tag nrho cov neeg ua haujlwm, feem). Qhov txawv yog pom!
Ntawm no yog tib cov ntaub ntawv nyob rau hauv ib qho kev nthuav qhia me ntsis - nyob rau hauv daim duab, cov txiaj ntsig rau tib lo lus tseem ceeb hauv qhov chaw ua haujlwm rau txoj haujlwm ntawm cov ntaub ntawv engineer thiab cov ntaub ntawv kws tshawb fawb tau nyob ib sab.
Qhov sib txawv loj tshaj plaws hauv lo lus tseem ceeb ntawm cov ntaub ntawv engineer thiab cov kws tshawb fawb cov ntaub ntawv
Qhov kev dhia loj tshaj plaws tom ntej uas kuv tau sau tseg yog nyob rau hauv Spark - cov ntaub ntawv engineer feem ntau yuav tsum tau ua haujlwm nrog cov ntaub ntawv loj. kuj tau nce los ntawm 20%, uas yog, yuav luag plaub zaug piv rau qhov tshwm sim rau cov ntaub ntawv kws tshawb fawb qhov chaw ua haujlwm. Kev hloov ntaub ntawv yog ib lub luag haujlwm tseem ceeb ntawm cov ntaub ntawv engineer. Thaum kawg, tus naj npawb ntawm cov lus hais yog 15% siab dua hauv kev ua haujlwm ntawm cov ntaub ntawv engineering rau Java, NoSQL, Redshift, SQL thiab Hadoop.
Tsawg nrov hauv cov ntaub ntawv engineering
Tam sim no cia saib seb cov thev naus laus zis twg tsis tshua nyiam hauv cov ntaub ntawv tus kws tshaj lij cov neeg ua haujlwm.
Qhov poob qis tshaj plaws piv rau cov ntaub ntawv tshawb fawb tau tshwm sim hauv : muaj nws tshwm sim nyob rau hauv kwv yees li 56% ntawm qhov chaw ua haujlwm, ntawm no - tsuas yog 17%. Txaus siab heev. R yog ib hom lus programming uas nyiam los ntawm cov kws tshawb fawb thiab cov neeg txheeb cais, thiab yog hom lus thib yim uas ntshai tshaj plaws hauv ntiaj teb.
kuj pom nyob rau hauv cov neeg ua hauj lwm rau txoj hauj lwm ntawm cov ntaub ntawv engineer ho tsis tshua muaj - qhov txawv yog 14%. SAS yog ib hom lus tsim los ua haujlwm nrog kev txheeb cais thiab cov ntaub ntawv. Cov ntsiab lus nthuav dav: txiav txim los ntawm cov txiaj ntsig , nws tau poob ntau hauv av tsis ntev los no - ntau dua li lwm yam thev naus laus zis.
Hauv kev thov hauv cov ntaub ntawv engineering thiab cov ntaub ntawv tshawb fawb
Nws yuav tsum raug sau tseg tias yim ntawm thawj kaum txoj haujlwm hauv ob qho tib si yog tib yam. SQL, Python, Spark, AWS, Java, Hadoop, Hive thiab Scala ua nws mus rau hauv kaum sab saum toj rau ob qho tib si cov ntaub ntawv engineering thiab cov ntaub ntawv tshawb fawb kev lag luam. Hauv daim duab hauv qab no koj tuaj yeem pom kaum tsib lub thev naus laus zis nrov tshaj plaws ntawm cov ntaub ntawv kws tshaj lij cov tswv ntiav haujlwm, thiab tom qab ntawd yog lawv qhov chaw seem rau cov kws tshawb fawb cov ntaub ntawv.
tswv yim pom zoo
Yog tias koj xav nkag mus rau hauv cov ntaub ntawv engineering, kuv xav qhia koj kom paub cov thev naus laus zis hauv qab no - Kuv teev lawv nyob rau hauv qhov kev txiav txim ntawm qhov tseem ceeb.
Kawm SQL. Kuv leaning ntawm PostgreSQL vim nws yog qhov qhib, nrov heev hauv zej zog, thiab nyob rau theem kev loj hlob. Koj tuaj yeem kawm paub siv hom lus los ntawm phau ntawv Kuv Memorable SQL - nws cov qauv tsav tsheb muaj .
Master Python, txawm tias tsis nyob rau theem hardcore tshaj plaws. Kuv Memorable Python yog tsim tshwj xeeb rau cov pib tshiab. Nws tuaj yeem yuav ntawm , luam tawm hluav taws xob lossis lub cev, koj xaiv, lossis rub tawm hauv pdf lossis epub hom .
Thaum koj paub txog Python, txav mus rau pandas, lub tsev qiv ntawv Python uas yog siv rau kev tu thiab ua cov ntaub ntawv. Yog tias koj tab tom npaj ua haujlwm hauv lub tuam txhab uas yuav tsum muaj peev xwm sau ntawv hauv Python (thiab qhov no yog feem ntau ntawm lawv), koj tuaj yeem paub tseeb tias kev paub ntawm pandas yuav raug suav tias yog lub neej ntawd. Tam sim no kuv tab tom ua tiav cov lus qhia qhia ua haujlwm nrog pandas - koj tuaj yeem ua tau kom tsis txhob nco lub sijhawm tso tawm.
Master AWS. Yog tias koj xav dhau los ua tus kws tshaj lij cov ntaub ntawv, koj tsis tuaj yeem ua yam tsis muaj huab platform hauv stash, thiab AWS yog qhov nrov tshaj plaws ntawm lawv. Cov chav kawm tau pab kuv ntau heev thaum kuv kawm , Kuv xav tias lawv kuj yuav muaj cov ntaub ntawv zoo ntawm AWS.
Yog tias koj twb tau ua tiav tag nrho cov npe no thiab xav kom loj hlob ntxiv hauv qhov muag ntawm cov tswv ntiav ua cov ntaub ntawv engineer, kuv xav kom ntxiv Apache Spark rau kev ua haujlwm nrog cov ntaub ntawv loj. Txawm hais tias kuv qhov kev tshawb fawb ntawm cov ntaub ntawv kws tshawb fawb tsis txaus ntseeg tau pom tias muaj kev txaus siab poob qis, ntawm cov ntaub ntawv engineers nws tseem tshwm sim nyob rau hauv yuav luag txhua qhov thib ob.
Thaum kawg
Kuv vam tias koj pom cov ntsiab lus ntawm cov thev naus laus zis hauv kev xav tau tshaj plaws rau cov ntaub ntawv engineers pab tau. Yog tias koj xav paub ntau npaum li cas cov haujlwm kws tshuaj ntsuam xyuas tau nyob deb, nyeem . Zoo siab engineering!
Tau qhov twg los: www.hab.com