Dab tsi tuaj yeem cuam tshuam nrog Data Science? Cov ntaub ntawv khaws tseg

Niaj hnub no muaj 100500 Data Science cov chav kawm thiab nws tau ntev tau paub tias cov nyiaj feem ntau hauv Data Science tuaj yeem khwv tau los ntawm cov chav kawm Data Science (vim li cas khawb thaum koj tuaj yeem muag duav?). Lub ntsiab tsis zoo ntawm cov kev kawm no yog tias lawv tsis muaj dab tsi ua nrog kev ua haujlwm tiag tiag: tsis muaj leej twg yuav muab koj cov ntaub ntawv huv, ua tiav hauv hom ntawv xav tau. Thiab thaum koj tawm hauv chav kawm thiab pib daws qhov teeb meem tiag tiag, ntau nuances tshwm sim.

Yog li ntawd, peb tab tom pib sau ntawv "Dab tsi tuaj yeem ua yuam kev nrog Data Science", raws li cov xwm txheej tiag tiag uas tshwm sim rau kuv, kuv cov phooj ywg thiab cov npoj yaig. Peb yuav txheeb xyuas cov haujlwm Data Science ib txwm siv cov piv txwv tiag tiag: qhov no tshwm sim li cas. Cia peb pib hnub no nrog txoj haujlwm sau cov ntaub ntawv.

Thiab thawj qhov uas tib neeg ntog thaum lawv pib ua haujlwm nrog cov ntaub ntawv tiag tiag yog sau cov ntaub ntawv no uas cuam tshuam rau peb. Cov lus tseem ceeb ntawm kab lus no:

Peb ua tib zoo saib xyuas lub sijhawm, cov peev txheej, thiab kev siv zog uas yuav tsum tau sau, ntxuav, thiab npaj cov ntaub ntawv.

Thiab qhov tseem ceeb tshaj plaws, peb yuav tham txog yuav ua li cas los tiv thaiv qhov no.

Raws li ntau yam kev kwv yees, kev tu, kev hloov pauv, kev ua cov ntaub ntawv, kev tsim vaj tsev, thiab lwm yam siv 80-90% ntawm lub sijhawm, thiab kev tshuaj xyuas 10-20%, thaum yuav luag tag nrho cov ntaub ntawv kawm tshwj xeeb rau kev tsom xam.

Cia peb saib qhov teeb meem yooj yim analytical nyob rau hauv peb lub versions raws li ib tug qauv piv txwv thiab saib dab tsi "ua rau mob hnyav" yog dab tsi.

Thiab ua piv txwv, dua, peb yuav xav txog cov kev hloov pauv zoo sib xws ntawm txoj haujlwm ntawm kev sau cov ntaub ntawv thiab sib piv cov zej zog rau:

1. Ob Reddit subreddits
2. Ob ntu ntawm Habr
3. Ob pawg ntawm Odnoklassniki

Conditional mus kom ze nyob rau hauv txoj kev xav

Qhib lub vev xaib thiab nyeem cov piv txwv, yog tias nws pom tseeb, tso tseg ob peb teev rau kev nyeem ntawv, ob peb teev rau cov cai siv cov piv txwv thiab kev debugging. Ntxiv ob peb teev rau kev sau. Muab pov rau hauv ob peb teev nyob rau hauv cia (sib npaug los ntawm ob thiab ntxiv N teev).

Cov ntsiab lus tseem ceeb: Lub sij hawm kwv yees yog nyob ntawm qhov kev xav thiab kev kwv yees txog ntev npaum li cas nws yuav siv sijhawm.

Nws yog ib qho tsim nyog los pib tshawb xyuas lub sij hawm los ntawm kev kwv yees cov kev ntsuas hauv qab no rau cov teeb meem raws li tau piav qhia saum toj no:

• Qhov loj ntawm cov ntaub ntawv yog dab tsi thiab ntau npaum li cas ntawm nws yuav tsum tau sau lub cev (*saib hauv qab*).
• Lub sijhawm sau rau ib cov ntaub ntawv yog dab tsi thiab koj yuav tsum tau tos ntev npaum li cas ua ntej koj tuaj yeem sau qhov thib ob?
• Xav txog kev sau cov lej uas txuag lub xeev thiab pib rov pib dua thaum (tsis yog) txhua yam ua tsis tiav.
• Txheeb xyuas seb peb puas xav tau kev tso cai thiab teem sijhawm kom tau txais kev nkag los ntawm API.
• Teem tus naj npawb ntawm qhov yuam kev raws li kev ua haujlwm ntawm cov ntaub ntawv nyuaj - ntsuas rau ib txoj haujlwm tshwj xeeb: cov qauv, pes tsawg qhov kev hloov pauv, dab tsi thiab yuav rho tawm li cas.
• Txhim kho network yuam kev thiab teeb meem nrog tus cwj pwm tsis yog qhov project.
• Ntsuas seb cov haujlwm uas yuav tsum tau muaj nyob rau hauv cov ntaub ntawv thiab yog tias tsis yog, ces yuav tsum tau npaum li cas thiab ntau npaum li cas rau kev ua haujlwm.

Qhov tseem ceeb tshaj plaws yog txhawm rau kwv yees lub sijhawm - koj yuav tsum siv sijhawm thiab siv zog ntawm "kev tshawb nrhiav hauv kev quab yuam" - tsuas yog tom qab ntawd koj txoj kev npaj yuav txaus. Yog li ntawd, tsis muaj teeb meem ntau npaum li cas koj raug thawb kom hais "nws yuav siv sijhawm ntev npaum li cas los sau cov ntaub ntawv" - yuav koj tus kheej qee lub sijhawm rau kev tshuaj xyuas ua ntej thiab sib cav los ntawm ntau npaum li cas lub sijhawm yuav txawv nyob ntawm qhov tsis sib xws ntawm qhov teeb meem.

Thiab tam sim no peb yuav ua kom pom cov piv txwv tshwj xeeb uas cov kev txwv yuav hloov.

Cov Ntsiab Lus Tseem Ceeb: Kev kwv yees yog raws li kev txheeb xyuas ntawm cov ntsiab lus tseem ceeb cuam tshuam txog qhov kev ua haujlwm thiab qhov nyuaj ntawm txoj haujlwm.

Kev kwv yees-raws li kev kwv yees yog ib txoj hauv kev zoo thaum cov ntsiab lus ua haujlwm me me thiab tsis muaj ntau yam uas tuaj yeem cuam tshuam qhov tsim ntawm qhov teeb meem. Tab sis nyob rau hauv cov ntaub ntawv ntawm ib tug xov tooj ntawm cov ntaub ntawv Science teeb meem, tej yam xws li ua heev heev heev thiab xws li ib tug mus kom ze yuav tsis txaus.

Kev sib piv ntawm Reddit cov zej zog

Cia peb pib nrog qhov yooj yim tshaj plaws (raws li nws hloov tawm tom qab). Feem ntau, kom ua ncaj ncees, peb muaj qhov yuav luag zoo tagnrho, cia peb tshawb xyuas peb cov ntawv txheeb xyuas qhov nyuaj:

• Nws muaj qhov zoo, meej thiab cov ntaub ntawv API.
• Nws yog qhov yooj yim heev thiab qhov tseem ceeb tshaj plaws, lub token tau txais tau txais.
• muaj python qhwv - nrog ntau yam piv txwv.
• Ib lub zej zog uas txheeb xyuas thiab sau cov ntaub ntawv ntawm reddit (txawm tias cov yeeb yaj kiab YouTube piav qhia yuav ua li cas siv python wrapper) Piv txwv li.
• Cov txheej txheem peb xav tau feem ntau yuav muaj nyob hauv API. Ntxiv mus, cov cai zoo li compact thiab huv si, hauv qab no yog ib qho piv txwv ntawm kev ua haujlwm uas sau cov lus ntawm ib qho kev tshaj tawm.

def get_comments(submission_id):
    reddit = Reddit(check_for_updates=False, user_agent=AGENT)
    submission = reddit.submission(id=submission_id)
    more_comments = submission.comments.replace_more()
    if more_comments:
        skipped_comments = sum(x.count for x in more_comments)
        logger.debug('Skipped %d MoreComments (%d comments)',
                     len(more_comments), skipped_comments)
    return submission.comments.list()

Sau los ntawm qhov no xaiv cov khoom siv yooj yim rau kev qhwv.

Txawm tias muaj tseeb tias qhov no yog qhov zoo tshaj plaws, nws tseem tsim nyog coj mus rau hauv tus account ntau yam tseem ceeb ntawm lub neej tiag tiag:

• API txwv - peb raug yuam kom nqa cov ntaub ntawv hauv pawg (pw ntawm kev thov, thiab lwm yam).
• Lub sijhawm sau - txhawm rau ua tiav kev txheeb xyuas thiab sib piv, koj yuav tsum teem sijhawm tseem ceeb rau kab laug sab taug kev los ntawm subreddit.
• Lub bot yuav tsum khiav ntawm lub server - koj tsis tuaj yeem khiav ntawm koj lub laptop, muab tso rau hauv koj lub hnab ev ntawv, thiab mus txog koj txoj kev lag luam. Yog li kuv tau khiav txhua yam ntawm VPS. Siv cov cai tshaj tawm habrahabr10 koj tuaj yeem txuag tau 10% ntawm tus nqi ntxiv.
• Kev nkag tsis tau ntawm lub cev ntawm qee cov ntaub ntawv (lawv pom rau cov thawj coj lossis nyuaj dhau los sau) - qhov no yuav tsum raug coj mus rau hauv tus account; hauv txoj cai, tsis yog txhua cov ntaub ntawv tuaj yeem sau tau raws sijhawm.
• Network yuam kev: Networking yog mob.
• Qhov no yog nyob cov ntaub ntawv tiag tiag - nws yeej tsis ntshiab.

Tau kawg, nws yog ib qho tsim nyog yuav tsum suav nrog cov nuances hauv kev txhim kho. Cov sijhawm tshwj xeeb / hnub nyob ntawm qhov kev paub dhau los lossis kev paub ua haujlwm ntawm cov haujlwm zoo sib xws, txawm li cas los xij, peb pom tias cov haujlwm no yog kev tsim vaj tsev nkaus xwb thiab tsis xav tau ntxiv lub cev txav los daws - txhua yam tuaj yeem ntsuas tau zoo heev, teem sijhawm thiab ua tiav.

Kev sib piv ntawm Habr ntu

Cia peb mus rau qhov nthuav dav ntxiv thiab tsis tseem ceeb ntawm kev sib piv cov xov thiab / lossis ntu ntawm Habr.

Cia peb tshawb xyuas peb cov ntawv txheeb xyuas qhov nyuaj - ntawm no, txhawm rau nkag siab txhua lub ntsiab lus, koj yuav tsum khawb me ntsis rau hauv txoj haujlwm nws tus kheej thiab sim.

• Thaum xub thawj koj xav tias muaj API, tab sis tsis muaj. Yog, yog, Habr muaj API, tab sis nws tsuas yog siv tsis tau rau cov neeg siv (lossis tej zaum nws tsis ua haujlwm txhua).
• Ces koj cia li pib parsing html - "ntshuam thov", dab tsi yuav mus tsis ncaj ncees lawm?
• Yuav ua li cas parse lawm? Qhov yooj yim tshaj plaws thiab feem ntau siv txoj hauv kev yog kom rov hais dua cov IDs, nco ntsoov tias nws tsis yog qhov ua tau zoo tshaj plaws thiab yuav tsum tau daws cov teeb meem sib txawv - ntawm no yog ib qho piv txwv ntawm qhov ceev ntawm ID tiag tiag ntawm txhua tus uas twb muaj lawm.

  
  Sau los ntawm qhov no lus.

• Cov ntaub ntawv nyoos qhwv hauv HTML saum lub vev xaib yog qhov mob. Piv txwv li, koj xav sau thiab txuag qhov kev ntaus nqi ntawm ib tsab xov xwm: koj rho cov qhab nia tawm ntawm html thiab txiav txim siab khaws nws ua tus lej rau kev ua haujlwm ntxiv: 

  1) int (cov qhab nia) cuam tshuam qhov yuam kev: txij li ntawm Habre muaj qhov rho tawm, piv txwv li, hauv kab “–5” - qhov no yog qhov nrawm, tsis yog kos npe rho tawm (piv txwv li, txoj cai?), yog li ntawm qee lub ntsiab lus kuv yuav tsum tau tsa tus parser rau lub neej nrog xws li ib tug txaus ntshai kho.

  try:
        score_txt = post.find(class_="score").text.replace(u"–","-").replace(u"+","+")
        score = int(score_txt)
        if check_date(date):
          post_score += score
  

  Tej zaum yuav tsis muaj hnub, pluses thiab minuses txhua (raws li peb pom saum toj no hauv kev ua haujlwm check_date, qhov no tshwm sim).

  2) Cov cim tshwj xeeb uas tsis tau dim - lawv yuav tuaj, koj yuav tsum tau npaj.

  3) Cov qauv hloov nyob ntawm seb hom ncej.

  4) Cov ntawv qub yuav muaj ** qauv txawv txawv **.

• Qhov tseem ceeb, kev ua yuam kev thiab qhov yuav tshwm sim lossis tsis tshwm sim yuav tsum tau ua thiab koj tsis tuaj yeem twv kom paub tseeb tias yuav ua li cas tsis raug thiab lwm tus qauv yuav ua li cas thiab yuav poob qhov twg - koj tsuas yog yuav tsum tau sim thiab coj mus rau hauv tus account. qhov yuam kev uas tus parser pov.
• Tom qab ntawd koj paub tias koj yuav tsum tau txheeb xyuas hauv ntau cov xov, txwv tsis pub parsing hauv ib qho yuav siv sijhawm 30+ teev (qhov no tsuas yog lub sijhawm ua tiav ntawm ib tus xov tooj uas twb ua haujlwm lawm, uas tsaug zog thiab tsis poob rau hauv ib qho kev txwv). IN qhov no tsab xov xwm, qhov no coj ntawm qee lub ntsiab lus mus rau qhov zoo sib xws:

Tag nrho cov checklist los ntawm complexity:

• Ua hauj lwm nrog lub network thiab html parsing nrog iteration thiab nrhiav los ntawm ID.
• Cov ntaub ntawv ntawm heterogeneous qauv.
• Muaj ntau qhov chaw uas tus lej tuaj yeem poob yooj yim.
• Yuav tsum sau || code.
• Cov ntaub ntawv tsim nyog, cov qauv code, thiab / lossis cov zej zog ploj lawm.

Lub sijhawm kwv yees rau txoj haujlwm no yuav yog 3-5 npaug siab dua li kev sau cov ntaub ntawv los ntawm Reddit.

Kev sib piv ntawm Odnoklassniki pawg

Cia peb txav mus rau qhov kev nthuav dav tshaj plaws uas tau piav qhia. Rau kuv, nws yog qhov nthuav meej meej vim tias thaum xub thawj siab ib muag, nws zoo li tsis tseem ceeb, tab sis nws tsis tig tawm mus rau txoj kev ntawd txhua - sai li sai tau thaum koj hnia nws.

Cia peb pib nrog peb cov ntawv txheeb xyuas nyuaj thiab nco ntsoov tias ntau ntawm lawv yuav dhau los ua nyuaj dua li qhov lawv saib thawj zaug:

• Muaj API, tab sis nws yuav luag tag nrho tsis muaj cov haujlwm tsim nyog.
• Rau qee lub luag haujlwm koj yuav tsum tau thov kev nkag los ntawm kev xa ntawv, uas yog, kev tso cai nkag tsis tau sai.
• Nws yog cov ntaub ntawv txaus ntshai (pib nrog, lus Lavxias thiab lus Askiv sib xyaw rau txhua qhov chaw, thiab tsis sib xws - qee zaum koj tsuas yog xav twv seb lawv xav tau los ntawm koj qhov chaw) thiab, ntxiv rau, tus qauv tsim tsis haum rau kev tau txais cov ntaub ntawv, piv txwv li. , txoj haujlwm peb xav tau.
• Yuav tsum muaj kev sib tham hauv cov ntaub ntawv, tab sis tsis siv tiag tiag - thiab tsis muaj txoj hauv kev los nkag siab txhua qhov tsis sib xws ntawm API hom uas tsis yog poking ncig thiab cia siab tias qee yam yuav ua haujlwm.
• Tsis muaj cov piv txwv thiab tsis muaj zej zog; tib lub ntsiab lus ntawm kev txhawb nqa hauv kev sau cov ntaub ntawv yog me me qhwv hauv Python (tsis muaj ntau yam piv txwv ntawm kev siv).
• Selenium zoo li yog qhov kev xaiv ua haujlwm tshaj plaws, txij li ntau cov ntaub ntawv tsim nyog raug kaw.
  1) Qhov ntawd yog, kev tso cai tshwm sim los ntawm tus neeg siv tsis tseeb (thiab sau npe los ntawm tes).

  2) Txawm li cas los xij, nrog Selenium tsis muaj kev lees paub rau kev ua haujlwm raug thiab rov ua haujlwm dua (tsawg kawg hauv rooj plaub ntawm ok.ru kom paub meej).

  3) Lub vev xaib Ok.ru muaj JavaScript yuam kev thiab qee zaum coj txawv txawv thiab tsis sib haum.

  4) Koj yuav tsum ua pagination, thauj khoom, thiab lwm yam ...

  5) API yuam kev uas lub wrapper muab yuav tsum tau ua txawv txawv, piv txwv li, zoo li no (ib daim ntawm kev sim code):

  def get_comments(args, context, discussions):
      pause = 1
      if args.extract_comments:
          all_comments = set()
  #makes sense to keep track of already processed discussions
          for discussion in tqdm(discussions): 
              try:
                  comments = get_comments_from_discussion_via_api(context, discussion)
              except odnoklassniki.api.OdnoklassnikiError as e:
                  if "NOT_FOUND" in str(e):
                      comments = set()
                  else:
                      print(e)
                      bp()
                      pass
              all_comments |= comments
              time.sleep(pause)
          return all_comments
  

  Kuv qhov yuam kev uas kuv nyiam tshaj yog:

  OdnoklassnikiError("Error(code: 'None', description: 'HTTP error', method: 'discussions.getComments', params: …)”)

  6) Thaum kawg, Selenium + API zoo li qhov kev xaiv zoo tshaj plaws.

• Nws yog ib qho tsim nyog los cawm lub xeev thiab rov pib lub system, daws ntau yam yuam kev, suav nrog kev coj tus cwj pwm tsis zoo ntawm lub xaib - thiab cov kev ua yuam kev no nyuaj rau xav txog (tshwj tsis yog koj sau parsers kev tshaj lij, tau kawg).

Lub sijhawm kwv yees lub sijhawm rau txoj haujlwm no yuav yog 3-5 zaug siab dua rau kev sau cov ntaub ntawv los ntawm Habr. Txawm tias muaj tseeb hais tias nyob rau hauv cov ntaub ntawv ntawm Habr peb siv ib tug frontal mus kom ze nrog HTML parsing, thiab nyob rau hauv cov ntaub ntawv ntawm OK peb yuav ua hauj lwm nrog API nyob rau hauv qhov chaw tseem ceeb.

tshawb pom

Tsis muaj teeb meem ntau npaum li cas koj yuav tsum kwv yees lub sijhawm kawg "ntawm qhov chaw" (peb tab tom npaj hnub no!) ntawm cov ntaub ntawv voluminous processing pipeline module, lub sijhawm ua tiav yuav luag tsis muaj peev xwm kwv yees txawm tias muaj txiaj ntsig yam tsis tau txheeb xyuas cov haujlwm ua haujlwm.

Ntawm cov lus qhia me ntsis ntxiv, cov tswv yim kwv yees agile ua haujlwm zoo rau kev ua haujlwm engineering, tab sis cov teeb meem uas muaj kev sim ntau dua thiab, ntawm qhov tseeb, "muaj tswv yim" thiab kev tshawb nrhiav, piv txwv li, tsis tshua muaj peev xwm, muaj teeb meem, xws li hauv cov piv txwv ntawm cov ncauj lus zoo sib xws, uas peb tau tham txog ntawm no.

Ntawm chav kawm, kev sau cov ntaub ntawv tsuas yog ib qho piv txwv tseem ceeb - nws feem ntau yog ib qho yooj yim heev thiab technically uncomplicated ua hauj lwm, thiab dab ntxwg nyoog yog feem ntau nyob rau hauv cov ntsiab lus. Thiab nws yog qhov tseeb ntawm txoj haujlwm no uas peb tuaj yeem qhia tag nrho cov kev xaiv uas muaj peev xwm ua tau rau qhov twg tuaj yeem mus tsis ncaj ncees lawm thiab raws nraim li cas lub sijhawm ua haujlwm tuaj yeem siv sijhawm.

Yog tias koj saib ntawm tus yam ntxwv ntawm txoj haujlwm yam tsis muaj kev sim ntxiv, tom qab ntawd Reddit thiab OK zoo sib xws: muaj API, python wrapper, tab sis qhov tseeb, qhov sib txawv loj heev. Kev txiav txim siab los ntawm cov kev txwv no, Habr's pars zoo li nyuaj dua li OK - tab sis hauv kev xyaum nws yog qhov tsis sib xws, thiab qhov no yog qhov tseeb uas tuaj yeem pom los ntawm kev sim yooj yim los tshuaj xyuas qhov teeb meem ntawm qhov teeb meem.

Hauv kuv qhov kev paub dhau los, txoj hauv kev ua tau zoo tshaj plaws yog kwv yees kwv yees lub sijhawm koj yuav xav tau rau qhov kev tshuaj xyuas ua ntej nws tus kheej thiab kev sim ua ntej yooj yim, nyeem cov ntaub ntawv - cov no yuav tso cai rau koj muab kev kwv yees tseeb rau tag nrho cov haujlwm. Nyob rau hauv cov nqe lus ntawm cov txheej txheem nrov agile, kuv hais kom koj tsim ib daim pib rau "kwv yees tsis ua hauj lwm", raws li kuv muaj peev xwm muab ib qho kev ntsuam xyuas ntawm dab tsi yuav ua tau tiav nyob rau hauv lub "sprint" thiab muab ib tug ntau tseeb kwv yees rau txhua. txoj hauj lwm.

Yog li ntawd, qhov kev sib cav zoo tshaj plaws zoo li yog ib qho uas yuav qhia tus kws tshaj lij "tsis yog txuj ci" ntau npaum li cas lub sij hawm thiab cov peev txheej yuav txawv nyob ntawm qhov tsis tau ntsuas.

Tau qhov twg los: www.hab.com