Thaum Xyoo Tshiab Olivier tiav lawm, kuv tsis muaj dab tsi ua, thiab kuv txiav txim siab rub tawm tag nrho cov ntawv los ntawm Habrahabr (thiab lwm lub platforms) rau kuv lub computer thiab tshawb nrhiav.
Muaj ob peb zaj dab neeg nthuav. Thawj ntawm lawv yog kev txhim kho cov hom ntawv thiab cov ncauj lus ntawm cov ntawv tshaj 12 xyoo ntawm qhov chaw nyob. Piv txwv li, lub zog ntawm qee cov ntsiab lus yog qhov qhia tau zoo heev. Txuas ntxiv - hauv qab txiav.
Kev txheeb xyuas
Yuav kom nkag siab tias Habr tau tsim li cas, nws yuav tsum mus dhau tag nrho nws cov ntawv thiab rho tawm cov ntaub ntawv meta los ntawm lawv (piv txwv li, hnub). Txoj kev bypass yog ib qho yooj yim, vim hais tias txuas mus rau tag nrho cov ntawv zoo li "habrahabr.ru/post/337722/", thiab cov lej tau muab nruj me ntsis raws li kev txiav txim. Paub tias qhov kawg tshaj tawm muaj tus lej me ntsis tsawg dua 350 txhiab, kuv tsuas yog mus dhau tag nrho cov ntaub ntawv tau id hauv lub voj (Python code):
import numpy as np
from multiprocessing import Pool
with Pool(100) as p:
docs = p.map(download_document, np.arange(350000))muaj nuj nqi download_document sim thauj cov nplooj ntawv nrog tus lej sib raug thiab sim rho tawm cov ntaub ntawv tseem ceeb los ntawm cov qauv html.
import requests
from bs4 import BeautifulSoup
def download_document(pid):
""" Download and process a Habr document and its comments """
# выгрузка документа
r = requests.get('https://habrahabr.ru/post/' +str(pid) + '/')
# парсинг документа
soup = BeautifulSoup(r.text, 'html5lib') # instead of html.parser
doc = {}
doc['id'] = pid
if not soup.find("span", {"class": "post__title-text"}):
# такое бывает, если статья не существовала или удалена
doc['status'] = 'title_not_found'
else:
doc['status'] = 'ok'
doc['title'] = soup.find("span", {"class": "post__title-text"}).text
doc['text'] = soup.find("div", {"class": "post__text"}).text
doc['time'] = soup.find("span", {"class": "post__time"}).text
# create other fields: hubs, tags, views, comments, votes, etc.
# ...
# сохранение результата в отдельный файл
fname = r'files/' + str(pid) + '.pkl'
with open(fname, 'wb') as f:
pickle.dump(doc, f)Nyob rau hauv tus txheej txheem ntawm parsing, kuv nrhiav tau ob peb yam tshiab.
Ua ntej, lawv hais tias tsim cov txheej txheem ntau dua li muaj cov cores hauv processor yog qhov tsis muaj txiaj ntsig. Tab sis nyob rau hauv kuv cov ntaub ntawv, nws tau muab tawm hais tias lub limiting kev pab cuam tsis yog lub processor, tab sis lub network, thiab 100 txheej txheem ua hauj lwm sai tshaj li 4 los yog, hais, 20.
Qhov thib ob, hauv qee qhov kev tshaj tawm muaj kev sib txuas ntawm cov cim tshwj xeeb - piv txwv li, euphemisms zoo li "%&#@". Nws muab tawm tias html.parser, uas kuv siv ua ntej, reacts rau kev sib xyaw ua ke &# mob siab, xav txog nws qhov pib ntawm html qhov chaw. Kuv twb tau mus ua khawv koob dub, tab sis lub rooj sab laj hais tias koj tuaj yeem hloov tus parser xwb.
Thirdly, kuv tau tswj kom unload tag nrho cov ntaub ntawv, tsuas yog rau peb. Cov ntaub ntawv naj npawb 65927, 162075, thiab 275987 tau raug tshem tawm tam sim los ntawm kuv cov tshuaj tiv thaiv kab mob. Cov no yog cov khoom, feem, hais txog ib tug javascript saw uas downloads ib tug phem pdf, SMS ransomware nyob rau hauv daim ntawv ntawm ib tug txheej ntawm browser plugins, thiab lub site CrashSafari.com uas xa iPhones mus rau hauv ib tug reboot. Antivirus pom lwm tsab xov xwm tom qab, thaum lub sij hawm luam theej duab: tshaj tawm 338586 txog cov ntawv sau rau ntawm lub khw muag tsiaj lub vev xaib uas siv tus neeg siv cov txheej txheem rau kuv cov cryptocurrency. Yog li peb tuaj yeem xav txog kev ua haujlwm ntawm antivirus yog qhov tsim nyog.
"Nyob" cov khoom tau muab tawm tsuas yog ib nrab ntawm cov peev txheej siab tshaj plaws - 166307 daim. Hais txog qhov seem, Habr muab cov kev xaiv "cov nplooj ntawv dhau los, tau raug tshem tawm lossis tsis muaj nyob txhua qhov." Zoo, txhua yam tuaj yeem tshwm sim.
Lub uploading ntawm cov khoom tau ua raws li kev ua haujlwm: piv txwv li, cov hnub tshaj tawm yuav tsum tau hloov los ntawm hom "'21 Kaum Ob Hlis 2006 ntawm 10:47 sawv ntxov" rau tus qauv datetime, thiab "12,8k" views - hauv 12800. Nyob rau theem no, ob peb qhov xwm txheej ntxiv tau tawm. Qhov kev lom zem tshaj plaws yog ua nrog kev suav pov npav thiab cov ntaub ntawv hom: qee qhov kev tshaj tawm qub tau dhau los thiab tau txais 65535 votes txhua.
Raws li qhov tshwm sim, cov ntawv ntawm cov ntawv (tsis muaj duab) coj kuv 1.5 gigabytes, cov lus pom nrog cov ntaub ntawv meta - lwm 3, thiab txog ib puas megabytes - cov ntaub ntawv meta-cov ntaub ntawv hais txog cov khoom. Qhov no tuaj yeem khaws cia tag nrho hauv RAM, uas yog qhov zoo siab rau kuv.
Kuv pib tshawb xyuas cov ntawv tsis yog los ntawm cov ntawv lawv tus kheej, tab sis los ntawm cov ntaub ntawv meta: hnub tim, cim npe, hubs, views thiab nyiam. Nws tau pom tias nws tuaj yeem qhia ntau yam.
Habrahabr Development Trends
Cov khoom ntawm lub xaib tau tshaj tawm txij li xyoo 2006; feem ntau intensive - hauv 2008-2016.
Yuav ua li cas cov kab lus no tau nyeem nyob rau lub sijhawm sib txawv tsis yooj yim rau kev ntsuam xyuas. Cov ntawv sau los ntawm 2012 thiab yau tau txais ntau cov lus pom thiab kev ntaus nqi, tab sis cov ntawv tshiab muaj ntau qhov kev pom thiab bookmarks. Cov kev ntsuas no coj tib txoj kev (ib nrab) ib zaug xwb, hauv 2015. Tej zaum, nyob rau hauv qhov xwm txheej ntawm kev lag luam thiab kev nom kev tswv, kev saib xyuas ntawm cov neeg nyeem tau hloov los ntawm IT blogs mus rau cov teeb meem mob ntau dua.
Ntxiv rau cov ntawv lawv tus kheej, kuv rub tawm ntau cov lus rau lawv. Muaj 6 lab cov lus pom, txawm li cas los xij, 240 txhiab ntawm lawv raug txwv ("UFO ya mus thiab luam tawm cov ntawv sau no ntawm no"). Ib qho khoom muaj txiaj ntsig ntawm cov lus pom yog tias lawv muaj lub sijhawm sau ntawv. Los ntawm kev kawm lub sijhawm ntawm cov lus pom, koj tuaj yeem nkag siab zoo thaum cov ntawv nyeem txhua.
Nws muab tawm tias feem ntau ntawm cov ntawv no yog ob qho tib si sau thiab tawm tswv yim qhov chaw ntawm 10 teev sawv ntxov txog 20 teev tsaus ntuj, piv txwv li. nyob rau hauv ib txwm Moscow hnub ua hauj lwm. Qhov no yuav txhais tau tias Habr tau nyeem rau cov hom phiaj kev tshaj lij, thiab tias qhov no yog txoj hauv kev zoo rau ncua sijhawm ua haujlwm. Los ntawm txoj kev, qhov kev faib ntawm lub sij hawm ntawm hnub no yog ruaj khov los ntawm lub hauv paus ntawm Habr mus rau niaj hnub nim no.
Txawm li cas los xij, qhov txiaj ntsig tseem ceeb ntawm lub sijhawm sau ntawv tsis yog lub sijhawm ntawm hnub, tab sis lub sijhawm ntawm "lub neej nquag" ntawm tsab xov xwm. Kuv xam yuav ua li cas lub sij hawm faib los ntawm kev tshaj tawm ntawm tsab xov xwm rau nws cov lus. Nws muab tawm tias tam sim no cov lus nruab nrab (kab ntsuab) tuaj txog li 20 teev, i.e. thawj hnub tom qab tshaj tawm, qhov nruab nrab, me ntsis ntau tshaj li ib nrab ntawm tag nrho cov lus ntawm tsab xov xwm raug tso tseg. Thiab nyob rau hauv ob hnub lawv tawm 75% ntawm tag nrho cov lus. Nyob rau tib lub sijhawm, cov ntawv ua ntej tau nyeem sai dua - piv txwv li, xyoo 2010, ib nrab ntawm cov lus tuaj hauv thawj 6 teev.
Nws tuaj raws li qhov xav tsis thoob rau kuv tias cov lus tau ntev lawm: qhov nruab nrab ntawm cov cim hauv kev tawm tswv yim tau yuav luag ob npaug rau lub neej ntawm Habr!
Kev tawm tswv yim yooj yim dua li cov lus pom yog kev pov npav. Tsis zoo li ntau lwm yam kev pab, ntawm Habre koj tuaj yeem tso tsis tau tsuas yog pluses, tab sis kuj minuses. Txawm li cas los xij, cov neeg nyeem tsis siv lub sijhawm dhau los ntau zaus: qhov feem ntau tsis nyiam tam sim no yog kwv yees li 15% ntawm tag nrho cov pov npav pov npav. Muaj ntau dua, tab sis dhau sijhawm, cov neeg nyeem tau ua siab zoo.
Cov ntawv lawv tus kheej tau hloov pauv raws sijhawm. Piv txwv li, qhov ntev ntawm cov ntawv nyeem tsis nres tsis tu ncua los ntawm kev tshaj tawm ntawm lub xaib, txawm tias muaj kev kub ntxhov. Hauv ib xyoo caum, cov ntawv tau dhau los yuav luag kaum zaus!
Cov style ntawm cov ntawv (rau ib qho kev kwv yees thawj zaug) kuj hloov. Nyob rau thawj xyoo ntawm Habr lub neej, piv txwv li, qhov sib koom ntawm cov lej thiab cov lej hauv cov ntawv tau nce:
Tom qab nkag siab tag nrho cov dynamics ntawm lub xaib, kuv txiav txim siab los ntsuas seb qhov kev nyiam ntawm ntau yam kev hloov pauv. Cov ntsiab lus tuaj yeem raug xaiv los ntawm cov ntawv, tab sis rau qhov pib, koj tsis tuaj yeem rov tsim lub log, tab sis siv cov ntawv npaj ua tiav los ntawm cov kws sau ntawv ntawm txhua tsab xov xwm. Kuv tau kos plaub qhov sib txawv ntawm daim duab. Lub ntsiab lus "Google" pib thawj zaug (tej zaum feem ntau yog vim SEO), tab sis tau poob phaus ntau xyoo. Javascript tau yog ib lub ntsiab lus nrov thiab txuas ntxiv mus qeeb, tab sis kev kawm tshuab tau pib sai sai tau nrov hauv xyoo tas los no. Linux, ntawm qhov tod tes, tseem muaj qhov sib npaug sib npaug thoob plaws xyoo caum.
Tau kawg, kuv tau xav txog cov ncauj lus dab tsi nyiam nyeem ntau cov haujlwm. Kuv suav qhov nruab nrab ntawm cov kev pom, kev pov npav thiab cov lus pom hauv txhua lub ncauj lus. Nov yog qhov tshwm sim:
- Cov ntsiab lus pom feem ntau: arduino, web tsim, txhim kho lub vev xaib, digest, txuas, css, html, html5, nginx, algorithms.
- Cov ntsiab lus "nyiam" tshaj plaws: vkontakte, lom zem, jquery, opera, c, html, web development, html5, css, web design.
- Cov ntsiab lus sib tham tshaj plaws: opera, skype, ywj pheej, vkontakte, ubuntu, ua haujlwm, nokia, nginx, arduino, firefox.
Los ntawm txoj kev, txij li thaum kuv tab tom sib piv cov ntsiab lus, koj tuaj yeem txheeb lawv los ntawm zaus (thiab sib piv cov txiaj ntsig nrog ).
- Rau tag nrho cov xyoo ntawm Habr lub neej, cov cim npe nrov tshaj plaws (hauv qhov kev txiav txim qis zuj zus) yog google, hauv xov tooj ntawd, javascript, microsoft, linux, php, apple, java, python, programming, startups, development, ios, startup, social networks
- Hauv xyoo 2017, qhov nrov tshaj plaws yog javascript, python, java, android, development, linux, c++, programming, php, c#, ios, machine learning, information security, microsoft, react.
Thaum muab piv rau cov kev ntsuas no, ib tus tuaj yeem mloog zoo, piv txwv li, rau kev yeej ntawm Python thiab kev ploj tuag ntawm php, lossis rau "hnub poob" ntawm cov ncauj lus pib thiab nce ntawm kev kawm tshuab.
Tsis yog tag nrho cov cim npe ntawm Habre muaj qhov pom tseeb thematic xim. Piv txwv li, ntawm no yog kaum lub cim uas tau ntsib ib zaug xwb, tab sis tsuas yog zoo li funny rau kuv. Yog li: "lub tswv yim yog tus tsav lub zog ntawm kev vam meej", "khau raj los ntawm ib tug floppy duab", "Iowa State", "drama", "superalesh", "steam engine", "yam yuav ua rau hnub Saturday", "Kuv muaj. hma nyob rau hauv ib tug nqaij grinder", "ib tug nws tig tawm raws li ib txwm", "peb tsis tuaj yeem tuaj nrog funny tags". Txhawm rau txiav txim siab cov ntsiab lus ntawm cov ntawv zoo li no, cov ntawv cim tsis txaus - koj yuav tsum tau ua cov qauv thematic ntawm cov ntawv ntawm cov ntawv.
Ib qho kev soj ntsuam ntxaws ntxiv ntawm cov ntsiab lus ntawm cov kab lus yuav nyob hauv cov ntawv tom ntej. Ua ntej, kuv yuav tsim cov qauv uas kwv yees tus naj npawb ntawm nplooj ntawv saib rau ib tsab xov xwm raws li nws cov ntsiab lus. Qhov thib ob, kuv xav qhia cov neural network los tsim cov ntawv nyeem tib yam li cov kws sau ntawv Habr. Yog li subscribe 🙂
PS Thiab ntawm no yog lub beeped .
Tau qhov twg los: www.hab.com