Apo Gore Idzva Olivier rakapera, ndakanga ndisina chokuita, uye ndakasarudza kutora zvinyorwa zvose kubva kuHabrahabr (uye mapuratifomu ane hukama) kumakombiyuta yangu uye kuongorora.
Paive nenyaya dzakati wandei dzainakidza. Chekutanga chazvo kuvandudzwa kwemafomati uye misoro yezvinyorwa pamusoro pemakore gumi nemaviri ekuvapo kwesaiti. Semuyenzaniso, dynamics yedzimwe misoro inotoratidza. Kuenderera mberi - pasi pekucheka.
Parsing process
Kuti unzwisise kuti Habr akakura sei, zvaive zvakakodzera kuti upfuure muzvinyorwa zvake zvese uye kubvisa meta-ruzivo kubva kwavari (semuenzaniso, mazuva). Kupfuura kwaive nyore, nekuti zvinongedzo kune zvese zvinyorwa zvinotaridzika se "habrahabr.ru/post/337722/", uye nhamba dzinopihwa zvakanyatsorongeka. Ndichiziva kuti positi yekupedzisira ine nhamba isingasviki zviuru mazana matatu nemakumi mashanu, ndangopfuura nepakati pese zvingangoita gwaro id mune loop (Python kodhi):
import numpy as np
from multiprocessing import Pool
with Pool(100) as p:
docs = p.map(download_document, np.arange(350000))shanda download_document inoedza kurodha peji ine id inofambirana uye inoedza kuburitsa ruzivo rwakakosha kubva muchimiro chehtml.
import requests
from bs4 import BeautifulSoup
def download_document(pid):
""" Download and process a Habr document and its comments """
# выгрузка документа
r = requests.get('https://habrahabr.ru/post/' +str(pid) + '/')
# парсинг документа
soup = BeautifulSoup(r.text, 'html5lib') # instead of html.parser
doc = {}
doc['id'] = pid
if not soup.find("span", {"class": "post__title-text"}):
# такое бывает, если статья не существовала или удалена
doc['status'] = 'title_not_found'
else:
doc['status'] = 'ok'
doc['title'] = soup.find("span", {"class": "post__title-text"}).text
doc['text'] = soup.find("div", {"class": "post__text"}).text
doc['time'] = soup.find("span", {"class": "post__time"}).text
# create other fields: hubs, tags, views, comments, votes, etc.
# ...
# сохранение результата в отдельный файл
fname = r'files/' + str(pid) + '.pkl'
with open(fname, 'wb') as f:
pickle.dump(doc, f)Mukati mekuongorora, ndakawana zvinhu zvitsva zvakawanda.
Kutanga, ivo vanoti kugadzira akawanda maitiro kupfuura kune macores mu processor hakubatsiri. Asi mune yangu, zvakazoitika kuti iyo inodzikamisa sosi haisi iyo processor, asi network, uye zana maitiro anoshanda nekukurumidza kupfuura 100 kana, toti, makumi maviri.
Chechipiri, mune mamwe matsamba maive nemusanganiswa wemavara akakosha - semuenzaniso, euphemisms se "%&#@". Zvakabva zvaita html.parser, yandakatanga kushandisa, inopindirana nekubatanidzwa &# zvinorwadza, tichifunga nezvekutanga kweiyo html entity. Ini ndanga ndatove kuzoita mashiripiti, asi foramu yakataura kuti unogona kungochinja parser.
Chechitatu, ndakakwanisa kuburutsa mabhuku ose, kunze kwematatu. Zvinyorwa zvine nhamba 65927, 162075, uye 275987 zvakabva zvadzimwa neantivirus yangu. Izvi zvinyorwa, zvichiteerana, nezve javascript cheni inorodha yakashata pdf, SMS ransomware muchimiro cheseti yebrowser plugins, uye CrashSafari.com saiti inotumira iPhones mukutangazve. Antivirus yakawana chimwe chinyorwa gare gare, panguva yekuongorora system: tumira 338586 nezve zvinyorwa pawebhusaiti yechitoro chemhuka dzinoshandisa processor yemushandisi kuchera cryptocurrency. Saka isu tinogona kufunga kuti basa reantivirus rakakwana.
"Rarama" zvinyorwa zvakazongove hafu chete yeinokwanisika - 166307 zvidimbu. Nezve zvimwe, Habr inopa sarudzo "iyo peji yakare, yakadzimwa kana yakanga isipo zvachose." Zvakanaka, chero chinhu chinogona kuitika.
Kukwidzwa kwezvinyorwa kwakateverwa nebasa rehunyanzvi: semuenzaniso, mazuva ekuburitswa aifanira kushandurwa kubva mufomati "'21 Zvita 2006 na 10:47 am" kuenda kune yakajairwa. datetime, uye "12,8k" maonero - muna 12800. Panguva ino, zvimwe zviitiko zvishomanana zvakabuda. Iyo inonakidza kwazvo ine chekuita nehuwandu hwemavhoti uye mhando dzedata: mamwe mapositi ekare aive nekufashukira uye akagamuchira 65535 mavhoti imwe neimwe.
Somugumisiro, zvinyorwa zvezvinyorwa (pasina mifananidzo) zvakanditora 1.5 gigabytes, zvinyorwa zvine meta-ruzivo - imwe 3, uye inenge zana megabytes - meta-ruzivo pamusoro pezvinyorwa. Izvi zvinogona kuchengetwa zvizere muRAM, izvo zvakandishamisa kushamisika.
Ndakatanga ongororo yezvinyorwa kwete kubva muzvinyorwa pachazvo, asi kubva kune meta-ruzivo: misi, tags, hubs, maonero uye zvaanoda. Zvakaratidza kuti aigona kutaura zvakawanda.
Habrahabr Development Trends
Zvinyorwa pane saiti zvakaburitswa kubva 2006; zvakanyanya zvakanyanya - muna 2008-2016.
Kuti zvinyorwa izvi zvakaverengwa sei panguva dzakasiyana hazvisi nyore kuongorora. Zvinyorwa kubva 2012 uye vadiki vakagamuchira mamwe makomendi uye zviyero, asi zvinyorwa zvitsva zvine mamwe maonero uye mabhukumaki. Aya metrics akazvibata nenzira imwechete (pakati) kamwe chete, muna 2015. Zvichida, mumamiriro ezvinhu ematambudziko ehupfumi uye ezvematongerwo enyika, kutarisa kwevaverengi kwakabva kune IT blogs kune dzimwe nyaya dzinorwadza.
Pamusoro pezvinyorwa zvacho, ndakadhaunirodha mamwe mashoko kwavari. Paiva nemamiriyoni matanhatu emashoko, zvisinei, 6 zviuru zvadzo zvakarambidzwa ("UFO yakabhururuka mukati ndokubudisa ichi chinyorwa pano"). Chinhu chinobatsira chemashoko ndechekuti vane chitambi chenguva. Nekufunda nguva yemhinduro, unogona kunzwisisa kana zvinyorwa zvichiverengwa zvachose.
Zvakazoitika kuti zvizhinji zvezvinyorwa zvose zvakanyorwa uye zvinotsanangurwa pane imwe nzvimbo kubva 10 am kusvika 20 pm, i.e. pazuva rekushanda reMoscow. Izvi zvinogona kureva kuti Habr inoverengerwa zvinangwa zvehunyanzvi, uye kuti iyi inzira yakanaka yekuverengera pabasa. Nenzira, uku kugoverwa kwenguva yezuva kwakagadzikana kubva pahwaro chaihwo hwaHabr kusvika nhasi.
Nekudaro, iyo yakakosha kubatsirwa kweiyo timestamp yekutaura haisi nguva yezuva, asi nguva ye "hupenyu hunoshanda" hwechinyorwa. Ndakaverenga kuti nguva inogovewa sei kubva pakudhindwa kwechinyorwa kusvika pakutaura kwayo. Zvakazoitika kuti ikozvino chirevo chepakati (green line) chinouya mumaawa makumi maviri, i.e. pazuva rokutanga pashure pokubudiswa, paavhareji, inopfuura zvishoma hafu yetsinhiro dzose pamusoro pechinyorwa chasara. Uye mumazuva maviri vanosiya 20% yemashoko ese. Panguva imwecheteyo, zvinyorwa zvekare zvakaverengwa zvakakurumidza - somuenzaniso, muna 75, hafu yemashoko yakauya mumaawa matanhatu ekutanga.
Zvakauya sechishamiso kwandiri kuti zvirevo zvarebesa: avhareji yenhamba yevatambi mumhinduro yakada kupeta kaviri pahupenyu hwaHabr!
Mhinduro iri nyore pane mhinduro mavhoti. Kusiyana nezvimwe zvakawanda zviwanikwa, paHabré iwe unogona kuisa kwete chete pluses, asiwo minuses. Nekudaro, vaverengi havashandisi mukana wekupedzisira kazhinji: mugove wazvino wekusafarira ungangoita 15% yemavhoti ese akakandwa. Kwaimbove nezvakawanda, asi nekufamba kwenguva, vaverengi vave vane mutsa.
Magwaro pachawo akachinja nekufamba kwenguva. Semuenzaniso, kureba kwechinyorwa hakuregi kukura zvishoma nezvishoma kubva pakuvhurwa kwesaiti, kunyangwe paine matambudziko. Mumakore gumi, zvinyorwa zvave kureba zvakapetwa kagumi!
Manyorerwo ezvinyorwa (kune fungidziro yekutanga) akachinjawo. Mumakore ekutanga ekuvapo kwaHabr, semuenzaniso, chikamu chekodhi uye manhamba muzvinyorwa zvakawedzera:
Mushure mekunzwisisa kusimba kwese kwesaiti, ndakafunga kuyera kuti kufarirwa kwemisoro yakasiyana kwakachinja sei. Misoro inogona kusarudzwa otomatiki kubva muzvinyorwa, asi kune vanotanga, haugone kudzoreredza vhiri, asi shandisa akagadzirira-akagadzirwa ma tag akaiswa nevanyori vechinyorwa chimwe nechimwe. Ndadhirowa mafambiro mana pachati. Iyo "Google" theme yakatanga kutonga (zvichida zvakanyanya nekuda kweSEO), asi yanga ichirasikirwa nehuremu nekufamba kwemakore. Javascript yanga iri nyaya yakakurumbira uye inoramba ichikura zvishoma nezvishoma, asi kudzidza kwemichina kwakatanga kukurumidza kuwana mukurumbira mumakore achangopfuura. Linux, kune rumwe rutivi, yakaramba yakaenzana yakaenzana mumakore gumi ose.
Ehe, ndakava kufarira kuti ndedzipi misoro inokwezva vaverengi vakawanda. Ndakaverenga nhamba yepakati pemaonero, mavhoti uye makomendi mumusoro wega wega. Izvi ndizvo zvakaitika:
- Misoro inotariswa zvakanyanya: arduino, dhizaini yewebhu, kuvandudza kwewebhu, digest, zvinongedzo, css, html, html5, nginx, algorithms.
- Misoro "inonyanya kufarirwa": vkontakte, kuseka, jquery, opera, c, html, kuvandudzwa kwewebhu, html5, css, dhizaini yewebhu.
- Misoro inonyanya kukurukurwa: opera, skype, freelance, vkontakte, ubuntu, basa, nokia, nginx, arduino, firefox.
Nenzira, sezvo ndiri kuenzanisa misoro, unogona kuzvimisa ne frequency (uye enzanisa mhedzisiro ne ).
- Kwemakore ese ekuvapo kwaHabr, iwo anonyanya kufarirwa ma tag (mukudzika kurongeka) ndeye google, android, javascript, microsoft, linux, php, apple, java, python, programming, kutanga, budiriro, ios, kutanga, masocial network.
- Muna 2017, ainyanya kufarirwa aive javascript, python, java, android, budiriro, linux, c++, programming, php, c#, ios, kudzidza muchina, kuchengetedzwa kweruzivo, microsoft, react.
Kana tichienzanisa zviyero izvi, munhu anogona kuterera, semuenzaniso, kune kukunda kufora kwePython uye kutsakatika kwephp, kana "kuvira kwezuva" kwemisoro yekutanga uye kusimuka kwekudzidza kwemichina.
Haasi ese ma tag paHabré ane yakajeka thematic coloring. Semuenzaniso, heano ma tag gumi nemaviri akasangana kamwe chete, asi aingoita seanosetsa kwandiri. Saka: "pfungwa ndiyo simba rekufambisa rekufambira mberi", "bhutsu kubva pamufananidzo wefloppy", "Iowa State", "drama", "superalesh", "steam engine", "zvinhu zvekuita neMugovera", "ndine gava muguyo wenyama", "a zvakazoitika semazuva ese", "hatina kukwanisa kuuya nematagi anosetsa". Kuti uone musoro wezvinyorwa zvakadaro, ma tag haana kukwana - iwe uchafanirwa kuita thematic modeling pane zvinyorwa zvezvinyorwa.
Ongororo yakadzama yezviri muzvinyorwa ichave mune inotevera positi. Kutanga, ini ndichavaka modhi inofanotaura huwandu hwemapeji ekuona kwechinyorwa chinoenderana nezviri mukati. Kechipiri, ini ndinoda kudzidzisa iyo neural network kugadzira zvinyorwa nenzira yakafanana nevanyori veHabr. Saka nyoresa 🙂
PS Uye heino beep .
Source: www.habr.com