I ka pau ʻana o ka makahiki hou ʻo Olivier, ʻaʻohe aʻu mea e hana ai, a ua hoʻoholo wau e hoʻoiho i nā ʻatikala a pau mai Habrahabr (a me nā kahua pili) i kaʻu kamepiula a e ʻimi.
He mau moʻolelo hoihoi. ʻO ka mua o lākou ka hoʻomohala ʻana i ke ʻano a me nā kumuhana o nā ʻatikala ma luna o 12 mau makahiki o ka noho ʻana o ka pūnaewele. No ka laʻana, ʻike ʻia ka dynamics o kekahi mau kumuhana. Hoʻomau - ma lalo o ka ʻoki.
Kaʻina hoʻokaʻawale
No ka hoʻomaopopo ʻana i ke ʻano o ka hoʻomohala ʻana o Habr, pono e hele i kāna mau ʻatikala āpau a unuhi i ka ʻike-meta mai lākou (e laʻa, nā lā). He mea maʻalahi ka hele ʻana, no ka mea, ua like nā loulou i nā ʻatikala a pau me "habrahabr.ru/post/337722/", a ua hāʻawi ʻia nā helu ma ke ʻano pololei. I ka ʻike ʻana he helu liʻiliʻi ka helu o ka pou hope ma mua o 350 tausani, ua hele wale wau i nā palapala id āpau i hiki i kahi loop (Python code):
import numpy as np
from multiprocessing import Pool
with Pool(100) as p:
docs = p.map(download_document, np.arange(350000))kuleana pili i download_document hoʻāʻo e hoʻouka i ka ʻaoʻao me ka id pili a hoʻāʻo e unuhi i ka ʻike koʻikoʻi mai ka hoʻolālā html.
import requests
from bs4 import BeautifulSoup
def download_document(pid):
""" Download and process a Habr document and its comments """
# выгрузка документа
r = requests.get('https://habrahabr.ru/post/' +str(pid) + '/')
# парсинг документа
soup = BeautifulSoup(r.text, 'html5lib') # instead of html.parser
doc = {}
doc['id'] = pid
if not soup.find("span", {"class": "post__title-text"}):
# такое бывает, если статья не существовала или удалена
doc['status'] = 'title_not_found'
else:
doc['status'] = 'ok'
doc['title'] = soup.find("span", {"class": "post__title-text"}).text
doc['text'] = soup.find("div", {"class": "post__text"}).text
doc['time'] = soup.find("span", {"class": "post__time"}).text
# create other fields: hubs, tags, views, comments, votes, etc.
# ...
# сохранение результата в отдельный файл
fname = r'files/' + str(pid) + '.pkl'
with open(fname, 'wb') as f:
pickle.dump(doc, f)I ke kaʻina hana ʻana, ua ʻike au i nā mea hou.
ʻO ka mea mua, ʻōlelo lākou ʻaʻole pono ka hana ʻana i nā kaʻina hana ma mua o nā cores. Akā i koʻu hihia, ua ʻike ʻia ʻaʻole ʻo ke kaʻina hana ka palena palena, akā ʻo ka pūnaewele, a ʻoi aku ka wikiwiki o nā kaʻina 100 ma mua o 4 a i ʻole, e ʻōlelo, 20.
ʻO ka lua, ma kekahi mau pou i hui pū ʻia nā huaʻōlelo kūikawā - no ka laʻana, nā euphemisms e like me "%&#@". ʻIke ʻia kēlā html.parser, ka mea aʻu i hoʻohana mua ai, pane i ka hui &# ʻeha, e noʻonoʻo ana i ka hoʻomaka ʻana o ka mea html. Ua hele mua wau e hana ʻeleʻele, akā ua ʻōlelo ka ʻaha kūkā e hiki iā ʻoe ke hoʻololi i ka parser.
ʻO ke kolu, ua hiki iaʻu ke wehe i nā puke a pau, koe wale nō ʻekolu. Ua holoi koke ʻia nā palapala i helu ʻia 65927, 162075, a me 275987 e kaʻu antivirus. ʻO kēia nā ʻatikala, kēlā me kēia, e pili ana i kahi kaulahao javascript e hoʻoiho i kahi pdf maikaʻi ʻole, SMS ransomware ma ke ʻano o kahi pūʻulu polokalamu kele pūnaewele, a me ka pūnaewele CrashSafari.com e hoʻouna ana i nā iPhones i kahi reboot. Ua ʻike ʻo Antivirus i kekahi ʻatikala ma hope aku, i ka wā o ka scan system: post 338586 e pili ana i nā palapala ma ka pūnaewele hale kūʻai holoholona e hoʻohana ana i ka mea hoʻohana i ka mea hoʻohana i kaʻu cryptocurrency. No laila hiki iā mākou ke noʻonoʻo i ka hana o ka antivirus ua lawa.
ʻO nā ʻatikala "Live" i lilo i hapalua wale nō o ka nui loa - 166307 mau ʻāpana. E pili ana i ke koena, hāʻawi ʻo Habr i nā koho "ua pau ka ʻaoʻao, ua holoi ʻia a ʻaʻole i loaʻa iki." ʻAe, hiki i kekahi mea ke hiki.
Ua ukali ʻia ka hoʻouka ʻana o nā ʻatikala e nā hana ʻenehana: no ka laʻana, pono e hoʻololi ʻia nā lā hoʻolaha mai ke ʻano "'21 Kekemapa 2006 ma 10:47 am" i ka maʻamau datetime, a me nā manaʻo "12,8k" - i ka makahiki 12800. I kēia manawa, ua puka mai kekahi mau hanana hou. ʻO ka mea leʻaleʻa e pili ana i nā helu balota a me nā ʻano ʻikepili: ua nui ka int o kekahi mau pou kahiko a loaʻa iā 65535 mau balota i kēlā me kēia.
ʻO ka hopena, ʻo nā kikokikona o nā ʻatikala (me nā kiʻi ʻole) ua lawe iaʻu i 1.5 gigabytes, nā ʻōlelo me ka ʻike-meta - ʻo 3 ʻē aʻe, a ma kahi o hoʻokahi haneli megabytes - ʻike-meta e pili ana i nā ʻatikala. Hiki ke mālama ʻia kēia i loko o ka RAM, he mea ʻoluʻolu iaʻu.
Ua hoʻomaka wau i ka nānā ʻana i nā ʻatikala ʻaʻole mai nā kikokikona iā lākou iho, akā mai ka ʻike-meta: nā lā, nā inoa, nā hubs, nā manaʻo a me nā makemake. Ua ʻike ʻia ua hiki iā ia ke haʻi nui.
Nā Kūlana Hoʻomohala Habrahabr
Ua paʻi ʻia nā ʻatikala ma ka pūnaewele mai ka makahiki 2006; ikaika loa - ma 2008-2016.
ʻAʻole maʻalahi ka nānā ʻana i ka ikaika o kēia mau ʻatikala i nā manawa like ʻole. Loaʻa i nā kikokikona mai ka makahiki 2012 a ʻoi aku ka nui o nā manaʻo a me nā helu, akā ʻoi aku ka nui o nā manaʻo a me nā kaha puke i nā kikokikona hou. Hoʻokahi wale nō hana o kēia mau ana i ka 2015. Malia paha, i ke kūlana o ka hoʻokele waiwai a me ka politika, ua neʻe ka manaʻo o ka poʻe heluhelu mai nā blog IT i nā pilikia ʻeha.
Ma waho aʻe o nā ʻatikala ponoʻī, ua hoʻoiho au i nā manaʻo hou aku iā lākou. Aia he 6 miliona mau manaʻo, akā naʻe, ua pāpā ʻia ka 240 tausani o lākou ("lele ka UFO a paʻi i kēia palapala ma aneʻi"). ʻO kahi waiwai kūpono o nā manaʻo he loaʻa iā lākou kahi hōʻailona manawa. Ma ke aʻo ʻana i ka manawa o nā manaʻo, hiki iā ʻoe ke hoʻomaopopo i ka wā e heluhelu ʻia ai nā ʻatikala.
Ua ʻike ʻia ʻo ka hapa nui o nā ʻatikala i kākau ʻia a ʻōlelo ʻia ma kekahi wahi mai 10 am a 20 pm, ʻo ia hoʻi. ma kahi lā hana maʻamau o Moscow. ʻO ia paha ke ʻano he heluhelu ʻia ʻo Habr no nā kumu ʻoihana, a he ala maikaʻi kēia e hoʻopaneʻe ai i ka hana. Ma ke ala, ua paʻa kēia māhele o ka manawa o ka lā mai ka hoʻokumu ʻana o Habr a hiki i kēia lā.
Eia naʻe, ʻo ka pōmaikaʻi nui o kahi hōʻike timestamp ʻaʻole ia ka manawa o ka lā, akā ʻo ka lōʻihi o ke "ola ola" o ka ʻatikala. Ua helu au i ka mahele ʻana o ka manawa mai ka paʻi ʻana o ka ʻatikala a hiki i kāna ʻōlelo. Ua ʻike ʻia i kēia manawa hiki mai ka ʻōlelo waena (laina ʻōmaʻomaʻo) ma kahi o 20 mau hola, ʻo ia hoʻi. ma ka lā mua ma hope o ka hoʻolaha ʻana, ma ka awelika, ʻoi aku ka hapa o nā ʻōlelo a pau ma ka ʻatikala. A i loko o ʻelua lā haʻalele lākou i ka 75% o nā ʻōlelo a pau. I ka manawa like, ua heluhelu wikiwiki ʻia nā ʻatikala mua - no ka laʻana, ma 2010, hiki mai ka hapalua o nā manaʻo i nā hola 6 mua.
He mea kupanaha iaʻu ua hoʻolōʻihi ʻia nā manaʻo: ʻo ka awelika o nā huaʻōlelo i loko o kahi manaʻo ua aneane pālua ʻia i ka wā o Habr!
ʻOi aku ka maʻalahi o nā manaʻo ma mua o nā manaʻo. ʻAʻole like me nā kumuwaiwai ʻē aʻe, ma Habré hiki iā ʻoe ke kau i nā mea hoʻohui wale nō, akā i nā minuses. Eia nō naʻe, ʻaʻole hoʻohana pinepine ka poʻe heluhelu i ka manawa hope loa: ʻo ka hapa o nā makemake ʻole i kēia manawa ma kahi o 15% o nā balota āpau. Ua nui aʻe ma mua, akā i ka manawa, ua ʻoi aku ka lokomaikaʻi o ka poʻe heluhelu.
Ua loli nā kikokikona i ka manawa. No ka laʻana, ʻaʻole e pau ka lōʻihi maʻamau o ka kikokikona i ka ulu mau ʻana mai ka hoʻomaka ʻana o ka pūnaewele, ʻoiai nā pilikia. I loko o nā makahiki he ʻumi, ua ʻoi aku ka lōʻihi o nā kikokikona!
Ua loli pū ke ʻano o nā kikokikona (i ka hoʻohālikelike mua). I loko o nā makahiki mua o ka noho ʻana o Habr, no ka laʻana, ua hoʻonui ka māhele o nā code a me nā helu i nā kikokikona:
Ma hope o ka hoʻomaopopo ʻana i ka dynamics holoʻokoʻa o ka pūnaewele, ua hoʻoholo wau e ana i ka loli ʻana o ka kaulana o nā kumuhana like ʻole. Hiki ke koho ʻia nā kumuhana mai nā kikokikona, akā no ka hoʻomaka ʻana, ʻaʻole hiki iā ʻoe ke hana hou i ka huila, akā e hoʻohana i nā hōʻailona mākaukau i hoʻopili ʻia e nā mea kākau o kēlā me kēia ʻatikala. Ua huki au i ʻehā mau ʻano maʻamau ma ka pakuhi. ʻO ke kumumanaʻo "Google" i hoʻomaka mua ʻia (ma muli paha o SEO), akā ua nalowale ke kaumaha i nā makahiki. He kumuhana kaulana ʻo Javascript a ke hoʻomau nei ka ulu mālie, akā ua hoʻomaka ke aʻo ʻana i ka mīkini e loaʻa koke ka kaulana i nā makahiki i hala iho nei. ʻO Linux, ma ka ʻaoʻao ʻē aʻe, ua paʻa like ka pili i nā makahiki he ʻumi.
ʻOiaʻiʻo, ua hoihoi au i nā kumuhana e hoʻonui i ka hana heluhelu. Ua helu au i ka helu waena o nā manaʻo, nā koho a me nā manaʻo i kēlā me kēia kumuhana. Eia ka mea i hanaia:
- Nā kumuhana i nānā nui ʻia: arduino, hoʻolālā pūnaewele, hoʻomohala pūnaewele, digest, loulou, css, html, html5, nginx, algorithms.
- ʻO nā kumuhana "makemake" loa: vkontakte, humor, jquery, opera, c, html, hoʻomohala pūnaewele, HTML5, css, hoʻolālā pūnaewele.
- ʻO nā kumuhana i kūkākūkā nui ʻia: opera, skype, freelance, vkontakte, ubuntu, hana, nokia, nginx, arduino, firefox.
Ma ke ala, ʻoiai ke hoʻohālikelike nei au i nā kumuhana, hiki iā ʻoe ke hoʻonohonoho iā lākou ma ke alapine (a hoʻohālikelike i nā hopena me ).
- No nā makahiki a pau o ka noho ʻana o Habr, ʻo nā inoa kaulana loa (ma ke ʻano iho) he google, android, javascript, microsoft, linux, php, apple, java, python, programming, startups, development, ios, startup, social networks.
- I ka makahiki 2017, ka mea kaulana loa he javascript, python, java, android, development, linux, c++, programming, php, c#, ios, machine learning, information security, microsoft, react
I ka hoʻohālikelike ʻana i kēia mau helu, hiki i kekahi ke hoʻolohe, no ka laʻana, i ka huakaʻi lanakila o Python a me ka pau ʻana o php, a i ʻole "ka lā" o nā kumuhana hoʻomaka a me ka piʻi ʻana o ke aʻo ʻana i ka mīkini.
ʻAʻole i loaʻa i nā hōʻailona a pau ma Habré ke kala ʻano like ʻole. Eia kekahi laʻana, eia nā hōʻailona he ʻumi i hui ʻia i hoʻokahi wale nō, akā he mea ʻakaʻaka iaʻu. No laila: "ʻo ka manaʻo ka mea hoʻoikaika o ka holomua", "boot mai kahi kiʻi floppy disk", "Iowa State", "drama", "superalesh", "mekini mokuahi", "nā mea e hana ai i ka Pōʻaono", "Ua loaʻa iaʻu. he alopeke i loko o ka mea wili ʻiʻo", "a ua hoʻololi ʻia e like me nā manawa a pau", "ʻaʻole hiki iā mākou ke hele mai me nā hōʻailona ʻakaʻaka". No ka hoʻoholo ʻana i ke kumuhana o ia mau ʻatikala, ʻaʻole lawa nā hōʻailona - pono ʻoe e hoʻokō i ka hoʻohālike thematic ma nā kikokikona o nā ʻatikala.
Aia ka ʻike kikoʻī o ka ʻatikala ma ka pou aʻe. ʻO ka mea mua, e kūkulu wau i kumu hoʻohālike e wānana i ka helu o nā ʻaoʻao ʻaoʻao no kahi ʻatikala e pili ana i kāna ʻike. ʻO ka lua, makemake wau e aʻo i ka neural network e hana i nā kikokikona ma ke ʻano like me nā mea kākau o Habr. No laila e kau inoa 🙂
PS A eia ka beep .
Source: www.habr.com