🥇All Habr kusizindalwazi esisodwa

Sawubona. Sekuphele iminyaka emi-2 yabhalwa. isihloko sokugcina mayelana nokwehlukanisa uHabr, futhi amanye amaphuzu ashintshile.

Lapho ngifuna ukuba nekhophi ka-Habr, nganquma ukubhala umhlahleli ozogcina konke okuqukethwe ababhali kusizindalwazi. Kwenzeka kanjani futhi yiziphi iziphambeko engihlangabezane nazo - ungafunda ngaphansi kokusika.

I-TLDR- isixhumanisi sesizindalwazi

Inguqulo yokuqala yomhlahleli. Uchungechunge olulodwa, izinkinga eziningi

Okokuqala nje, nginqume ukwenza i-prototype yeskripthi, lapho isihloko sizocutshungulwa ngokushesha lapho silanda futhi sibekwe kusizindalwazi. Ngaphandle kokucabanga kabili, ngasebenzisa i-sqlite3, ngoba. bekunzima kakhulu ukusebenza: asikho isidingo sokuba neseva yendawo, okubukeka sengathi kudaliwe kanye nezinto ezinjalo.

one_thread.py

from bs4 import BeautifulSoup
import sqlite3
import requests
from datetime import datetime

def main(min, max):
    conn = sqlite3.connect('habr.db')
    c = conn.cursor()
    c.execute('PRAGMA encoding = "UTF-8"')
    c.execute("CREATE TABLE IF NOT EXISTS habr(id INT, author VARCHAR(255), title VARCHAR(255), content  TEXT, tags TEXT)")

    start_time = datetime.now()
    c.execute("begin")
    for i in range(min, max):
        url = "https://m.habr.com/post/{}".format(i)
        try:
            r = requests.get(url)
        except:
            with open("req_errors.txt") as file:
                file.write(i)
            continue
        if(r.status_code != 200):
            print("{} - {}".format(i, r.status_code))
            continue

        html_doc = r.text
        soup = BeautifulSoup(html_doc, 'html.parser')

        try:
            author = soup.find(class_="tm-user-info__username").get_text()
            content = soup.find(id="post-content-body")
            content = str(content)
            title = soup.find(class_="tm-article-title__text").get_text()
            tags = soup.find(class_="tm-article__tags").get_text()
            tags = tags[5:]
        except:
            author,title,tags = "Error", "Error {}".format(r.status_code), "Error"
            content = "При парсинге этой странице произошла ошибка."

        c.execute('INSERT INTO habr VALUES (?, ?, ?, ?, ?)', (i, author, title, content, tags))
        print(i)
    c.execute("commit")
    print(datetime.now() - start_time)

main(1, 490406)

Yonke into ingeyakudala - sisebenzisa Isobho Elihle, izicelo kanye nesibonelo esisheshayo sekulungile. Lokho nje...

Ukulanda ikhasi kuchungechunge olulodwa
Uma uphazamisa ukusebenza kweskripthi, khona-ke yonke imininingwane egciniwe ngeke iye ndawo. Phela, ukuzibophezela kwenziwa kuphela ngemva kokwahlukanisa konke.
Yebo, ungenza izinguquko kusizindalwazi ngemva kokufaka ngakunye, kodwa-ke isikhathi sokwenza iskripthi sizokhula kakhulu.
Ukuhlaziya izindatshana zokuqala eziyi-100 kungithathe amahora ayi-000.

Okulandelayo ngithola isihloko somsebenzisi kuhlanganiswe, engikufundile ngathola ama-hacks ambalwa empilo ukusheshisa le nqubo:

Ukusebenzisa i-multithreading kuyasheshisa ukulanda ngezinye izikhathi.
Awukwazi ukuthola inguqulo ephelele ye-habr, kodwa inguqulo yayo yeselula.
Isibonelo, uma i-athikili ehlanganisiwe enguqulweni yedeskithophu inesisindo esingu-378 KB, khona-ke enguqulweni yeselula isivele ingu-126 KB.

Inguqulo yesibili. Imicu eminingi, ukuvinjelwa kwesikhashana okuvela ku-Habr

Lapho ngihlola i-intanethi esihlokweni sokufunda okuningi ku-python, ngakhetha inketho elula nge-multiprocessing.dummy, ngabona ukuthi izinkinga zivela kanye ne-multithreading.

I-SQLite3 ayifuni ukusebenza ngochungechunge olungaphezu kowodwa.
kulungisiwe check_same_thread=False, kodwa leli phutha akulona lodwa, lapho uzama ukufaka ku-database, ngezinye izikhathi kwenzeka amaphutha engingakwazi ukuwaxazulula.

Ngakho-ke, nginquma ukushiya ukufakwa ngokushesha kwezihloko ngqo ku-database futhi, ngikhumbula isisombululo esihlanganisiwe, nginquma ukusebenzisa amafayela, ngoba azikho izinkinga ngokubhala okunezintambo eziningi efayeleni.

U-Habr uqala ukuvimba ngokusebenzisa imicu engaphezu kwemithathu.
Imizamo yentshiseko yokudlulela ku-Habr ingagcina ngokuvinjelwa kwe-ip amahora ambalwa. Ngakho-ke kufanele usebenzise imicu emi-3 kuphela, kodwa lokhu sekuvele kuhle, njengoba isikhathi sokuphindaphinda ama-athikili angaphezu kwe-100 sincishisiwe sisuka kumasekhondi angama-26 kuye kwayi-12.

Kuyaqapheleka ukuthi le nguqulo ayizinzile, futhi ukulanda ngezikhathi ezithile kwehla enanini elikhulu lama-athikili.

async_v1.py

from bs4 import BeautifulSoup
import requests
import os, sys
import json
from multiprocessing.dummy import Pool as ThreadPool
from datetime import datetime
import logging

def worker(i):
    currentFile = "files\{}.json".format(i)

    if os.path.isfile(currentFile):
        logging.info("{} - File exists".format(i))
        return 1

    url = "https://m.habr.com/post/{}".format(i)

    try: r = requests.get(url)
    except:
        with open("req_errors.txt") as file:
            file.write(i)
        return 2

    # Запись заблокированных запросов на сервер
    if (r.status_code == 503):
        with open("Error503.txt", "a") as write_file:
            write_file.write(str(i) + "n")
            logging.warning('{} / 503 Error'.format(i))

    # Если поста не существует или он был скрыт
    if (r.status_code != 200):
        logging.info("{} / {} Code".format(i, r.status_code))
        return r.status_code

    html_doc = r.text
    soup = BeautifulSoup(html_doc, 'html5lib')

    try:
        author = soup.find(class_="tm-user-info__username").get_text()

        timestamp = soup.find(class_='tm-user-meta__date')
        timestamp = timestamp['title']

        content = soup.find(id="post-content-body")
        content = str(content)
        title = soup.find(class_="tm-article-title__text").get_text()
        tags = soup.find(class_="tm-article__tags").get_text()
        tags = tags[5:]

        # Метка, что пост является переводом или туториалом.
        tm_tag = soup.find(class_="tm-tags tm-tags_post").get_text()

        rating = soup.find(class_="tm-votes-score").get_text()
    except:
        author = title = tags = timestamp = tm_tag = rating = "Error" 
        content = "При парсинге этой странице произошла ошибка."
        logging.warning("Error parsing - {}".format(i))
        with open("Errors.txt", "a") as write_file:
            write_file.write(str(i) + "n")

    # Записываем статью в json
    try:
        article = [i, timestamp, author, title, content, tm_tag, rating, tags]
        with open(currentFile, "w") as write_file:
            json.dump(article, write_file)
    except:
        print(i)
        raise

if __name__ == '__main__':
    if len(sys.argv) < 3:
        print("Необходимы параметры min и max. Использование: async_v1.py 1 100")
        sys.exit(1)
    min = int(sys.argv[1])
    max = int(sys.argv[2])

    # Если потоков >3
    # то хабр банит ipшник на время
    pool = ThreadPool(3)

    # Отсчет времени, запуск потоков
    start_time = datetime.now()
    results = pool.map(worker, range(min, max))

    # После закрытия всех потоков печатаем время
    pool.close()
    pool.join()
    print(datetime.now() - start_time)

Inguqulo yesithathu. Okokugcina

Ngenkathi ngilungisa iphutha lenguqulo yesibili, ngathola ukuthi u-Habr, kungazelelwe, une-API efinyelelwa yinguqulo yeselula yesayithi. Ilayisha ngokushesha kunenguqulo yeselula, njengoba iyi-json nje, engadingi ngisho nokwahlukaniswa. Ekugcineni, nginqume ukubhala kabusha isikripthi sami futhi.

Ngakho, ngemva kokuthola lesi sixhumanisi API, ungaqala ukuyihlaziya.

async_v2.py

import requests
import os, sys
import json
from multiprocessing.dummy import Pool as ThreadPool
from datetime import datetime
import logging

def worker(i):
    currentFile = "files\{}.json".format(i)

    if os.path.isfile(currentFile):
        logging.info("{} - File exists".format(i))
        return 1

    url = "https://m.habr.com/kek/v1/articles/{}/?fl=ru%2Cen&hl=ru".format(i)

    try:
        r = requests.get(url)
        if r.status_code == 503:
            logging.critical("503 Error")
            return 503
    except:
        with open("req_errors.txt") as file:
            file.write(i)
        return 2

    data = json.loads(r.text)

    if data['success']:
        article = data['data']['article']

        id = article['id']
        is_tutorial = article['is_tutorial']
        time_published = article['time_published']
        comments_count = article['comments_count']
        lang = article['lang']
        tags_string = article['tags_string']
        title = article['title']
        content = article['text_html']
        reading_count = article['reading_count']
        author = article['author']['login']
        score = article['voting']['score']

        data = (id, is_tutorial, time_published, title, content, comments_count, lang, tags_string, reading_count, author, score)
        with open(currentFile, "w") as write_file:
            json.dump(data, write_file)

if __name__ == '__main__':
    if len(sys.argv) < 3:
        print("Необходимы параметры min и max. Использование: asyc.py 1 100")
        sys.exit(1)
    min = int(sys.argv[1])
    max = int(sys.argv[2])

    # Если потоков >3
    # то хабр банит ipшник на время
    pool = ThreadPool(3)

    # Отсчет времени, запуск потоков
    start_time = datetime.now()
    results = pool.map(worker, range(min, max))

    # После закрытия всех потоков печатаем время
    pool.close()
    pool.join()
    print(datetime.now() - start_time)

Iqukethe izinkambu ezihlobene kokubili isihloko ngokwaso kanye nombhali oyibhalile.

I-API.png

Angizange ngilahle i-json ephelele ye-athikili ngayinye, kodwa ngilondoloze izinkambu engangizidinga kuphela:

id
i_imfundiso
isikhathi_okushicilelwe
isihloko
sokuqukethwe
ukubalwa_kwamazwana
isiLang ulimi okubhalwe ngalo isihloko. Kuze kube manje, ine-en ne-ru kuphela.
tags_string - wonke amathegi asuka kokuthunyelwe
ukufunda_ukubala
umbhali
amaphuzu - isilinganiso sesihloko.

Ngakho, ngisebenzisa i-API, ngehlise isikhathi sokwenza iskripthi saba imizuzwana eyi-8 nge-url eyi-100 ngayinye.

Ngemva kokulanda idatha esiyidingayo, sidinga ukuyicubungula futhi siyifake kusizindalwazi. Angizange ngibe nezinkinga nalokhu:

umhlaseli.py

import json
import sqlite3
import logging
from datetime import datetime

def parser(min, max):
    conn = sqlite3.connect('habr.db')
    c = conn.cursor()
    c.execute('PRAGMA encoding = "UTF-8"')
    c.execute('PRAGMA synchronous = 0') # Отключаем подтверждение записи, так скорость увеличивается в разы.
    c.execute("CREATE TABLE IF NOT EXISTS articles(id INTEGER, time_published TEXT, author TEXT, title TEXT, content TEXT, 
    lang TEXT, comments_count INTEGER, reading_count INTEGER, score INTEGER, is_tutorial INTEGER, tags_string TEXT)")
    try:
        for i in range(min, max):
            try:
                filename = "files\{}.json".format(i)
                f = open(filename)
                data = json.load(f)

                (id, is_tutorial, time_published, title, content, comments_count, lang,
                 tags_string, reading_count, author, score) = data

                # Ради лучшей читаемости базы можно пренебречь читаемостью кода. Или нет?
                # Если вам так кажется, можно просто заменить кортеж аргументом data. Решать вам.

                c.execute('INSERT INTO articles VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)', (id, time_published, author,
                                                                                        title, content, lang,
                                                                                        comments_count, reading_count,
                                                                                        score, is_tutorial,
                                                                                        tags_string))
                f.close()

            except IOError:
                logging.info('FileNotExists')
                continue

    finally:
        conn.commit()

start_time = datetime.now()
parser(490000, 490918)
print(datetime.now() - start_time)

Izibalo

Nokho, ngokwesiko, ekugcineni, ungakhipha izibalo ezithile kudatha:

Kokulandiwe okungu-490 okulindelekile, yizindatshana ezingama-406 kuphela ezilandiwe. Kuvela ukuthi ngaphezu kwengxenye (228) yezindatshana ezikhuluma nge-Habré zazifihliwe noma zasuswa.
Yonke imininingwane egciniwe, ehlanganisa cishe ingxenye yesigidi, inesisindo esingu-2.95 GB. Ngefomu elicindezelwe - 495 MB.
Sebebonke, abantu abangama-37804 bangababhali be-Habré. Ngiyakukhumbuza ukuthi lezi zibalo zivela kokuthunyelwe bukhoma kuphela.
Umbhali okhiqiza kakhulu ku-Habré - u-alizar - 8774 izihloko.
Isihloko esilinganiselwe phezulu - 1448 ama-plus
Isihloko esifundwa kakhulu - 1660841 ukubukwa
I-athikili Okuxoxwe Kakhulu - Amazwana angama-2444

Well, ngendlela eziqongweniAbabhali abaphezulu abayi-15
Top 15 ngokulinganisa
Okuphezulu okungu-15 kufundwe
Top 15 Okuxoxiwe

Source: www.habr.com

Zonke i-Habr ku-database eyodwa

Inguqulo yokuqala yomhlahleli. Uchungechunge olulodwa, izinkinga eziningi

Inguqulo yesibili. Imicu eminingi, ukuvinjelwa kwesikhashana okuvela ku-Habr

Inguqulo yesithathu. Okokugcina

Izibalo

Engeza amazwana Отменить ответ