After finishing the New Year's Olivier salad, I had nothing to do, so I decided to download all the articles from Habr (and related platforms) to my computer for research.
I found several interesting plots. The first one is the development of article formats and themes over the 12 years of the site's existence. For example, the dynamics of certain topics are quite revealing. The continuation is below the cut.
Parsing Process
To understand how Habr developed, I needed to go through all its articles and extract metadata (like dates) from them. The traversal was easy, as links to all articles are in the format "habrahabr.ru/post/337722/", with the IDs being strictly sequential. Knowing that the last post has a number slightly less than 350,000, I simply iterated through all possible document IDs in a loop (code in Python):
import numpy as np
from multiprocessing import Pool
with Pool(100) as p:
docs = p.map(download_document, np.arange(350000))Function download_document tries to load the page with the corresponding ID and attempts to extract substantive information from the HTML structure.
import requests
from bs4 import BeautifulSoup
def download_document(pid):
""" Download and process a Habr document and its comments """
# Document download
r = requests.get('https://habrahabr.ru/post/' +str(pid) + '/')
# Document parsing
soup = BeautifulSoup(r.text, 'html5lib') # instead of html.parser
doc = {}
doc['id'] = pid
if not soup.find("span", {"class": "post__title-text"}):
# This happens if the article did not exist or was deleted
doc['status'] = 'title_not_found'
else:
doc['status'] = 'ok'
doc['title'] = soup.find("span", {"class": "post__title-text"}).text
doc['text'] = soup.find("div", {"class": "post__text"}).text
doc['time'] = soup.find("span", {"class": "post__time"}).text
# Create other fields: hubs, tags, views, comments, votes, etc.
# ...
# Saving the result to a separate file
fname = r'files/' + str(pid) + '.pkl'
with open(fname, 'wb') as f:
pickle.dump(doc, f)During the parsing process, I discovered several new insights.
Firstly, it is said that creating more processes than there are cores in the processor is useless. However, in my case, it turned out that the limiting resource was not the processor but the network, and 100 processes performed faster than 4 or, say, 20.
Secondly, some posts contained combinations of special characters — for example, euphemisms like "%&#@". It turned out that html.parser, which I initially used, reacts to the combination &# painfully, considering it the start of an HTML entity. I was about to resort to black magic, but a suggestion on the forum indicated that I could simply switch parsers.
Thirdly, I managed to export all publications except for three. My antivirus immediately deleted documents numbered 65927, 162075, and 275987. These articles are respectively about a JavaScript chain loading a malicious PDF, an SMS extortion scheme in the form of a set of browser plugins, and the site CrashSafari.com, which causes iPhones to reboot. Another article was detected later by the antivirus during a system scan: post 338586 about scripts on a pet store's website using the user's CPU for cryptocurrency mining. So, the antivirus can be considered quite effective.
"Live" articles turned out to be only half of the potential maximum—166,307. For the others, Habr gives options like "the page has expired, was deleted, or did not exist at all." Well, anything can happen.
After exporting the articles, I began the technical work: for example, publication dates needed to be converted from the format "Dec 21, '06 at 10:47" to a standard format, datetime, and "12.8k" views into 12,800. At this stage, several more quirks emerged. The most amusing was related to vote counting and data types: some older posts experienced an integer overflow, and they received 65,535 votes.
As a result, the article texts (without images) took up 1.5 gigabytes, comments with metadata—another 3 gigabytes, and about a hundred megabytes—metadata about the articles. This can be fully held in RAM, which was a pleasant surprise for me.
I began analyzing the articles not with the texts themselves, but with the metadata: dates, tags, hubs, views, and "likes." It turned out that it can reveal a lot.
Trends in the Development of Habr
Articles on the site have been published since 2006; most intensively from 2008 to 2016.
It's not so easy to assess how actively these articles were read at different times. Texts from 2012 and later received more comments and ratings, but newer texts have more views and bookmarks. These metrics only behaved similarly (dropped by half) once, in 2015. Perhaps, in the context of economic and political crises, readers' attention shifted from IT blogs to more pressing issues.
In addition to the articles themselves, I also extracted the comments on them. There were 6 million comments, though 240,000 of them turned out to be banned ("a UFO landed and published this message here"). A useful feature of the comments is that they have timestamps. By studying the timing of the comments, one can get a rough idea of when the articles are being read.
It turned out that the majority of articles are both written and commented on between 10 AM and 8 PM, that is, during a typical workday in Moscow. This may indicate that people read Habr for professional purposes, and it also serves as a good way to procrastinate at work. By the way, this distribution of reading times has remained stable from Habr's inception to the present day.
However, the main benefit of the comment timestamp is not the time of day, but the "active life" span of an article. I calculated the time distribution from the publication of an article to its first comment. It turned out that currently, the median comment (green line) appears roughly 20 hours later, meaning that in the first 24 hours after publication, a little over half of all comments on an article are left. After two days, 75% of all comments are made. In the past, articles were read even faster — for instance, in 2010, half of the comments came within the first 6 hours.
I was surprised to find that comments have lengthened: the average number of characters in a comment has nearly doubled over the existence of Habr!
A simpler form of feedback than comments is votes. Unlike many other platforms, on Habr, you can give both positive and negative votes. However, readers don't use the latter option as often; currently, the share of dislikes accounts for about 15% of all votes cast. This used to be higher, but over time readers have become more charitable.
The texts themselves have changed over time as well. For instance, the typical length of texts has consistently grown since the site launched, despite various crises. Over the decade, texts have become nearly ten times longer!
The stylistics of the texts (at a first glance) have also changed. In the early years of Habr's existence, for example, the proportion of code and numbers in the texts increased:
After analyzing the overall dynamics of the site, I decided to measure how the popularity of various topics had changed. Topics can be automatically identified from texts, but for starters, we can use the predefined tags set by the authors of each article. I highlighted four typical trends in the graph. The topic "Google" initially dominated (possibly mainly due to SEO optimization), but its significance waned each year. Javascript was a popular topic that continues to maintain gradual interest, while machine learning has only started to gain traction in recent years. Linux, however, remains consistently relevant throughout the decade.
Of course, I became curious about which topics attract the most reader engagement. I calculated the median number of views, votes, and comments for each topic. Here are the results:
- The most viewed topics: arduino, web design, web development, digest, links, css, html, html5, nginx, algorithms.
- The most "likable" topics: vk, humor, jquery, opera, c, html, web development, html5, css, web design.
- The most discussed topics: opera, skype, freelancing, vk, ubuntu, work, nokia, nginx, arduino, firefox.
By the way, since I'm comparing topics, I can rank them by frequency (and compare the results with ).
- Over the years of its existence, the most popular tags on Habr (in descending order) have been google, android, javascript, microsoft, linux, php, apple, java, python, programming, startups, development, ios, startup, social networks.
- In 2017, the most popular were javascript, python, java, android, development, linux, c++, programming, php, c#, ios, machine learning, information security, microsoft, react.
When comparing these rankings, one can note, for example, the victorious march of Python and the decline of php, or the "twilight" of startup themes and the rise of machine learning.
Not all tags on Habr have such an obvious thematic implication. For example, here are a dozen tags that appeared only once but I found amusing. So: "idea is the driving force of progress", "booting from a floppy disk image", "state of Iowa", "drama", "super-alesha", "steam engine", "what to do on Saturday", "I have a fox in the meat grinder", "but it turned out as always", "couldn't come up with funny tags". To determine the theme of such articles, tags alone are insufficient — thematic modeling on the texts of the articles will be necessary.
A more detailed analysis of the content of the articles will be in the next post. First, I plan to build a model predicting the number of views of an article based on its content. Secondly, I would like to teach the neural network to generate texts in the same style as the authors of Habr. So, subscribe 🙂
P.S. And here is the redacted .
Source: habr.com
