Bypassing the LinkedIn search limit by playing with the API

Limit

There is such a limitation on LinkedIn - Commercial use limit. It is highly likely that you, like me until recently, have never encountered or heard of him.

The essence of the limit is that if you use the search for people outside your contacts too often (there are no exact metrics, the algorithm decides, based on your actions - how often and how much you searched, added people), then the search result will be limited to three profiles, instead of 1000 ( default 100 pages, 10 profiles per page). The limit resets at the beginning of each month. Naturally, Premium accounts do not have this limitation..

But not so long ago, for a pet project, I started playing around with LinkedIn search and suddenly got this limitation. Naturally, I didn't really like it, because I did not use it for any commercial purposes, so my first thought was to study the limitation and try to work around it.

[An important clarification - the materials in the article are presented for informational and educational purposes only. The author does not encourage their use for commercial purposes.]

We study the problem

We have: instead of ten profiles with pagination, the search returns only three, after which a block with a “recommendation” of a premium account is inserted and blurry and non-clickable profiles go below.

Immediately, the hand reaches into the developer console to see these hidden profiles - perhaps we can remove some styles that put a blur, or extract information from a block in the markup. But, quite expectedly, these profiles are just placeholder pictures and no information is stored.

Okay, now let's take a look at the Network tab and check if the alternative search results really work, returning only three profiles. We find the request we are interested in to “/api/search/blended” and look at the response.

Profiles come in the `included` array, but there are already 15 entities in it. In this case, the first three of them are objects with additional information, each object contains information on a specific profile (for example, whether the profile is premium).

The next 12 are real profiles - search results, of which only three will be shown to us. As you might guess, it shows only those who receive additional information (the first three objects). For example, if you take an answer from a profile without a limit, then 28 entities will come - 10 objects with additional. information and 18 profiles.

Answer for profile without limit

Why there are more than 10 profiles, although exactly 10 are requested, and they do not participate in the display in any way, even on the next page they will not be - I don’t know yet. If you analyze the request URL, you can see that count=10 (how many profiles to return in the response, maximum 49).

I would appreciate any comments on this.

Experimenting

Well, the most important thing we now know for sure is that there are more profiles in the answer than they show us. So we can get more data, despite the limit. Let's try to pull the api ourselves, directly from the console, using fetch.

As expected, we get an error, 403. This is due to security, we are not sending a CSRF token here (CSRF on Wikipedia. In a nutshell, a unique token is added to each request, which is checked on the server for authenticity).

It can be copied from any other successful request or from cookies, where it is stored in the 'JSESSIONID' field.

Where to find the tokenHeader of another request:

Or from cookies, directly through the console:

Let's try again, this time we pass the settings to fetch, in which we specify our csrf-token as a parameter in the header.

Success, we receive all 10 profiles. :tada:

Due to the difference in headers, the structure of the response is slightly different from what comes in the original request. We can get the same structure by adding 'Accept: 'application/vnd.linkedin.normalized+json+2.1' to our object, next to the csrf token.
Example of a response with an added header

More about the Accept header

What's next?

Then you can edit (manually or automate) the `start` parameter, indicating the index, starting from which we will be given 10 profiles (default = 0) from the entire search result. In other words, incrementing it by 10 after each request, we get the usual paging, 10 profiles at a time.

At this stage, I had enough data and freedom to continue working on the pet project. But it was a sin not to try to display this data right on the spot, since they are on hand. In Ember, which is used at the front, we will not climb. jQuery was connected to the site, and having dug up knowledge of the basic syntax in memory, you can create the following in a couple of minutes.

jQuery Code

/* рендер блока, принимаем данные профиля и вставляем блок в список профилей используя эти данные */
const  createProfileBlock = ({ headline, publicIdentifier, subline, title }) => {
    $('.search-results__list').append(
        `<li class="search-result search-result__occluded-item ember-view">
            <div class="search-entity search-result search-result--person search-result--occlusion-enabled ember-view">
                <div class="search-result__wrapper">
                    <div class="search-result__image-wrapper">
                        <a class="search-result__result-link ember-view" href="/en/in/${publicIdentifier}/">
                            <figure class="search-result__image">
                                <div class="ivm-image-view-model ember-view">
                                    <img class="lazy-image ivm-view-attr__img--centered EntityPhoto-circle-4  presence-entity__image EntityPhoto-circle-4 loaded" src="http://www.userlogos.org/files/logos/give/Habrahabr3.png" />
                                </div>
                            </figure>
                        </a>
                    </div>
                    
                    <div class="search-result__info pt3 pb4 ph0">
                        <a class="search-result__result-link ember-view" href="/en/in/${publicIdentifier}/">
                            <h3 class="actor-name-with-distance search-result__title single-line-truncate ember-view">
                                ${title.text}
                            </h3>
                        </a>

                        <p class="subline-level-1 t-14 t-black t-normal search-result__truncate">${headline.text}</p>

                        <p class="subline-level-2 t-12 t-black--light t-normal search-result__truncate">${subline.text}</p>
                    </div>
                </div>
            </div>
        <li>`
    );
};

// дергаем апи, получаем данные и рендерим профили
const fetchProfiles = () => {
    // токен
   const csrf = 'ajax:9082932176494192209';
    
   // объект с настройками запроса, передаем токен
   const settings = { headers: { 'csrf-token': csrf } }

    // урл запроса, с динамическим индексом старта в конце
   const url = `https://www.linkedin.com/voyager/api/search/blended?count=10&filters=List(geoRegion-%3Ejp%3A0,network-%3ES,resultType-%3EPEOPLE)&origin=FACETED_SEARCH&q=all&queryContext=List(spellCorrectionEnabled-%3Etrue,relatedSearchesEnabled-%3Etrue)&start=${nextItemIndex}`; 
    /* делаем запрос, для каждого профиля в ответе вызываем рендер блока, и после инкрементируем стартовый индекс на 10 */
    fetch(url, settings).then(response => response.json()).then(data => {
        data.elements[0].elements.forEach(createProfileBlock);
        nextItemIndex += 10;
});
};


// удаляем все профили из списка
$('.search-results__list').find('li').remove();
// вставляем кнопку загрузки профилей
$('.search-results__list').after('<button id="load-more">Load More</button>');
// добавляем функционал на кнопку
$('#load-more').addClass('artdeco-button').on('click', fetchProfiles);

// ставим по умолчания индекс профиля для запроса
window.nextItemIndex = 0;

If done directly in the console on the search page, this will add a button that loads 10 new profiles on each click and renders them as a list. Of course, before that, change the token and url to the necessary one. The profile block will contain the name, job title, location, link to the profile, and a placeholder image.

Conclusion

Thus, with a minimum of effort, we were able to find a weak spot and regain search without restrictions. It was enough to analyze the data and their path, to look into the request itself.

I cannot say that this is a serious problem for LinkedIn, because it does not pose any threat. The maximum is the lost profit due to such “bypasses”, which allows you not to pay for the premium. Perhaps such a server response is necessary for the correct operation of other parts of the site, or it's just laziness of the developers, a lack of resources that does not allow them to do well. (The restriction has appeared since January 2015, before this limit was not).

PS

Naturally, the jQuery code is a rather primitive example of the possibilities. At the moment, I created a browser extension for my needs. It adds control buttons and renders full profiles with pictures, invite button and general connections. Plus, it dynamically collects filters for locations, companies, and other things, and extracts a token from cookies. So you don't need to hardcode anything. Well, it adds additional settings fields, a la “how many profiles to request at a time, up to 49”.

I am still working on this add-on and plan to make it publicly available. Write if you are interested.

Source: habr.com