🥇Obtaining Amplitude Data via API

Introduction

Amplitude has established itself as a reliable tool for product analytics, thanks to its straightforward event setup and flexible visualizations. Often, there arises a need to establish a custom attribution model, perform user clustering, or create a dashboard in another BI system. Such manipulation is only possible if one has raw event data from Amplitude. This article will explain how to obtain this data with minimal programming knowledge.

Prerequisite

A project in Amplitude where events are correctly set up and statistics are being collected.
Python installed (I am using version 3.8.3), and the potential reader should be able to work with it at least at a basic level.

Instruction

Step 1. Obtaining API key and secret key

To export data, the first step is to obtain the API key and secret key.

They can be found by following this path:

‘Manage data’ (located in the bottom left corner of the screen)
Select the desired project from which data will be exported and go to it.
In the opened project menu, select ‘Project settings’
Find the API key and secret key rows, copy them, and save them in a safe place.

Without clicking through, you can go directly to the link, which generally looks like this:
analytics.amplitude.com/$$$$$$$/manage/project/*******/settings,
where $$$$$$$ is your organization’s login in Amplitude, and ****** is the project number.

Step 2. Checking for required libraries

The good news is that these libraries are almost certainly already installed by default or downloaded, but it’s necessary to check. A complete list of libraries I am using at the time of writing this article (with versions indicated where applicable):

requests (2.10.0) - sending a request via API to get data
pandas (1.0.1) - reading JSON, creating a dataframe, and subsequently writing to a file
zipfile - extracting files from the archive obtained via API
gzip - unpacking JSON files from .gz
os - getting a list of files from the extracted archive
time - optional, measuring the script's runtime
tqdm - optional, for convenience in observing the progress of file processing

Step 3. Writing the data loading script

Hint: the full loading script is located at the end of the article; you can take it right away and refer to the step-by-step explanations if needed.

Step 3.1. Importing libraries

We import all the libraries listed in the second step.

# Импорт библиотек
import requests
import pandas as pd
import zipfile
import gzip
import os
import time
import tqdm
from tqdm import tqdm

Step 3.2. Sending a request to Amplitude

We will log the start of the script execution and store it in variable a.

startdate and enddate define the period for data extraction and are embedded in the text of the sent request to server Amplitude; in addition to the date, you can also specify the hour by changing the value after 'T' in the request.

api_key and secret_key correspond to the values obtained in the first step; for security, I'm using random sequences instead of my own here.

a = time.time()
# Parameters for start and end dates
startdate = '20200627'
enddate = '20200628'

api_key = 'kldfg844203rkwekfjs9234'
secret_key = '094tfjdsfmw93mxwfek'
# Sending a request to Amplitude
response = requests.get('https://amplitude.com/api/2/export?start=' + startdate + 'T0&end=' + enddate + 'T0', auth = (api_key, secret_key))
print('1. Request sent')

Step 3.3. Downloading the archive with data

We come up with a name for the archive and store it in variable filename. For my convenience, I specify the period and that this is Amplitude data. Next, we write the received response from Amplitude into the archive.

# Скачивание архива с данными
filename = 'period_since'+startdate+'to'+enddate+'_amplitude_data'
with open(filename + '.zip', "wb") as code:
    code.write(response.content)
print('2. Архив с файлами успешно скачан')

Step 3.4. Extracting files to a folder on the computer

The zipfile library comes into play to assist with file extraction. In the third line, be sure to specify your path where it is most convenient for you to extract.

# Извлечение файлов в папку на компьютере
z = zipfile.ZipFile(filename + '.zip', 'r')
z.extractall(path = 'C:\Users\...\'+filename)
print('3. Архив с файлами извлечен и записан в папку ' + filename)

Step 3.5. Converting json

After extracting files from the archive, json files in .gz format need to be converted and written into a dataframe for further work.

I would like to emphasize that you need to change the path here to your own, and instead of 000000, write your project number from Amplitude (or manually open the path where the archive was extracted and check the folder name inside).

In sequence:

Recording the directory in a variable, obtaining a list of files from the directory, creating an empty dataframe, time.sleep(1) for the correct operation of tqdm, within the loop opening .gz files and immediately reading json using pandas to fill the specified dataframe.

# Преобразование json к обычному табличному формату
directory = 'C:\Users\...\'+filename+'\000000'
files = os.listdir(directory)
amplitude_dataframe = pd.DataFrame()
print('Прогресс обработки файлов:')
time.sleep(1)
for i in tqdm(files):
    with gzip.open(directory + '\' + i) as f:
        add = pd.read_json(f, lines = 'True')
    amplitude_dataframe = pd.concat([amplitude_dataframe, add])
time.sleep(1)    
print('4. JSON файлы из архива успешно преобразованы и записаны в dataframe')

Step 3.6. Writing dataframe to excel

Exporting to excel here is just an example. In many cases, it is more convenient to work with the resulting data frame inside Python or store the data in a warehouse.

You will also need to change the export path here to your own.

# Записать полученной таблицы в Excel-файл
amplitude_dataframe.to_excel('C:\Users\...\'+filename+'.xlsx',index=False)
print('5. Dataframe успешно записан в файл ' + filename)

Step 3.7. Measuring the script's execution time

Recording the current time in variable b, calculating the difference and the number of minutes, and displaying the total minutes. This is the final step.

b = time.time()
diff = b-a
minutes = diff//60
print('The code execution took: {:.0f} minute(s)'.format(minutes))

Conclusion

You can call the table and start working with it by invoking the variable amplitude_dataframe, where the data has been recorded. It will contain about 50 columns, of which in 80% of cases you'll use: event_type — the name of the event, event_properties — parameters of the event, event_time — time of the event, uuid — client id, user_properties — client parameters; it's essential to start with these first. And when comparing figures from your own calculations with metrics from Amplitude dashboards, remember that the system uses its own methodology for calculating unique clients/funnels, etc., and you must familiarize yourself with Amplitude's documentation beforehand.

Thank you for your attention! You can now export raw event data into Amplitude and fully utilize it in your work.

The entire script:

# Импорт библиотек
import requests
import pandas as pd
import zipfile
import gzip
import os
import time
import tqdm
from tqdm import tqdm
a = time.time()
# Параметры начальной и конечной даты
startdate = '20200627'
enddate = '20200628'

api_key = 'd988fddd7cfc0a8a'
secret_key = 'da05cf1aeb3a361a61'
# Отправление запроса в Amplitude
response = requests.get('https://amplitude.com/api/2/export?start='+startdate+'T0&end='+enddate+'T0', auth = (api_key, secret_key))
print('1. Запрос отправлен')

# Скачивание архива с данными
filename = 'period_since'+startdate+'to'+enddate+'_amplitude_data'
with open(filename + '.zip', "wb") as code:
    code.write(response.content)
print('2. Архив с файлами успешно скачан')  

# Извлечение файлов в папку на компьютере
z = zipfile.ZipFile(filename + '.zip', 'r')
z.extractall(path = 'C:\Users\...\'+filename)
print('3. Архив с файлами извлечен и записан в папку ' + filename)

# Преобразование json к обычному табличному формату
directory = 'C:\Users\...\'+filename+'\000000'
files = os.listdir(directory)
amplitude_dataframe = pd.DataFrame()
print('Прогресс обработки файлов:')
time.sleep(1)
for i in tqdm(files):
    with gzip.open(directory + '\' + i) as f:
        add = pd.read_json(f, lines = 'True')
    amplitude_dataframe = pd.concat([amplitude_dataframe, add])
time.sleep(1)    
print('4. JSON файлы из архива успешно преобразованы и записаны в dataframe')

# Записать полученной таблицы в Excel-файл
amplitude_dataframe.to_excel('C:\Users\...\'+filename+'.xlsx',index=False)
print('5. Dataframe успешно записан в файл ' + filename)

b = time.time()
diff = b-a
minutes = diff//60
print('Выполнение кода заняло: {:.0f} минут(ы)'.format( minutes))

Source: habr.com

Getting data from Amplitude via the API

Introduction

Prerequisite

Instruction

Conclusion