yandex metrica
Data Export via Logs API
with a free ready to use Postman workspace
Logs API is designed for getting non-aggregated data stored in Yandex Metrica. Someone may also call it raw data, but in reality this is slightly transformed* data.
* the author of this article assumes YM operates with extended datasets that are not available for export.
Web analysts may experience challenges with getting data through API as this requires some technical skills that are rarely needed in their everyday job. This article is a step by step guidance to help everyone who does not have experience and skills working with API (Application Programming Interface).
* the author of this article assumes YM operates with extended datasets that are not available for export.
Web analysts may experience challenges with getting data through API as this requires some technical skills that are rarely needed in their everyday job. This article is a step by step guidance to help everyone who does not have experience and skills working with API (Application Programming Interface).
Why to use Logs API
- To be fluent in the data you work with. If you understand what data is captured and how it's stored, you level up your professional competence and widen the vision of possible analytics scenarios measurement and analysis.
- To run data analysis beyond Metrica reporting capabilities. Even for troubleshooting because sometimes it's hardly possible to figure out why Metrica reports return this instead of that. For example, how it counts Users.
- To help your friendly data engineers to enrich DWH or Data Lake with Metrica data. They will need to understand data structure, connection options and many other aspects of how to onboard web analytics data. Having at least general understanding of what the Logs are will help you effectively collaborate with the team during the project.
- Expanding the previous bullet point, if you are a data engineer or a member of the team building something robust on YM data, then Logs API is a must have building block you should know well.
Access Provisioning
We'll start with getting credentials for connecting to Logs API as the access provided for running reports in Metrica UI is insufficient.
Open this page https://yandex.com/dev/metrika/ and click on "Get an OAuth access token" (or simply click-through to this URL https://oauth.yandex.com/client/new).
You'll be redirected to Yandex authorisation page, sign in with your regular credentials you are using for Metrica reporting UI (Yandex account) if you are not already authenticated.
Fill in the form:
1. Service name - this is how the credentials will be displayed in the future. You may want to enter "Yandex Metrica - You Company Name", for example.
2. Tick the box Web services and click on the blue link placed under the Callback URL field to fill it in with the default value as shown in the screenshot https://oauth.yandex.com/verification_code
3. Scroll the list of services unless you find Yandex.Metrica, expand it and tick the box "Access to statistics and ability to view all counter settings". This will be enough for getting reports and exporting data via Logs API.
4. Scroll the page till the very end and optionally fill in other fields. For example, it makes sense to add some comments in the Description as a reminder of what this service (credentials) is going to be used for.
Click Create app to finish the setup.
Click Create app to finish the setup.
You will get a confirmation of the successful app creation and the information required for establishing connection via Logs API:
- ClientID (*do not confuse with the ClientID used for visitors identification in YM)
- Client secret
- ClientID (*do not confuse with the ClientID used for visitors identification in YM)
- Client secret
When you need to get back to this page (to the app credentials), use this link https://oauth.yandex.com
Postman
If you have never worked with API, I advise to use Postman. It's available as an app for both Windows and macOS.
Use a public workspace to work with Yandex Metrica API following this link
https://www.postman.com/andreyosadchuk/workspace/y...
https://www.postman.com/andreyosadchuk/workspace/y...
It's configured with two Environments - demo and the other one for your personal tag. So if you do not have data in your own tag, you can connect to the Demo tag.
Regardless of what tag and environment you are going to use, you should connect with your own ClientID and Client secret.
Let's configure an environment.
If you have not used Postman, please note that everything your will be entering will stay with you, nothing will be shared with anyone by design, you are just using a preconfigured template.
1. Select Environments, select "Yandex Metrica My Test Counter" (you can rename it after creating a copy of the workspace under your Postman account)
2. Fill in the table cells in the CURRENT VALUE column:
yandex_client_id: ClientID
yandex_access_token: Client secret
yandex_counter_id: Yandex Metrica tag number you want to connect to
If you have not used Postman, please note that everything your will be entering will stay with you, nothing will be shared with anyone by design, you are just using a preconfigured template.
1. Select Environments, select "Yandex Metrica My Test Counter" (you can rename it after creating a copy of the workspace under your Postman account)
2. Fill in the table cells in the CURRENT VALUE column:
yandex_client_id: ClientID
yandex_access_token: Client secret
yandex_counter_id: Yandex Metrica tag number you want to connect to
Then switch to Collections to open up a list of preconfigured methods for using Yandex Metrica Logs API.
- Evaluate - checks if the request you are going to schedule can be processed by Yandex at all (is it feasible to run such a data export?)
- Create - creates a request (job) for data export
- GetInfo - returns status for a given request to check if the data is already available for download or if it's still getting prepared
- GetList - returns information about all requests
- Download - download the successfully processed requests
- Clean - delete previously created request
Start working with Logs API
Click on the Create and fill in the table:
- source - table name (either visits or hits)
- date1 - the first date of the reporting period
- date2 - the last date of the reporting period
- fields - comma separated list of fields (dimensions and metrics). By default the value is mapped to the environment variable to pull all fields. Do not change it unless you want to narrow the output.
Click Send.
You will see a response as shown below:
You will see a response as shown below:
This means that the request was successfully received and now it's being prepared. Copy the assigned request_id.
Now you need to wait for a while until the data is ready for download.
Switch to GetInfo, fill in the VALUE with the request_id and hit Send.
Now you need to wait for a while until the data is ready for download.
Switch to GetInfo, fill in the VALUE with the request_id and hit Send.
When the data is ready for download you will see "status": "processed" in the response. If a dataset prepared as per the request is relatively big, it will be divided into several parts. Check how many parts are listed in the response (array "parts", and inside "part_number" starting from zero).
Switch to Download.
Enter request_id and part_number.
Enter request_id and part_number.
Then open a drop down menu next to the Send button and click "Send and Download".
Save the response as a txt file since Logs API returns data in the format of a text tab-delimited file (aka .tsv).
Save the response as a txt file since Logs API returns data in the format of a text tab-delimited file (aka .tsv).
I will not cover the other methods available in the workspace collections as they are self-explanatory. Here is a link to the Logs API documentation explaining all the request methods.