In most cases, when working with the response received from the API or any other data that has a complex tree structure, you will encounter JSON and XML formats.
These formats have numerous advantages: they store data compactly and help avoid excessive duplication of information.
The downside of these formats is the complexity of their processing and analysis. Unstructured data cannot be used in computations nor can visualizations be built based on it.
This article logically continues the publication It will help you transform unstructured data constructs into a familiar, analyzable tabular form using the tidyrpackage, which is part of the tidyversecore library, and its family of functions unnest_*().
Content
If you're interested in data analysis, you might find my and channels interesting. Most of their content is devoted to the R language.
Introduction
Rectangling (translator's note, I couldn't find appropriate translations for this term, so I'll leave it as is.) is the process of converting unstructured data with nested arrays into a two-dimensional table consisting of familiar rows and columns. In tidyr there are several functions that will help you expand nested list-columns and convert the data into a rectangular, tabular form:
unnest_longer()takes each element of the list-column and creates a new row.unnest_wider()takes each element of the list-column and creates a new column.unnest_auto()automatically determines which function is most appropriate to use.
unnest_longer()orunnest_wider().hoist()is similar tounnest_wider()but selects only the specified components and allows working with multiple levels of nesting.
Most problems associated with converting unstructured data with several levels of nesting into a two-dimensional table can be solved by combining the listed functions with dplyr.
To demonstrate these techniques, we will use the package repurrrsive, which provides several complex, multi-level lists obtained from a web API.
library(tidyr)
library(dplyr)
library(repurrrsive)GitHub users
Let's start with gh_users, a list that contains information about six GitHub users. First, let's transform the list gh_users downward API support (simultaneously with this in tibble frame:
users <- tibble( user = gh_users ) This seems a bit illogical: why transition to a more complex data structure? gh_usersHowever, the data frame has a significant advantage: it combines multiple vectors, so everything is tracked within a single object.
Each element of the object users represents a named list, where each element corresponds to a column.
names(users$user[[1]])
#> [1] "login" "id" "avatar_url"
#> [4] "gravatar_id" "url" "html_url"
#> [7] "followers_url" "following_url" "gists_url"
#> [10] "starred_url" "subscriptions_url" "organizations_url"
#> [13] "repos_url" "events_url" "received_events_url"
#> [16] "type" "site_admin" "name"
#> [19] "company" "blog" "location"
#> [22] "email" "hireable" "bio"
#> [25] "public_repos" "public_gists" "followers"
#> [28] "following" "created_at" "updated_at"There are two ways to turn list components into columns. unnest_wider() which takes each component and creates a new column:
users %>% unnest_wider(user)
#> # A tibble: 6 x 30
#> login id avatar_url gravatar_id url html_url followers_url
#>
#> 1 gabo… 6.60e5 https://a… "" http… https:… https://api.…
#> 2 jenn… 5.99e5 https://a… "" http… https:… https://api.…
#> 3 jtle… 1.57e6 https://a… "" http… https:… https://api.…
#> 4 juli… 1.25e7 https://a… "" http… https:… https://api.…
#> 5 leep… 3.51e6 https://a… "" http… https:… https://api.…
#> 6 masa… 8.36e6 https://a… "" http… https:… https://api.…
#> # … with 23 more variables: following_url , gists_url ,
#> # starred_url , subscriptions_url , organizations_url ,
#> # repos_url , events_url , received_events_url ,
#> # type , site_admin , name , company , blog ,
#> # location , email , public_repos , public_gists ,
#> # followers , following , created_at , updated_at ,
#> # bio , hireableIn this case, we received a table consisting of 30 columns, and most of them are unnecessary, so instead we can unnest_wider() to use hoist(). hoist() allows us to extract selected components using the same syntax as purrr::pluck():
users %>%) hoist(user,
followers = "followers",
login = "login",
url = "html_url"
)
#> # A tibble: 6 x 4
#> followers login url user
#>
#> 1 303 gaborcsardi https://github.com/gaborcsardi
#> 2 780 jennybc https://github.com/jennybc
#> 3 3958 jtleek https://github.com/jtleek
#> 4 115 juliasilge https://github.com/juliasilge
#> 5 213 leeper https://github.com/leeper
#> 6 34 masalmon https://github.com/masalmonhoist() removes the specified named components from the list-column user, so you can think of it as hoist() moving components from the internal list of the data frame to its top level.
GitHub repositories
Aligning the list gh_repos we start similarly by transforming it into tibble:
repos # A tibble: 6 x 1
#> repo
#>
#> 1
#> 2
#> 3
#> 4
#> 5
#> 6This time the elements user represent a list of repositories belonging to that user. Each repository is a separate observation, so according to the concept of tidy data (note: tidy data) they should become new rows, which is why we use unnest_longer() instead of unnest_wider():
repos % unnest_longer(repo)
repos
#> # A tibble: 176 x 1
#> repo
#>
#> 1
#> 2
#> 3
#> 4
#> 5
#> 6
#> 7
#> 8
#> 9
#> 10
#> # … with 166 more rowsNow we can use unnest_wider() or hoist() :
repos %>% hoist(repo,
login = c("owner", "login"),
name = "name",
homepage = "homepage",
watchers = "watchers_count"
)
#> # A tibble: 176 x 5
#> login name homepage watchers repo
#>
#> 1 gaborcsardi after 5
#> 2 gaborcsardi argufy 19
#> 3 gaborcsardi ask 5
#> 4 gaborcsardi baseimports 0
#> 5 gaborcsardi citest 0
#> 6 gaborcsardi clisymbols "" 18
#> 7 gaborcsardi cmaker 0
#> 8 gaborcsardi cmark 0
#> 9 gaborcsardi conditions 0
#> 10 gaborcsardi crayon 52
#> # … with 166 more rowsNote the use of c("owner", "login"): this allows us to obtain the second-level value from the nested list ownerAn alternative approach is to obtain the entire list owner and then use a function unnest_wider() to place each of its elements in a column:
repos %>%
hoist(repo, owner = "owner") %>%
unnest_wider(owner)
#> # A tibble: 176 x 18
#> login id avatar_url gravatar_id url html_url followers_url
#>
#> 1 gabo… 660288 https://a… "" http… https:… https://api.…
#> 2 gabo… 660288 https://a… "" http… https:… https://api.…
#> 3 gabo… 660288 https://a… "" http… https:… https://api.…
#> 4 gabo… 660288 https://a… "" http… https:… https://api.…
#> 5 gabo… 660288 https://a… "" http… https:… https://api.…
#> 6 gabo… 660288 https://a… "" http… https:… https://api.…
#> 7 gabo… 660288 https://a… "" http… https:… https://api.…
#> 8 gabo… 660288 https://a… "" http… https:… https://api.…
#> 9 gabo… 660288 https://a… "" http… https:… https://api.…
#> 10 gabo… 660288 https://a… "" http… https:… https://api.…
#> # … with 166 more rows, and 11 more variables: following_url ,
#> # gists_url , starred_url , subscriptions_url ,
#> # organizations_url , repos_url , events_url ,
#> # received_events_url , type , site_admin , repoInstead of pondering over the choice of the right function unnest_longer() or unnest_wider() you can use unnest_auto(). This function employs several heuristic methods to select the most appropriate function for data transformation and outputs a message about the chosen method.
tibble(repo = gh_repos) %>%
unnest_auto(repo) %>%
unnest_auto(repo)
#> Using `unnest_longer(repo)`; no element has names
#> Using `unnest_wider(repo)`; elements have 68 names in common
#> # A tibble: 176 x 67
#> id name full_name owner private html_url description fork url
#>
#> 1 6.12e7 after gaborcsa… 2 4.05e7 argu… gaborcsa… 3 3.64e7 ask gaborcsa… 4 3.49e7 base… gaborcsa… 5 6.16e7 cite… gaborcsa… 6 3.39e7 clis… gaborcsa… 7 3.72e7 cmak… gaborcsa… 8 6.80e7 cmark gaborcsa… 9 6.32e7 cond… gaborcsa… <nam… FALSE https:/… TRUE http…
#> 10 2.43e7 cray… gaborcsa… # … with 166 more rows, and 58 more variables: forks_url ,
#> # keys_url , collaborators_url , teams_url ,
#> # hooks_url , issue_events_url , events_url ,
#> # assignees_url , branches_url , tags_url ,
#> # blobs_url , git_tags_url , git_refs_url ,
#> # trees_url , statuses_url , languages_url ,
#> # stargazers_url , contributors_url , subscribers_url ,
#> # subscription_url , commits_url , git_commits_url ,
#> # comments_url , issue_comment_url , contents_url ,
#> # compare_url , merges_url , archive_url ,
#> # downloads_url , issues_url , pulls_url ,
#> # milestones_url , notifications_url , labels_url ,
#> # releases_url , deployments_url , created_at ,
#> # updated_at , pushed_at , git_url , ssh_url ,
#> # clone_url , svn_url , size , stargazers_count ,
#> # watchers_count , language , has_issues ,
#> # has_downloads , has_wiki , has_pages ,
#> # forks_count , open_issues_count , forks ,
#> # open_issues , watchers , default_branch ,
#> # homepageGame of Thrones characters
got_chars has an identical structure to gh_users: this is a set of named lists, where each element of the inner list describes some attribute of a Game of Thrones character. Converting got_chars to tabular format, we start by creating a data frame, just like in the previous examples, and then we will translate each element into a separate column:
chars # A tibble: 30 x 1
#> char
#>
#> 1
#> 2
#> 3
#> 4
#> 5
#> 6
#> 7
#> 8
#> 9
#> 10
#> # … with 20 more rows
chars2 % unnest_wider(char)
chars2
#> # A tibble: 30 x 18
#> url id name gender culture born died alive titles aliases father
#> <int> <chr> <chr> <chr> <chr> <chr> <lgl> <list> <list> <chr>
#> 1 http… 1022 Theo… Male Ironbo… In 2… "" TRUE <chr … <chr [… ""
#> 2 http… 1052 Tyri… Male "" In 2… "" TRUE <chr … <chr [… ""
#> 3 http… 1074 Vict… Male Ironbo… In 2… "" TRUE <chr … <chr [… ""
#> 4 http… 1109 Will Male "" "" In 2… FALSE <chr … <chr [… ""
#> 5 http… 1166 Areo… Male Norvos… In 2… "" TRUE <chr … <chr [… ""
#> 6 http… 1267 Chett Male "" At H… In 2… FALSE <chr … <chr [… ""
#> 7 http… 1295 Cres… Male "" In 2… In 2… FALSE <chr … <chr [… ""
#> 8 http… 130 Aria… Female Dornish In 2… "" TRUE <chr … <chr [… ""
#> 9 http… 1303 Daen… Female Valyri… In 2… "" TRUE <chr … <chr [… ""
#> 10 http… 1319 Davo… Male Wester… In 2… "" TRUE <chr … <chr [… ""
#> # … with 20 more rows, and 7 more variables: mother , spouse ,
#> # allegiances , books , povBooks , tvSeries ,
#> # playedByStructure got_chars more complex than gh_users, as some components of the list char are lists themselves, resulting in columns — lists:
chars2 %>% select_if(is.list)
#> # A tibble: 30 x 7
#> titles aliases allegiances books povBooks tvSeries playedBy
#>
#> 1
#> 2
#> 3
#> 4
#> 5
#> 6
#> 7
#> 8
#> 9
#> 10
#> # … with 20 more rowsYour next steps depend on your analysis goals. You may need to include information for each book and series where the character appears in rows:
chars2 %>%
select(name, books, tvSeries) %>%
pivot_longer(c(books, tvSeries), names_to = "media", values_to = "value") %>%
unnest_longer(value)
#> # A tibble: 180 x 3
#> name media value
#>
#> 1 Theon Greyjoy books A Game of Thrones
#> 2 Theon Greyjoy books A Storm of Swords
#> 3 Theon Greyjoy books A Feast for Crows
#> 4 Theon Greyjoy tvSeries Season 1
#> 5 Theon Greyjoy tvSeries Season 2
#> 6 Theon Greyjoy tvSeries Season 3
#> 7 Theon Greyjoy tvSeries Season 4
#> 8 Theon Greyjoy tvSeries Season 5
#> 9 Theon Greyjoy tvSeries Season 6
#> 10 Tyrion Lannister books A Feast for Crows
#> # … with 170 more rowsOr perhaps you want to create a table that will allow you to map the character to the work:
chars2 %>%
select(name, title = titles) %>%
unnest_longer(title)
#> # A tibble: 60 x 2
#> name title
#>
#> 1 Theon Greyjoy Prince of Winterfell
#> 2 Theon Greyjoy Captain of Sea Bitch
#> 3 Theon Greyjoy Lord of the Iron Islands (by law of the green lands)
#> 4 Tyrion Lannister Acting Hand of the King (former)
#> 5 Tyrion Lannister Master of Coin (former)
#> 6 Victarion Greyjoy Lord Captain of the Iron Fleet
#> 7 Victarion Greyjoy Master of the Iron Victory
#> 8 Will ""
#> 9 Areo Hotah Captain of the Guard at Sunspear
#> 10 Chett ""
#> # … with 50 more rows(Note the blank values "" in the field title, this is due to errors made when entering data in got_chars: in fact, characters for which there are no corresponding book and series titles should have a vector of length 0, not a vector of length 1 containing an empty string.) title We can rewrite the above example using the function
. This approach is convenient for one-off analyses, but should not be relied upon unnest_auto()for regular use. The thing is, if your data structure changes unnest_auto() it may change the chosen data transformation mechanism; if it initially unfolded list-columns into rows using unnest_auto() , then with a change in the structure of incoming data, the logic may shift towards unnest_longer(), and using such an approach on a permanent basis may lead to unforeseen errors. unnest_wider(), and using such an approach on a permanent basis can lead to unforeseen errors.
tibble(char = got_chars) %>%
unnest_auto(char) %>%
select(name, title = titles) %>%
unnest_auto(title)
#> Using `unnest_wider(char)`; elements have 18 names in common
#> Using `unnest_longer(title)`; no element has names
#> # A tibble: 60 x 2
#> name title
#>
#> 1 Theon Greyjoy Prince of Winterfell
#> 2 Theon Greyjoy Captain of Sea Bitch
#> 3 Theon Greyjoy Lord of the Iron Islands (by law of the green lands)
#> 4 Tyrion Lannister Acting Hand of the King (former)
#> 5 Tyrion Lannister Master of Coin (former)
#> 6 Victarion Greyjoy Lord Captain of the Iron Fleet
#> 7 Victarion Greyjoy Master of the Iron Victory
#> 8 Will ""
#> 9 Areo Hotah Captain of the Guard at Sunspear
#> 10 Chett ""
#> # … with 50 more rowsGeocoding with Google
Next, we will look at a more complex data structure obtained from the Google geocoding service. Caching credentials contradicts the terms of service for the Google Maps API, so I will first write a simple wrapper for the API. This will be based on storing the Google Maps API key in an environment variable; if you do not have the key stored in your environment variables, the code snippets presented in this section will not run.
has_key <- !identical(Sys.getenv("GOOGLE_MAPS_API_KEY"), "")
if (!has_key) {
message("No Google Maps API key found; code chunks will not be run")
}
# https://developers.google.com/maps/documentation/geocoding
geocode <- function(address, api_key = Sys.getenv("GOOGLE_MAPS_API_KEY")) {
url <- "https://maps.googleapis.com/maps/api/geocode/json"
url <- paste0(url, "?address=", URLencode(address), "&key=", api_key)
jsonlite::read_json(url)
}The list returned by this function is quite complex:
houston List of 2
#> $ results:List of 1
#> ..$ :List of 5
#> .. ..$ address_components:List of 4
#> .. .. ..$ :List of 3
#> .. .. .. ..$ long_name : chr "Houston"
#> .. .. .. ..$ short_name: chr "Houston"
#> .. .. .. ..$ types :List of 2
#> .. .. .. .. ..$ : chr "locality"
#> .. .. .. .. ..$ : chr "political"
#> .. .. ..$ :List of 3
#> .. .. .. ..$ long_name : chr "Harris County"
#> .. .. .. ..$ short_name: chr "Harris County"
#> .. .. .. ..$ types :List of 2
#> .. .. .. .. ..$ : chr "administrative_area_level_2"
#> .. .. .. .. ..$ : chr "political"
#> .. .. ..$ :List of 3
#> .. .. .. ..$ long_name : chr "Texas"
#> .. .. .. ..$ short_name: chr "TX"
#> .. .. .. ..$ types :List of 2
#> .. .. .. .. ..$ : chr "administrative_area_level_1"
#> .. .. .. .. ..$ : chr "political"
#> .. .. ..$ :List of 3
#> .. .. .. ..$ long_name : chr "United States"
#> .. .. .. ..$ short_name: chr "US"
#> .. .. .. ..$ types :List of 2
#> .. .. .. .. ..$ : chr "country"
#> .. .. .. .. ..$ : chr "political"
#> .. ..$ formatted_address : chr "Houston, TX, USA"
#> .. ..$ geometry :List of 4
#> .. .. ..$ bounds :List of 2
#> .. .. .. ..$ northeast:List of 2
#> .. .. .. .. ..$ lat: num 30.1
#> .. .. .. .. ..$ lng: num -95
#> .. .. .. ..$ southwest:List of 2
#> .. .. .. .. ..$ lat: num 29.5
#> .. .. .. .. ..$ lng: num -95.8
#> .. .. ..$ location :List of 2
#> .. .. .. ..$ lat: num 29.8
#> .. .. .. ..$ lng: num -95.4
#> .. .. ..$ location_type: chr "APPROXIMATE"
#> .. .. ..$ viewport :List of 2
#> .. .. .. ..$ northeast:List of 2
#> .. .. .. .. ..$ lat: num 30.1
#> .. .. .. .. ..$ lng: num -95
#> .. .. .. ..$ southwest:List of 2
#> .. .. .. .. ..$ lat: num 29.5
#> .. .. .. .. ..$ lng: num -95.8
#> .. ..$ place_id : chr "ChIJAYWNSLS4QIYROwVl894CDco"
#> .. ..$ types :List of 2
#> .. .. ..$ : chr "locality"
#> .. .. ..$ : chr "political"
#> $ status : chr "OK"Fortunately, we can solve the problem of converting this data into tabular format step by step using functions tidyr. To make the task a bit more complex and realistic, I will begin by geocoding a few cities:
city <- c ( "Houston" , "LA" , "New York" , "Chicago" , "Springfield" ) city_geo <- purrr::map (city, geocode) I will convert the obtained result into tibble, and for convenience, I will add a column with the corresponding city name.
loc # A tibble: 5 x 2
#> city json
#>
#> 1 Houston
#> 2 LA
#> 3 New York
#> 4 Chicago
#> 5 SpringfieldThe first level contains components status and result, which we can expand using unnest_wider() :
loc %>%
unnest_wider(json)
#> # A tibble: 5 x 3
#> city results status
#>
#> 1 Houston OK
#> 2 LA OK
#> 3 New York OK
#> 4 Chicago OK
#> 5 Springfield OKNote that results is a hierarchical list. Most cities have 1 item (representing a unique value corresponding to the geocoding API), but Springfield has two. We can extract these into separate rows using unnest_longer() :
loc %>%
unnest_wider(json) %>%
unnest_longer(results)
#> # A tibble: 5 x 3
#> city results status
#>
#> 1 Houston OK
#> 2 LA OK
#> 3 New York OK
#> 4 Chicago OK
#> 5 Springfield OKNow all of them have the same components, which can be verified using unnest_wider():
loc %>%
unnest_wider(json) %>%
unnest_longer(results) %>%
unnest_wider(results)
#> # A tibble: 5 x 7
#> city address_componen… formatted_addre… geometry place_id types status
#>
#> 1 Houst… Houston, TX, USA <named … ChIJAYWN… 2 LA Los Angeles, CA… <named … ChIJE9on… 3 New Y… New York, NY, U… <named … ChIJOwg_… 4 Chica… Chicago, IL, USA <named … ChIJ7cv0… 5 Sprin… Springfield, MO… <named … ChIJP5jI… <lis… OKWe can find the latitude and longitude coordinates of each city by unfolding the list geometry:
loc %>%
unnest_wider(json) %>%
unnest_longer(results) %>%
unnest_wider(results) %>%
unnest_wider(geometry)
#> # A tibble: 5 x 10
#> city address_compone… formatted_addre… bounds location location_type
#>
#> 1 Hous… Houston, TX, USA <name… 2 LA Los Angeles, CA… <name… 3 New … New York, NY, U… <name… 4 Chic… Chicago, IL, USA <name… 5 Spri… Springfield, MO… <name… # … with 4 more variables: viewport , place_id , types ,
#> # statusAnd then the location, which requires unfolding location:
loc %>%
unnest_wider(json) %>%
unnest_longer(results) %>%
unnest_wider(results) %>%
unnest_wider(geometry) %>%
unnest_wider(location)
#> # A tibble: 5 x 11
#> city address_compone… formatted_addre… bounds lat lng location_type
#>
#> 1 Hous… Houston, TX, USA 2 LA Los Angeles, CA… 3 New … New York, NY, U… 4 Chic… Chicago, IL, USA 5 Spri… Springfield, MO… # … with 4 more variables: viewport , place_id , types ,
#> # statusAgain, unnest_auto() it simplifies the described operation with certain risks that may arise from changes in the structure of incoming data:
loc %>%
unnest_auto(json) %>%
unnest_auto(results) %>%
unnest_auto(results) %>%
unnest_auto(geometry) %>%
unnest_auto(location)
#> Using `unnest_wider(json)`; elements have 2 names in common
#> Using `unnest_longer(results)`; no element has names
#> Using `unnest_wider(results)`; elements have 5 names in common
#> Using `unnest_wider(geometry)`; elements have 4 names in common
#> Using `unnest_wider(location)`; elements have 2 names in common
#> # A tibble: 5 x 11
#> city address_compone… formatted_addre… bounds lat lng location_type
#>
#> 1 Hous… Houston, TX, USA 2 LA Los Angeles, CA… 3 New … New York, NY, U… 4 Chic… Chicago, IL, USA 5 Spri… Springfield, MO… # … with 4 more variables: viewport , place_id , types ,
#> # statusWe can also simply look at the first address for each city:
loc %>%
unnest_wider(json) %>%
hoist(results, first_result = 1) %>%
unnest_wider(first_result) %>%
unnest_wider(geometry) %>%
unnest_wider(location)
#> # A tibble: 5 x 11
#> city address_compone… formatted_addre… bounds lat lng location_type
#>
#> 1 Hous… Houston, TX, USA 2 LA Los Angeles, CA… 3 New … New York, NY, U… 4 Chic… Chicago, IL, USA 5 Spri… Springfield, MO… # … with 4 more variables: viewport , place_id , types ,
#> # statusOr use hoist() for a multi-level dive to go directly to lat and lng.
loc %>
hoist(json,
lat = list("results", 1, "geometry", "location", "lat"),
lng = list("results", 1, "geometry", "location", "lng")
)
#> # A tibble: 5 x 4
#> city lat lng json
#>
#> 1 Houston 29.8 -95.4
#> 2 LA 34.1 -118.
#> 3 New York 40.7 -74.0
#> 4 Chicago 41.9 -87.6
#> 5 Springfield 37.2 -93.3Discography of Sharly Gelfand
In conclusion, let's look at the most complex structure — Sharly Gelfand's discography. As in the examples above, we start by converting the list into a data frame with a single column, and then we will expand it so that each component is a separate column. I will also transform the column date_added into the appropriate date and time format in R.
discs %
unnest_wider(disc) %>%
mutate(date_added = as.POSIXct(strptime(date_added, "%Y-%m-%dT%H:%M:%S")))
discs
#> # A tibble: 155 x 5
#> instance_id date_added basic_information id rating
#>
#> 1 354823933 2019-02-16 17:48:59 7496378 0
#> 2 354092601 2019-02-13 14:13:11 4490852 0
#> 3 354091476 2019-02-13 14:07:23 9827276 0
#> 4 351244906 2019-02-02 11:39:58 9769203 0
#> 5 351244801 2019-02-02 11:39:37 7237138 0
#> 6 351052065 2019-02-01 20:40:53 13117042 0
#> 7 350315345 2019-01-29 15:48:37 7113575 0
#> 8 350315103 2019-01-29 15:47:22 10540713 0
#> 9 350314507 2019-01-29 15:44:08 11260950 0
#> 10 350314047 2019-01-29 15:41:35 11726853 0
#> # … with 145 more rowsAt this level, we have obtained information about when each disc was added to Sharly's discography, but we still do not see any data about these discs. To address this, we need to expand the column basic_information:
discs %>% unnest_wider(basic_information)
#> Column name `id` must not be duplicated.
#> Use .name_repair to specify repair.Unfortunately, we will get an error because there is a column with the same name basic_information . When encountering such an error, to quickly determine its cause, you can use basic_informationnames_repair = "unique" names_repair = "unique":
discs %>% unnest_wider(basic_information, names_repair = "unique")
#> New names:
#> * id -> id...6
#> * id -> id...14
#> # A tibble: 155 x 15
#> instance_id date_added labels year artists id...6 thumb title
#>
#> 1 354823933 2019-02-16 17:48:59 <list… 2015 2 354092601 2019-02-13 14:13:11 <list… 2013 3 354091476 2019-02-13 14:07:23 <list… 2017 4 351244906 2019-02-02 11:39:58 <list… 2017 5 351244801 2019-02-02 11:39:37 <list… 2015 6 351052065 2019-02-01 20:40:53 <list… 2019 7 350315345 2019-01-29 15:48:37 <list… 2014 8 350315103 2019-01-29 15:47:22 <list… 2015 9 350314507 2019-01-29 15:44:08 <list… 2017 10 350314047 2019-01-29 15:41:35 <list… 2017 # … with 145 more rows, and 7 more variables: formats ,
#> # cover_image , resource_url , master_id ,
#> # master_url , id...14 , ratingThe issue is that basic_information it duplicates the id column which is also stored at the top level, so we can simply remove it:
discs %>%
select(-id) %>%
unnest_wider(basic_information)
#> # A tibble: 155 x 14
#> instance_id date_added labels year artists id thumb title
#>
#> 1 354823933 2019-02-16 17:48:59 <list… 2015 2 354092601 2019-02-13 14:13:11 <list… 2013 3 354091476 2019-02-13 14:07:23 <list… 2017 4 351244906 2019-02-02 11:39:58 <list… 2017 5 351244801 2019-02-02 11:39:37 <list… 2015 6 351052065 2019-02-01 20:40:53 <list… 2019 7 350315345 2019-01-29 15:48:37 <list… 2014 8 350315103 2019-01-29 15:47:22 <list… 2015 9 350314507 2019-01-29 15:44:08 <list… 2017 10 350314047 2019-01-29 15:41:35 <list… 2017 # … with 145 more rows, and 6 more variables: formats ,
#> # cover_image , resource_url , master_id ,
#> # master_url , ratingAlternatively, we could use hoist():
discs %>%
hoist(basic_information,
title = "title",
year = "year",
label = list("labels", 1, "name"),
artist = list("artists", 1, "name")
)
#> # A tibble: 155 x 9
#> instance_id date_added title year label artist
#>
#> 1 354823933 2019-02-16 17:48:59 Demo 2015 Tobi… Mollot
#> 2 354092601 2019-02-13 14:13:11 Obse… 2013 La V… Una B…
#> 3 354091476 2019-02-13 14:07:23 I 2017 La V… S.H.I…
#> 4 351244906 2019-02-02 11:39:58 Oído… 2017 La V… Rata …
#> 5 351244801 2019-02-02 11:39:37 A Ca… 2015 Kato… Ivy (…
#> 6 351052065 2019-02-01 20:40:53 Tash… 2019 High… Tashme
#> 7 350315345 2019-01-29 15:48:37 Demo 2014 Mind… Desgr…
#> 8 350315103 2019-01-29 15:47:22 Let … 2015 Not … Phant…
#> 9 350314507 2019-01-29 15:44:08 Sub … 2017 Not … Sub S…
#> 10 350314047 2019-01-29 15:41:35 Demo 2017 Pres… Small…
#> # … with 145 more rows, and 3 more variables: basic_information ,
#> # id , ratingHere, I quickly extract the name of the first label and artist by index, diving into the nested list.
A more systematic approach involves creating separate tables for the artist and the label:
discs %>%
hoist(basic_information, artist = "artists") %>%
select(disc_id = id, artist) %>%
unnest_longer(artist) %>%
unnest_wider(artist)
#> # A tibble: 167 x 8
#> disc_id join name anv tracks role resource_url id
#>
#> 1 7496378 "" Mollot "" "" "" https://api.discog… 4.62e6
#> 2 4490852 "" Una Bèstia… "" "" "" https://api.discog… 3.19e6
#> 3 9827276 "" S.H.I.T. (… "" "" "" https://api.discog… 2.77e6
#> 4 9769203 "" Rata Negra "" "" "" https://api.discog… 4.28e6
#> 5 7237138 "" Ivy (18) "" "" "" https://api.discog… 3.60e6
#> 6 13117042 "" Tashme "" "" "" https://api.discog… 5.21e6
#> 7 7113575 "" Desgraciad… "" "" "" https://api.discog… 4.45e6
#> 8 10540713 "" Phantom He… "" "" "" https://api.discog… 4.27e6
#> 9 11260950 "" Sub Space … "" "" "" https://api.discog… 5.69e6
#> 10 11726853 "" Small Man … "" "" "" https://api.discog… 6.37e6
#> # … with 157 more rows
discs %>%
hoist(basic_information, format = "formats") %>%
select(disc_id = id, format) %>%
unnest_longer(format) %>%
unnest_wider(format) %>%
unnest_longer(descriptions)
#> # A tibble: 280 x 5
#> disc_id descriptions text name qty
#>
#> 1 7496378 Numbered Black Cassette 1
#> 2 4490852 LP Vinyl 1
#> 3 9827276 "7"" Vinyl 1
#> 4 9827276 45 RPM Vinyl 1
#> 5 9827276 EP Vinyl 1
#> 6 9769203 LP Vinyl 1
#> 7 9769203 Album Vinyl 1
#> 8 7237138 "7"" Vinyl 1
#> 9 7237138 45 RPM Vinyl 1
#> 10 13117042 "7"" Vinyl 1
#> # … with 270 more rowsYou can then join them back to the original dataset as needed.
Conclusion
The core of the library tidyverse includes many useful packages united by a common philosophy of data processing.
In this article, we explored the family of functions unnest_*(), which are aimed at working with the extraction of elements from nested lists. This package contains many other useful functions that simplify data transformation according to the concept of Tidy Data.
Source: habr.com
