PARSE LIST OF LISTS OF DICTIONARIES USING PYTHON | JSON | NESTED LISTS (2023)

Introduction

This video will go over 𝐥𝐢𝐬𝐭 𝐨𝐟 𝐥𝐢𝐬𝐭 𝐨𝐟 𝐝𝐢𝐜𝐭𝐢𝐨𝐧𝐚𝐫𝐢𝐞𝐬 (nested lists of dictioanries), I will parse 3 different ways for the example with 𝐏𝐲𝐭𝐡𝐨𝐧. There is also, a further nested dictionary with lists as values that will be parse and stored using default dictionary lists (this was tricky). Turn on the 🔔 notification

Join this channel to get access to perks:
www.youtube.com/channel/UCbni-TDI-Ub8VlGaP8HLTNw/join

➡ Patreon: www.patreon.com/mrfugudatasci
➡ Buy Me A Coffee: www.buymeacoffee.com/mrfugudatasci
➡ Github: github.com/MrFuguDataScience
➡ Twitter: @MrFuguDataSci
➡ Instagram: @mrfugudatascience

Code From Today:
github.com/MrFuguDataScience/JSON/blob/master/Json%20_listsOfdict.ipynb

𝐀𝐦𝐚𝐳𝐨𝐧 𝐀𝐟𝐟𝐢𝐥𝐢𝐚𝐭𝐞 𝐋𝐢𝐧𝐤𝐬: (I receive a small commission on purchases)
* Prices & Availability Subject to change
--------------------------------------------
Audible Gift Membership: amzn.to/3pAfw7W (End Date: On Going)

Try Audible: amzn.to/3PETRWS (End Date: On Going)

Apple Certified Type C Charger & USB Wall Charger 20W with 2 cables: amzn.to/3dMdqPA

𝐕𝐢𝐝𝐞𝐨𝐬 𝐘𝐨𝐮 𝐌𝐚𝐲 𝐀𝐥𝐬𝐨 𝐋𝐢𝐤𝐞

▶️ PARSE JSON FROM AN API: USING PYTHON: youtu.be/iaN1FxjBuGk

▶️ TWITTER API WITH PYTHON | NLTK BASICS (NLP): youtu.be/kS_0RNdUjrQ

▶️ NLP Using R-Studio: youtu.be/TXhEl7LTQKo

▶️ R-STUDIO: LABEL & ONE HOT ENCODING WITH MULITPLE EXAMPLES: youtu.be/ZTeGJpOSrQo

▶️ REGULAR EXPRESSIONS (Regex) for Parsing ADDRESSES using Python:
youtu.be/nRq8J_aYKT0

Gif at end: Moos-media on pixabay.com

Thumbnail: pixabay.com/vectors/squirrel-reading-books-surprise-304021/

end screen: www.instagram.com/footage3.0/ and on pixabay

Music &. Intro Pic: Special Thanks

Pixabay: instagram (subscribe gif): @imotivationitas

Music: Oshóva - Tidal Dance on
Soundcloud: soundcloud.com/osh-va ,
youtube: www.youtube.com/channel/UC_uiomVq61yUos1FUuOdsTw

Content

Welcome everybody, you're watching mr fugu data science.

Today, we're parsing a list of lists of dictionaries.

We will solve one problem.

Multiple ways.

I got this question from a recent viewer and decided to make a video to try to help them.

We need to do a few imports:.

Let's import json, pandas, from collections, import default, dictionary, and chain map.

The data set we're using today comes from a previous example.

I did parsing New York times.

Api.

This is some raw data where each value or row;, if we are putting this into a data frame, is a list of dictionaries.

To get a clear picture of what these data look like;.

We will take the first entry and notice that in this list of dictionaries, we have a few different options.

But.

We notice that we have inside of this another list of dictionaries from one of our values.

So.

We will need to parse inside of the media further due to nesting.

Where, we'll notice that there are three url's, formats, heights, and widths.

Within, the media metadata.

For each entry;.

We have 20 entries, okay.

If.

We took for instance, one specific entry and threw it into a data.

Frame.

It would look something like this.

But.

Really.

What we would like to have: is all of these.

In one single row.

So you, you would essentially like to have one list of all the urls, a list of all the formats, the heights, and the widths for each entry.

Because.

This was from one entry, not three, okay.

So.

Essentially, what I would like is to create this url:, where we have a list of the three urls.

Assume, that this is the second entry, and this is their three urls and putting it onto one data frame for two rows for this.

And.

I just cut and paste this.

It's the same exact data just to illustrate what I'm trying to achieve later.

If.

You did not want it this way and wanted to expand it.

You could always do the pandas explode.

If that was something you wanted.

Just remember, it gives you more rows.

Let's get into our first coding for the chain map from collections., And.

Think of it as taking multiple dictionaries and basically condensing it into one.

There's.

A few useful examples of showing this for the official documentation.

Let's just call this multiple dictionary to single.

Store this as a list and let's iterate in the range of values within the length of data frame.

We would like to parse.

And, take the column of media:, where I'm going to store this as a dictionary and call in the chain, map.

Taking.

All of the arguments for the data frame that I would like to parse and then you're going to see something kind of funky.

So, I'm using the iterator but.

Then I'm also going through all rows and columns to the end, all right.

And.

Then I'm going to call in my good old trusty list and append a little old friend, and let's see what this looks, like.

Now.

This is going to get us most of the way.

So.

This spreads everything out from the media, column.

Mind you, we we're going to concatenate at the end and put everything together.

Because.

This is one column that we are parsing.

And.

If you notice, we still have the metadata that we need to take care of later.

We can do this another way and if you notice there are also rows that are completely null that we'll take care.

Of.

Let's do a list.

Comprehension.

Let's, just call this merge; and I just found this simple little function online on stack, overflow., So, let's iterate inside of this/ And, then we would like to return the dictionary with everything.

And.

Let's do our list comprehension for the items inside of our list.

Now, I'm going to I don't know, I'll, just say..

I'll, just call this: function, (func) with iterator to do a list.

Because we're going to iterate inside of the data frame as usual for the media.

And.

Then, let's make a temp variable and iterate through each piece., Let's append everything.

Let's call that out put it into a data frame and there we are.

We get the same exact thing doing a different way, okay.

What, you could do is check how fast each one of these operations is versus the amount of memory using a profiler and see which is more beneficial out of these two.

If you're curious or more inclined.

Now, let's do a super crazy version.

I would say.

But, don't get scared, it won't bite.

So, let's walk through it, so I need to create a default dictionary, I'm going to create a empty list and I'm going to iterate, like we've been doing through this specific column.

And.

What do I want to do?? I'm going to check if the length of this is greater than zero.

And? What's going on here? Well? Basically, each list can be empty or filled which will equal one, because, if you're looking at lists that we're iterating through some will have something some want.

So if it has something, then we will then iterate inside of it.

Take, our iterator to go through each individual one.

Going inside of a list of lists.

We're.

Taking this because, it's now going to be a dictionary converted into tuples using items.

Which.

Would be useful for creating our default dictionary.

So? I could just leave a note here, so we could understand that we're converting a dictionary to tuples and iterating.

We would like to append the index value as well as j.

Which is going to be our tuples.

And you'll see why this is important in a second.

Because.

Basically we're trying to keep the frame of reference using the index with the tuple.

Because.

If it is not like this list, we need to figure out a way to take care of that.

So, we're going to I'll make a note of this in a second.

I'm, going to create a tuple and I'm going to zip together the data frame for the first element and I'm going to take the keys.

What.

I'm trying to do here is anything that didn't fall into this.

I need to create a tuple that has the same keys that are inside of our dictionary and then multiply it by the length or pardon me.

Yeah, and the values are going to be multiplied by the length of the keys, so it matches up.

And.

I want to mention something very specific here:.

If you had dictionaries that had mismatching keys.

This would not be what you would want to do right, here.

Since.

I know that for this specific example, it does work.

I'm using it.

If.

It did not.

You just have to do it a little bit, differently.

Okay,! So, what's going on with this? We have our first index value and we have the type the subtype, the caption, copyright, approved, metadata, right.

Then does the same thing for all of these.

Now.

Let's skip down to number seven, and this is what I wanted to match up, because when I create the default dictionary, here's my key as the type and then each value.

I want it to line up properly, and if I didn't do this else statement, we would have had an issue.

So.

Let's finish this up and let's create our default dictionary for the first entry in the tuple is the key.

The second entry is our value.

Throw.

This into a data frame, see what that looks.

Like.

But, I made a mistake.

There we go.

So.

This looks like we're on the right track and we did the same exact thing.

Once again.

Let's scroll, down.

So, we did the same thing., That's, perfect.

Let's, just call this new df, clean this up a little bit.

So it's not ticking up on my page.

So.

We did pretty good here.

But.

I want to note something once again.

If.

You have a list of dictionaries of varying lengths.

You need to do one more step, all right.

Don't forget, that.

We will consider doing something.

Like: basically find the unique keys and then you could perform.

If you wanted a set operator.

So, then you would have all of the keys and then you could do an if else statement to say if it's there put this, if not do that.

So here is basically what was going on with creating our tuple, so it cooperated with the formatting type.

The innermost list of lists for a media metadata is very interesting.

This threw me off and took me some time.

That's why this video is delayed.

So.

Basically, what I'm doing is I'm creating this function, where I have my default dictionary once again.

And, I'm iterating through my data frame column of choice, checking if it's a string.

Basically, if we don't have a list of dictionaries, then it's going to be a string value, saying nope or whatever I declared, or you have some empty value or whatever.

Then.

We have to iterate through our dictionary; turn it into tuples again and use our default dictionary and return.

That.

I have to iterate inside of the column.

Do the same type of comparison and then throw in our function has our default dictionary.

Then.

I used an else statement: because.

I wanted to keep the formatting.

S entry number 7, 13 and 16, or something like that.

Do not have a list.

And.

They are strings.

So.

We take care of that by creating this zipped tuples for our keys and our values, and I chuck that together.

Then.

I create a data frame and expand it.

And.

So for each entry in this: remember there were three urls, three formats, heights, and widths.

These.

Last three columns are always the same.

So.

You have this height, and this particular width for this thumbnail.

You have this format with this height and this width, etc.

And.

It was difficult to distinguish.

What's going on with these, you can see that each one of them is different, all right.

So.

I know the formatting is there.

I have this set up as a list of strings.

So? If you further decide to expand this out and flatten it:, if you needed to you, have that option.

Easily, all right.

So just scroll back up, and I want to mention this.

I decided to print these out.

So.

I can actually do a comparison if you see 10/3 10/3, 10/29 10/29 and you look at them and they're all different because.

They have this different separator for these images or whatever they are, right.

It looks like I did the formatting for everything: properly.

Then we concatenate and put everything together.

The original data, the new data frame with the media, and the expanded url information.

Now.

We end up with the same amount of rows.

So that checks out.

But.

Instead, you have 32 columns because it's expanded out now.

All right.

So, let's scroll back to the top and look here's everything that worked out from our nested stuff that I wanted to retain as a list.

I kept this just as a reference.

And.

Then these short ones right here came from the original media, all right.

And, then the rest of them are just the original data.

All right.

That wasn't too bad of a video.

Last thoughts.

I'd just like to say:.

If you wanted to speed this up, consider using list comprehensions, vs loops.

Understand that there are times when indexed values forgot, the e.

When index values actually do matter, and you would like to preserve the order.

So.

You need to adjust your code and remind yourself of that.

And.

Finally, people could write elegant code.

All the time.

But, it just comes with experience.

But, do realize that you may see elegant code, but it may not always be the most efficient.

May not always solve the problem.

Faster, okay.

So, don't get hung up on thinking.

Your code has to look the best.

Just, leave doc, strings and notes.

So people understand what's going on.

But.

As always.

Thank you for watching this video.

I hope I brought utility to someone.

Please, like share and subscribe, and if you subscribe turn on that notification, bell.

I'll see you in the next video.

Bye.