The Mega Evolution of Pokémons in a Graph (Neo4j Online Meetup #61)

Just another WordPress site

The Mega Evolution of Pokémons in a Graph (Neo4j Online Meetup #61)

– Hello graph nerds! We’re here I know someone in the live chat said we were late and they’re right, it’s 11:31 now, technically we are one minute late So, sorry for holding you guys up, we were getting ourselves set up So thanks for joining us today We have Igor I’m not gonna pronounce your last name because I feel like I’ma mess it up Rozani? – Yes Rozani, very good Okay, Igor Rozani who’s going to be doing a talk on his project, “The Mega Evolution of Pokemons in a Graph” and if you watch before you might know me, I’m Karin Wolok I’m the community manager at Neo and we have Mark Needham Hi Mark – Hello – Developer relations engineer at Neo4j And yeah, I think we can get started Right? Is there something that I’m, is there’s something else I should say Mark? Oh, live chat – Yes, if you have any– – Live chat – Yeah, if you have any questions, you can ask those in the live chat if you’re watching live, otherwise, we’ll put a link to the Neo4j Community forum where you can ask those if watching afterwards Igor is going to be showing some code I think I did talk on this some viewers of the Neo4j browser So you’ll want to make sure your resolution is set to 720p or higher and I guess the other thing is we found the project, the Pokemon project on a random sort of search of GitHub and we were like, “Oh, this seems very cool” So I’m looking forward to this one and I guess maybe we can hand over to you now and you can you can introduce yourself and the project as well – Okay Let me just share my screen – [Karin] Okay Set you, I’m going to set you as a presenter – [Mark] Cool. Yeah, I can see it now – [Igor] So it’s showing? Okay – Yeah – [Igor] So I’m going to present to you guys The Mega evolution of Pokemons in a Graph First starting, my name is Igor Rozani So I’m from Brazil and I have been working with software development since 2010 and most of my experience is with dotnet applications and I got into Neo4j recently and I started doing a few database just to to know it a little better and if you want to contact me, these are my contacts is basically “IgorRozani” everywhere So GitHub, LinkedIn, Medium or in the Neo4j Community forum – [Karin] Yes! (Igor chuckles) – [Igor] And this is a quick summary from my presentation So I’m going to talk what is this project, the pages that I utilized the scrapper, the database and what what I’m planning for this project So the project is kind of simple It’s a scrapper that reads the Bulbapedia database sorry, the Bulbapedia pages and transform it to a Neo4j script If you don’t know what is the Bulbapedia, I’ll find here, Bulbapedia is a famous encyclopedia about Pokemons So I basically come here and read the data from the page and generate this script It works like a–the Bulbapedia works like a Wikipedia So it’s open for everyone to change and add more informations and there are really rich informations about Pokemons here So since it’s a scrapper I need to scrap some pages So which pages I’m utilizing, I used four page from the Bulbapedia That’s one is the list of Pokemons So here you can see all the Pokemons that exist in the that exist in the Pokemon world And they have just all When I started watching it was 150 and now it’s more than 800 Pokemons And in here, in the end there is this one’s with not so much informations because they were not released yet It’s because it’s for the new game The second page that I utilized is the list of Pokemons by evolution family So here you can see all the Pokemon evolutions all the informations that you need to see, who evolves to what and what you need to evolve The third page is the list of Pokemon with form differences So here you can see, after the generation two they started to to include Pokemons with different forms So, and here you can find all the data that you need

all the information from that different Pokemons And the last one is the mega evolution that was introducing the Generation X & Y The last evolution for Pokemon can mega evolve to a new Pokemon temporarily And here you can see all the possibilities Whoops, wrong So this project, I split in two repositories in two different projects and one is the scrapper and the other one just the database So, the scrapper I developed using C# with dotnet core And it’s just a console application that you go and read all that that goes reading and generate this script Let me close out, please Okay So, if you– This is the Visual Studio, is the project okay And, so if you’re not familiar with dotnet in the program is where all the magic happens So here I go and call everyone Call the–Let me run first, sorry So, in the programs that where it run the main application, and you can see here that what it does, so it goes scrap the list of Pokemons, the mega evolutions, the evolutions that forms, generate a script and save this file Pretty simple To be to be able to run it I had to create a few configurations that you can see here So I have the Bulbapedia configuration that basically have all the paths from the pages that I need, the images and where I’m going to export the file A curiosity is that when you go to the Bulbapedia, looking up my page here, the link here it’s not the real link they have kind of alias So, when you try to scrap these pages like this way, will not work because it’s going to read direct so you don’t have a real HTML to read So, I had to find the real link and that’s why they are different So the mega evolution, this link is exactly the same thing as this one So, but this one is possible to grab the data In my project I separate in a few context So, I have just the representation of that configuration in some class here in the configuration And then I create the models that is a representation of the data that I grabbed In this relational thing that relates in the object form, so in a class So here you can see the Pokemon and all the data that I’m grabbing So the numbers, the name, the pictures, the profile, all the types, evolutions, forms, mega evolutions, and regional variants So here it’s basically takes the basic structure that I will use to generate the graph database And and here in the service I I separate for three little context So following just in order here from the application So to be able to run the project, to be able to scrap this datas, first, I need to go and grab the Pokemons that exist in the Pokemon world So I came to this page, and I basically read all the data from this table For reading this was kind of simple, I look for the main DIV in the page and look for all the tables because these are all tables, it is splitted by generations but but they are just tables so I just came here and read the data from the row so we read the number, then the rational number, then the natural number, the picture, the name and the types So this is here in the scrapper and every scrapper has its own folder So the first one you need to run is the Pokemon list

and here I come and it’s pretty simple as I said just read the data and from this page I capture two different data The first one is, of course, is the Pokemons and their regional variants That is the–because in the Alola version they released some new weird Pokemons that were different from the traditional ones like this one, that’s the Sandshrew So that is the normal and in this Alola version like Sandslash and Vulpix and other ones So in this first scrapper I go here and read this table all these two data and generate a list that I use to– that I’m going to use in the other parts of the scrapper Okay So the second scraper that I goes is the evolution so I came to this page and again look for the tables on it And I go and grab all the evolutions So here I grab what is the condition and who evolves to who And this page was a little more trickier because there is not a specific pattern like this one That’s always a Pokemon with all the same But here there is Pokemon that can evolve two times, Pokemons that can evolve only once, Pokemon can that can evolve once and have two options or just Pokemon that doesn’t have evolutions like Farfetch’d So this one I have to do a lot of treatments to see what I’m reading and how to fix and going fixing it in my– sorry, and how to fit it in my objects So here you can see that there is a lot of ifs and specifically codes for just for that And from this list I grabbed two to kind of data, the evolutions and the Unown evolution–and the Unown forms Where is it? Here Because here is the best place that I found about the Unown that they have the here’s the specific forms Okay. So after it, I went to the mega evolutions and the mega evolutions is kind of simple like the list of Pokemons I only come here and grab the table and go checking grabbing the data So, to mega evolve a Pokemon you need the mega stone and the mega evolution can may change the types So so I had to scrap it it trhough And just that Here I only grabbed the mega evolution and the stone And it was, it had a few tricks that like Charizard or Mewtwo that they have two mega evolutions So it’s just that I read and generate this data Again, I grabbed the list of Pokemons and update the list with the mega evolutions for the specific Pokemon And the last scrapper that I created is the scrapper for the forms And this one was trickier too like the evolution Because, first off there is a few a few evolutions that’s not– Sorry, a few forms that are not actually forms It’s just a Pokemon with dresses like this Pikachu, so I decided to ignore it and they are not, they don’t exist in my database And as I showed before the Unown here is kind of bad showing, so that’s why I decided to read from here and not from the forms And if you’re not just familiar with Pokemons, these forms, they are like, there’s Pokemons that can go and change forms according to specific triggers like this Castform that the painting of the weather will become one thing And there is Pokemons that they can just change forms for dependening what he wants to do and there is other ones that they are just variations of the same Pokemon So here I came

and look for two kinds of specific tables So I look for this kind of table like these on the Deoxys and the ones in in the like this one from the Castform With these two I was able to read all the forms from the Pokemons that are in this database So these are the first scrappers that I created for now and after it I had to create–I have to export my class that My class of Pokemons and generate a script So for this I created my class that will that basically that will generate this script for me So, first off I create a class that was able to create the “Create” command And the relationships, you create commands from nodes and relationships And here I basically passed by the Pokemon which Pokemon and go organize it in the structure that I created And after you’re running you can just– Sorry, you can just come here and see that this Pokemon file in here is all the data that I get from Bulbapedia So you can see all the types, all the Pokemons, – [Mark] There’s a lot of Pokemons – [Igor] all the forms, Alola versions, mega evolutions and the relationships between the nodes that’s there are the evolutions, the other things So basically my scrapper is that It just go reads and generate this script And with this kind of–with all this data, I created this database So I decided to create only four types of nodes So I have the nodes of Pokemon form type and mega evolution The type is pretty simple, it’s just a name And the Pokemon as I show you like here in the page – [Karin] You ready? – [Igor] Yeah So, I’m saving all this, all this data that’s showing here I generated this nodes with this information So it has number, the–No, sorry The name, the numbers, the pictures, and the profile link for the Pokemon that is this link So since a Pokemon has a type, since the Pokemon is from some type I had to create this relationship between Pokemon and type So a Pokemon is this type And in this relationship I just decide to not store any data just doing the relationship And a Pokemon can mega evolve so a Pokemon mega evolved to a mega evolution And as explained before the mega evolution has a name and I grabbed the name and the picture and for trigger it you have to have a mega stone So in the relationship between these nodes I decided to store the mega stone and the imagery in the mega evolve type relationship And a Pokemon can evolve to other Pokemon So so and to to evolve you have to have specific reasons conditions, sorry, it can be by leveling up, the Pokemon can be using a specific stone or for knowing some move So because of this in the evolved I decided to start the conditions of this And and then I can include a Pokemon that evolves to another Pokemon The less data that I have in my database is the forms So a Pokemon can have multiple forms or just one form So in the form I decided to store the name, the picture, if it’s Alola and if it’s interchangeable So the regional variants that I commented before that is the Alola specific Pokemons

I decided to not create a specific node And I just keep it in the form because basically they are the same they are just a different form from that same Pokemon So they are stored here and I identify it with this “isAlola” put here And it’s important to know that this is interchangeable because as I said before Pokemons, some Pokemons can just change forms depending of the trigger like normal, some like this Castform that can trigger depending on the weather or these depending of what kind of moves you want to do and this are according to the season and there is Pokemons like this one that they just can change if they were born in the West Sea they are pink, if they were born in the East Sea they are water No, sorry, they are blue And since the– And just that So a Pokemon has a form, has different forms So this is the representation of this kind of data And since the forms and the mega evolutions can have different types I decide to recreate the relationship between the Pokemon and type in the form and type and the mega evolution with type So both has the specific relationships for that – [Mark] Igor question for you before you move on to that Just for I guess the people watching, is this the first graph model you’ve created? Or like well, maybe you can explain about your process of learning how to do growth modeling – [Igor] Okay This is not my first modeling project Whoops, I closed I’m sure I have here in my GitHub I created one before that was a mapping of So, in Sao Paulo the city has this metropolitan transport network and I use this one to study Neo4j and create my first project, starting to understand how to model the data from this So I basically grab this image and generate this – [Mark] Oh wow, that’s a much more complicated one – [Igor] Yes. It’s a lot more complicated It’s still to not so good There is a few data that I don’t think I have still putted right I want to refactor it But with these I spend a few months trying to– Breaking my head and discovering how to create this database study And after that, when I got a little happy with these I decided to just create another project like this one with the Pokemons because I wanted to develop a scrapper and I wanted to study a little more from Neo4j so I decided to unify this is two project in just one I created my scraper in my other database with Pokemons – [Mark] Cool. So if you would like say, say someone’s watching and they’ve never done any graph modeling or never created a graph before what would it be like your main piece of advice if you have got a data set, like I found a website I’ve scrape it, how would I decide what should be a node, for example, like as a starting point? – [Igor] So I watched some videos on the YouTube, that were explaining the basics so there I saw what is a graph, because when I started, I will be honest, I knew nothing, and I saw it and I thought like what a weird database and so I start to try to understand better and where to apply it and I watched some YouTube And then I started to Let me – [Mark] Ah, the GraphGists, cool – [Igor] Yes, there is this good example of projects that here in the Neo4j website that I used to learn about it Even there is one of Pokemons that I kind of based in this one too Right here, this is the Pokemon X & Y it’s a really good graph of the base and there is a lot of good explanations And these were basically my guide these projects and the documentation,

the docs from Neo4j – [Mark] Cool. Yeah, and search, just for everybody else, this graph, just if you search on another tab, maybe for Neo4j Graph Gallery, you can now actually install a graph app into the Neo4j desktop which has many of those Yeah. So the top, the top link on there – This one? – Yeah So if you scroll down you can see these are now up you can actually access them directly inside the Neo4j desktop and you can if there are other guides in there that you can just play straight onto the database Cool. Okay, so that’s it So that’s you suggest that reading through those just and trying to find if there’s one similar or if not luckily for you and your case exactly the same as the domain – Yes – [Mark] that you would like to model Cool, okay – Yes, it’s a really good material that they have there – Cool – That you guys have here – [Mark] You carry on then – [Igor] Okay I’m not so good at “keries” Queries, sorry but I created a few examples So let’s do it and show what you can find here with this database So I’m using the Neo4j desktop version And here I have all my Pokemon in my script running And let me grab here So I put all the queries in the repository So if someone wants to go there and see and play so feel free to do it And so the first thing that I created after this was: “How many Pokemons– Sorry. “How many types– Sorry “How many Pokemons has in each type?” So, what is the quantity of Pokemon per type? And with these database we can easily solve this by this query And you can see here that the most common type of Pokemon is “Water” followed by “Normal” Pokemons And the most rare one is the “Ice” And but since a Pokemon can have mega evolutions, different forms, the discount can change a little bit So basically knowing all this all the possibilities of a Pokemon What is the still the most common types? So, “Water” is this to the most With this query we can search and ignore just not looking a specific for a Pokemon, we are looking for everything And here we can see that for the “Water” is the most common Pokemon it’s still the most common but “Normal” is not any more than the second it dropped to the fourth And now it’s “Flying” and “Psychic” the second and third And still the most rare one it’s “Ice” Other interesting data that I’m able to to grab with this project with this database is that “Which Pokemon has forms and how many forms they have?” So you can see here that, for example, the Unown, they have 28 possibilities forms, different forms So – [Mark] Are these one-to-one relationships so you can only have one form and be of one type or can you have multiple forms or types? – [Igor] Yes, there is the types can have, a form can have max. 2 types but a Pokemon can have any types Like these Unown they are all the same types Where is it? Here they are all the same one, but they don’t change the specific type like here they are always “Psychic” But for example if you go and check where’s is this Pokemon the Rotom he has six different types Sorry, six different forms and each form has a different type So they are “Electric” and “Fire”, “Electric” and “Ghost”, so “Electric” and “Water” How’s that? – [Mark] Yeah. Cool – [Igor] And, so in here you can see, yes, I could have included the types and other things, but it was just really simple query The other example that I have here

is “What are the mega evolutions of a Pokemon?” So I go and check the relationship between Pokemons and mega evolution and I export here in this this table all the wishful so here you have the Pokemon that can mega evolve and what is the name of this mega evolution So here you can see that most of the Pokemons have only one, but one–only one mega evolution, but there is two Pokemons that can have two mega evolutions That is the Mewtwo And Mewtwo X, and Mewtwo Y, and the Charizard X and Charizard Y And I have more two examples here The other example is that “What are the different forms and mega evolutions that a Pokemon can have?” So, I’m basically putting together the previous, the two previous queries in just one So you have the Absol that has only the mega evolution or you can see that sales has all these 18 different forms And I think it’s just these For me it’s an interesting thing It’s a really interesting thing And the last example that I have here with me is So you wanna, you have a Pokemon and you wanna know what are the possibilities of this Pokemon what he’s going to be able to evolve, so here for example, I have the Charmander And here you can see that the Charmander can evolve to Charmeleon that will evolve to Charizard, and you can have two possible mega evolutions, that is the mega Charizard Y and mega Charizard X And – [Mark] So using the sort of star on the query here right next to the mega evo, can you explain that for everybody – [Igor] Yes. So if you don’t put the star you are going to only grab the next evolution of that Pokemon So when I put the star I say I wanna all the paths from that relationships So with these I can pass from the Charmander to Charmeleon to Charizard and the mega evolutions – [Mark] Sou you do like four different evolutions – Yes – Oh, cool. Okay So then I guess this is the one of the queries, like you could imagine the previous ones you’ve showed, you could reasonably easy reproduce– reasonably easily reproduce those in a relational database But this one would be, this is now, like, is quite difficult to reproduce this quite a graphic query if you like – [Igor] Yes, this one’s the the most graphic query that I’ve created and yes, we’ve found traditional database I would have to do a lot of joints I wouldn’t be able to do that I’m going to tell you, it would be really complicated But with this is just, I put this just easy star and problem solved And an important thing that I forgot to tell about the model of my database is that I’m not ignoring the direction of a relationship So mostly in the Pokemon evolve to another Pokemon because the direction of this relationship is important to you know that who is the previous evolution who is the base and where he’s going to evolve And a Pokemon cannot devolve I don’t know if this word does exist but cancel this evolution and go back to that evolution So it’s really important to consider in that the direction of these relationships and that is another detail here in this query that there is the direction, right here, to identify that And and the project is not over yet I still want to add more data to this database and create more stuffs So one thing that I want to do is add the moves from the Pokemons

So I want to grab all the possible move that a Pokemon can have and storing in this database and I think it’s going to be really cool to have this data And do you have more questions? What I have to say is my presentation is just that so we’re more open from more questions from the chat – [Mark] Oh, yeah, and there’s no questions in the lobby You got a lot of fans watching But they are all writing in Portuguese, so I’m not sure I can try and guess what there’s that but– – [Igor] (laughs) They say they were doing– They were going to do a little mess in the chat saying that and I spammed to everyone I know – [Karin] (laughs) That’s funny – [Mark] Yes, I guess, I mean, I can think like what would you do, like if you did your project from scratch again? Like, what have you learned from doing it? Like what, guess I don’t know, what lessons have you learned that if you did it again you would do differently? – [Igor] Well, I learned with— It’s a simple project but with that I learned a little more about the relationships My preview project I was ignoring the direction in a relationship So it wasn’t important And with this I learned that the importance of a direction in a relationship I know that there is case that it will not be important But here I think it is And and it was in the start was really really complicated to put the data in the right place So for example in the form, I’m still not– Let me show the slide In the form, I’m still not so happy about the place that I put is interchangeable or if it’s Alola Maybe it should be here in the “HAS” But I don’t know maybe in the future I can change but– – Yeah, I think for that one it kind of sort of depends right? Like if you were to move it to the relationship then what you’re kind of indirectly saying is that there could be two Pokemon where they both have that relationship to the form but it’s possible that they have different values for “isinterchangeable” or “isAlola” whereas if given that they are of particular form they always have the same values for those two things then what you have it is absolutely fine The only reason you would move it is if there were some reason why it might be different for a specific Pokemon – [Igor] Okay, so it’s right, like, so since it doesn’t change, so I think they were put in the right place – [Mark] Yeah, just kind of duplicating the same information and there’s not really any advice doing that One thing I notice which works fine in this example, but I would do differently if you had to work with a bigger dataset, is if you– Do you have your Sipho script, where you created all the data all the data, handy? – [Igor] Yes – [Mark] So I think you did it actually exactly the same as what I’ve done on my first graph project You generated like all the “Cypher” statements or the “create” statements – [Igor] Yes – [Mark] Yeah. So this works really well for small amounts of data, but if you have to do or deal like with, I don’t know How long does it take to run that actually? Is it pretty quick? – [Igor] Yeah. It was pretty fast I can run here, just to show you – Yeah, so– – There’s not so much data – [Mark] Yeah, if you were working with the big dataset what I would do instead of generating the decipher because it takes a long time for it to to compile and run if you generate the script, but this is exactly the way you would likely do it If you had done the sequel before An easier approach is to actually create some CSV files where one rep row is representing each of the items in here and then you can use the load CSv command to iterate through the CSV file and going create, either create the nodes or create the relationships between then Another one for watch I mean for this works absolutely fine, but if you were to work with a bigger dataset say with hundreds of thousands or millions of items it would handle that better If you try to generate a million rows of this you’d probably–(snickers) you’d probably crash the site I expect – [Igor] Yes, I didn’t know about this way of creating database, but that’s really interesting And yes for this one

I wanted a little bit just through Since it’s a short I don’t think it would be a problem And another thing that I could have done is connect to the Neo4j and insert it element by element and not generating a script – [Mark] Oh, from your, from the SeaShell Code – Yes – Yeah Yeah, that’s a good idea Yeah, that’s another, maybe another way that you could do it – [Igor] Yes Yeah, I think there is a lot of improvement yet to– that I can do in this project – [Mark] Yeah, I mean, this is always the case isn’t any project you build there’s always some things that you can change here but I think it’s really nice what you’ve done already – [Igor] Thank you – [Karin] Great job Igor – [Igor] Thank you And Okay, and I’m not sure you have more questions but before that, I want to thank you Mark and Karin for this amazing opportunity – [Karin] Oh, and thank you for, you know, creating the project and agreeing to present – [Igor] Okay, thanks Yeah, I was really happy for this opportunity – [Karin] Oh yeah, we’re glad, that’s awesome We’re really glad that you’re, you know, able to share because I think that a lot of people learn from it as well as you You know there’s some things that like with Mark just giving some feedback and thoughts on it too – [Igor] Yes – I think that’s helpful too So as you mentioned on this slide, there is the community thread So if people are not watching this live and have questions, you can go directly to the community threat I actually posted it earlier in the chat and then you can talk to Igor directly and ask him questions or you know comments, anything And I think that’s it Right? – The next thing we’re doing some football, right? I think – Oh yeah, be yeah – I’ll put a link to the to the meetup page If you want to register and grab the YouTube live link for next week’s also It’s the same time again next Thursday And again shameless self-promotion Don’t forget to like the video so that other people other Pokemon fans and Neo4J fans can find us – [Igor] And before I forget I have to thank you, thanks, Koji Annoura Sorry if I said the wrong name – Oh yeah, Koji – [Igor] Yes, He did a nice code review in my script He saw that there is some stuff not following the patterning from the decipher – Oh, nice – So, he helped me with that and thank you so much Koji – Koji is amazing Where did he help you with that? How did you guys connect? – [Igor] He emailed me He saw my project and he emailed me with the changes that– – Oh, so he saw in the community site – [Igor] Yes – Oh, very cool Yeah, Koji is amazing That’s nice of you to thank him on the meetup Alright, well, thank you again Igor and Mark – Yeah, thanks everybody for taking the time to watch and if you have any questions afterwards, yeah, don’t forget, you can know you can ask them on the community site, or you can ask them in the YouTube comments, – And we will see you guys next week! – Cool Thanks everyone – Bye