Hi, and welcome back. Following up on the tasks that we performed in the previous video, in this lecture, you're going to learn how to extract various pieces of information from a web page. This is going to be very interesting and very useful at the same time. So let's get started. First of all, let's go to the target webpage, the one I mentioned in the previous lecture, you can find the link attached to this lecture as well. On this web page, just right click anywhere and select view page source.

This option is available in any browser. I'm using Google Chrome here. But if you're using Firefox or Safari, the same feature should be available, perhaps with a slightly different name. Now, as soon as I hit view page source, a new tab opens up where you can see the entire HTML code of this particular page. Leave this page open and let's go to the Python command line for a bit to make sure that we have everything set up and ready for action. So here, I am just reusing the code that we already discussed in the previous lecture.

At this point, we are ready to start extracting data from this web page using beautifulsoup. And we will perform this extraction based on HTML tags back to the browser now. So let's say that we want to get the contents of the head tag right here. For this, just go back to idle. And let's use the Beautiful Soup object to get the desired outcome. So all we have to do is result dot head.

That's it. That is the content of the head section of your web page. Pretty easy. Am I right? Now we can also assign this to a variable for easier reference in your application code. So let's say head equals result dot head.

Let's also check the type of this variable. So type of head. This is called a tag object. Another important thing that you should keep in mind is that this operation right here returns only the content of the first tag by that name. So not only The tags with that name only the first one it points on the web page. Now back to the website.

Let's do the same thing, this time for a different tag, let's say an h2 heading. So let's search for the first h2 heading in our HTML code, h2. This is it right here. Let me increase the font a bit. So this would be the h2 header that we are going to obtain by using another attribute that I'm going to use right now in the Python interpreter. So simple enough, just type in h2 equals result dot h2.

Now let's see h2. So we have top items being scraped right now. Let's go back to the web page. And this is indeed the result we were looking for. Great. What about the third title tag, for instance, so let's search for title.

So this would be the result that we are going to obtain in the Python interpreter. Let's check this as well. So let's say title equals result dot title. Enter. Let's see title right now. And sure enough, we get the title tag and its content as a result.

What else, we can also access the name of each tag by using the name attribute. So let's try this head dot name, h2 dot name. And also, let's try title dot name. Of course, this returns the tag name itself, depending on what kind of tag Are you referring to? No surprises so far. But what if you want to change the name of a tag in your local instance of the web page?

Well, that's also very easy to do. Just use the name attribute and the equal sign as an assignment operator. For instance, for the title tag, we can just use title dot name, equals, let's say my title. Now, if we check this tag, once again, we're going to see the name change in effect. So title, and indeed, this is the correct result. Next, let's talk a bit about the HTML tag.

The themselves, each tag may have any number of HTML attributes. For example, going back to our web page, let's consider the header tag on the web page. This tag has two attributes. The first one is role, and it has the value of banner. And the second one is class having three values navbar navbar, fixed top and navbar static. That's why in this case roll is called a single valued attribute.

And class is called a multivalued attribute. Now, first of all, let's extract the header tag and its contents as we did earlier with other tags. So let's go to the Python interpreter and just use header equals result dot header. Now let's see the header. Let's double click on this button. Okay, this is the header right here.

By the way, you can also use the predefined methods to have the header content shown in a much nicer way. So let me show you print of header dot gratify enter. Okay, this output is a lot better a lot nicer and cleaner as compared with the previous one. Now back to the HTML attributes of the header, we can use an object attribute called add RS to view the tags on attributes as a dictionary having the HTML attribute name as a key, and the attribute value as the value in that specific key value pair. Let me show you what I mean by that. So let's sorry, let's use header dot add Rs.

So as I previously said, using the A TT RS object attribute, we can get a dictionary with the HTML attributes as the keys. Notice role and class right here, and the HTML values as the values in the dictionary. Also notice that four multivalued HTML attributes you get a list of the values very neat and clean. Great. Now, as with any other dictionary, you can perform various tasks to play around with its elements. For instance, let's see how to access the value of a specific HTML attribute in our dictionary, well, very easy to do header of roll, Enter.

And similarly header of class and we get the list. Okay, we can achieve the same results by using the get method like this. So we have header dot get of roll. And likewise, we have header dot get of class, and the results indeed are identical. Please note that this is the get method from within the beautifulsoup module, not the get method from the request module. Okay.

Furthermore, we can also modify a text attribute value as we will do inside any other dictionary. To do that, we can use Heather of role equals, let's say, I don't know Sam's I'm not very creative today, enter. Now let's see header dot add Rs. Notice the value of role being changed accordingly. Great. What else?

Let's add a new tag attributes to our dictionary. For instance, let's say we want to add header of, let's say new equals Python. Now if we use header dot add RS once again, there is a new tag attribute and its value. Finally, let's see how to remove a tag attribute. Well, simple enough as we did with dictionaries, they'll have Heather new. Let's check this once again.

And the attribute called New is now gone from the dictionary. Pretty straightforward, I think. Next, I'll show you how to extract the string, the text part only from a certain HTML tag. And let's consider the h2 tag from earlier in this video. So h2 assuming that we want to extract only the string this string right here Top items being scraped right now, all there is to do is just use the string object attribute like this h2 string, and then the string nice and clean. The last thing that I'm going to show you in this video is how to zoom into a certain area of the HTML tree on your web page.

What do I mean by that? Well, let's take a look at the header area once again, inside our HTML code. This is it right here. You can see that it has a div tag nested underneath it. This div tag has yet another div tag nested, so this one right here, which in turn contains on a tag, bottom tag, and for span tags at various levels of nesting. Let's say that you want to access the bottom tag.

So this would be this one right here, along with its content, which is several nesting levels underneath the header tag itself. How would you refer to the bottom header given this HTML? Code structure. Well, it's way easier than you might think. Let me show you. So we can use result dot header dot div dot div dot A dot button.

This way you can zoom into whatever part of the HTML page you desire. In this case, even more convenient would be to use result dot header dot bottom, since this is the only button tag in the header, and the result is, of course the same. However, keep in mind that using the tag name as an attribute will return only the first tag by that name. For instance, here we have four span tags, these ones right here, but when using the same zooming method, only the first one will be returned. So let me test this. We have result dot header dot span.

And indeed We have toggle navigation, which if we look on the website is indeed only the first span tag underneath the hether tech, you will learn how to find all occurrences of a certain tag in the next lecture. So I will see you there