How to Fix Robots.txt Not Found
How to Fix Robots.txt Not Found In this video, I’ll explain to you what that is, why it’s important.
Why do you want to fix it, how to fix it, the tools you’ll need to fix it, and then if you don’t have those tools, I’ll show you a pretty cool little shortcut in fixing it escalate probably better and easier than actually using the tools.
But that will be at the end of the video. So first the robots.txt file. What is that?
So that I think the robots.txt file is like the bouncer to a website and it’s just like the bouncer to a club or a music venue, concert, or something.
The bouncer is like the person with the list. So you’re, you’re going to an event. You go to the front door, there’s somebody at the front door who has a list.
You give them your name. They look to see if you’re on the list. They either let you in, they don’t let you in.
Maybe they let you in with restricted access or maybe they give you VIP access, etc., but they’re controlling. Who’s coming in and, and kind of broadly where they’re going once they get in, that’s kind of what the robots.txt file does, for your website.
It determines what browsers can be led in. So if a user is using, you know, Chrome or Firefox or whatever you typically gonna let either all browsers and or no browsers, and it lets the Google bot in and the other search engine being and Yahoo, those bots, it lets them in and tells them essentially where they can go.
Just on, on a broad sense, not, not specifically down to the URL level, but it kind of is it’s the first, impression that Google is going to get when it visits your site.
So you want, that’s why it’s important. That’s why you want to make sure that you give Google a good first impression, which is good housekeeping.
It’s probably, wouldn’t be a terrible problem if you didn’t have a robots.txt file for a little bit, but after time, you can imagine if like a concert didn’t have, you know, people checking tickets at the door, probably it’d be, it’d be fine for a little while, but eventually, a word would get out and you know, everybody, you know, you can imagine what would happen VMs.
So to fix this, you’re going to need, two things. You’re gonna need an FTP program and you’re gonna need a text file editor.
So you’re first going to need to create the robots.txt file. This can be any program that can create a text file.
Anything that just ends in .txt, in the Mac world. I think that’s a text editor. There’s probably a lot of them.
Text editor is the one that, comes default. I don’t know what the default text editor is for windows, but I’m sure they have one.
So the first thing you would need to do is create this text file, and then you’re gonna need to connect to your server via FTP.
So in order to do that, you’ll need an FTP program. So I’ll show you what this looks like. So I’m using an FTP program called transmit, and it’s a pretty good program.
I recommend it. It’s just for Mac. It’s not free. However, I don’t know the cost, but I think it’s, 30, 40, $50.
It’s not cheap. But it is a good one and I use it a lot. So, it’s a good investment for me, but the first thing you want to do is you’re going to use your text file editor and you’ll create a file and call it, robots.txt.
And this is the text that you’ll want to put in it. These first three lines are essentially default. As long as you’re using WordPress, you’re going to want to have these.
And then there is the fourth line, and you can skip a line here and not, it doesn’t really matter, but you’re going to want to specify where your site map is.
So you’re just going to type the word site map, colon space, and then the full URL of your site map and I’ll put all this text below so you can copy and paste. All you’ll need to do is obviously remove my domain, replace it with yours, and then make sure it’s pointing to your site map.
So once you’ve created this file saved it open up your FTP program, which will probably look something like this. You’ll want to take this file and drag it into the root directory of your website.
So this is the root directory of my website. These are typically called, you know, public underscore HTML. You will be able to identify it because it’ll have an in there in this directory, it will, there will be a file called index dot PHP.
So you take this file drag and drop it here, and that’s it problem solved. Now if you don’t have, or don’t want to bother downloading, and connecting an FTP program to your website, which I can understand, because that can be, can be kind of difficult to be honest, FTP programs are, they’re known for being very finicky.
You can use the Yoast SEO plugin to do this, and I’ll show you how so. I’m not a fan of the Yoast SEO plugin, as in terms of using it for SEO purposes.
I much prefer the all-in-one SEO pack and I recommend that plugin. However, there’s a couple of neat little tricks that Yost can do this being one of them.
So if you don’t have the Yoast plugin go to your plugins page and you can click add new, and then in this search box right here, just do a search for Yoast SEO.
It should pop up. It should be one of the first installs once you’ve installed it. It will show up here in your list of plug-ins.
You’ll want to activate it. Now, if you’re using another plugin for SEO like I am, I’m using the all-in-one SEO.
You may get a warning sign here that, Hey, you’re using two SEO plugins. That’s definitely something to avoid. You really do not want to leave two active SEO plugins on your site.
You are guaranteed to have problems. So once we do this, we just need to make sure to deactivate the plugin, but for now, scroll down on your left side navigation over SEO.
You’ll see the little Yoast logo scroll over this, and you’ll get a dropdown. You can select tools. And then on this next page, click on file editor.
Now, this is going to show you a couple of the kinds of standard files that you will need on pretty much any website.
It’s going to have these, it’s going to have a robots., or it should have a robots.txt file and what they call an HT access file.
Since you don’t have a robots.txt file, it’s normally was showing it to you here. Instead, you’re going to have this button and you’re just going to click it.
And it’s going to populate it with this default information. Once you do that and again, you’ll want to add your, the fourth line here with your site map, and then your URL.
Once you’ve done that, click save changes to robots.txt. Then we’ll go back to the plugins page and we will scroll down to Yoast and we’ll deactivate that plugin.
So what we’ve just done is we’ve just installed Yoast, activate it, created the plugin, and then deactivated the plugin after creating that robots.txt file.
That robots.txt file is still going to be there. Deactivating the Yoast plugin will not remove that file and in fact, if we go back over to our, FTP program, we will see that this robots.txt file is here.
And if we open it up, it will even have, you’ll see, that’s not my accurate URL for the site map, obviously.
So I’m going to want to replace this. I’ll take this version, which I know is accurate and I’ll move it here and I’ll replace that file.
So that’s it. That’s, solved the issue now to verify that this has worked, this is a nice little trick.
You can go to your website, whatever it is, .com and then after the M of the.com slash robots.txt, click enter, and you should see your robots.txt file here.
If you don’t see something happen and, you can back up and go over those steps again. But if you see this, then you are, you’re all set.