Refer 2.1 and Robots.txt

Posted on 12.16.03. Tagged with All

I have noticed an increasing problem with having my Textism Refer page indexed by google and basically clog it up with useless results. People are searching for Foamys Rant, and instead of getting the script, they get my refer page.

I have come across this myself a few times in the last month, when searching things in google, and it is quite frustrating to think you found something only to be directed to their referal page. This can easily be remedied, however, by either creating (or adding) the following to your robots.txt file (which should be placed in your root)

User-agent: *
Disallow: /refer/

The star means the rule applies to any robot (as opposed to a specific one like googlebot). The next line tells any spider that comes here to not crawl or index http://gtmcknight.com/refer. If you want to add other folders to block, add another line with Disallow on it. One directory per line. If you need more help, check out Search Engine World's Robots.txt Tutorial

So go grab the latest Refer 2.1 from Textism (just released Nov 27th), please, please create your robots.txt file, and help keep Google clean for our children!

6 Comments

Holly
07:25 AM on 12.16.03

Foamy fucking rocks my socks... I got my dad into it, it's great.

kitta
12:43 PM on 12.16.03

That sounds like a TV ad...

"Keep our web clean for future generations"

And I created a robots.txt a while back, I don't want those spiders indexing some things. I really do wish they'd index me when you search for 'kitta' on google though. :o\

Oh and Tay, send me your postal address dingo!

Mark
01:54 PM on 12.16.03

I did that robots.txt thing a few months back. Same problem. It made me wonder how I got linked to these things in the first place. "holy crap this chick is hot" was always in my refer list, but Googling for it just brought up the refer page. Catch 22.

Taylor
02:34 PM on 12.16.03

Holly: Yeah, have you seen the 2nd one? Its not nearly as good.

Kitta: Always one step ahead of me. I just emailed you my address, and shoe size ;)

Mark: Actually, you are the first person I assumed would have one, but I tried going to shinyplasticbag.com/robots.txt and there wasn't one. Did you give up on it?

jamblo
04:13 AM on 12.17.03

Hey, I just checked my web stats, wanted to thank you for linking me!

Cheers
Jamblo - www.jamblo.org

Jenni
10:31 PM on 01.04.04

I know this isn't the correct place to post this, but I don't see any kind of troubleshooting document for Refer 2.1 and I'm having a problem. I cannot get it to stop recording directory names from within my site. I've done what the instructions say to do, but alas, it's still recording them. Any ideas on how to get it to stop??

Joe Grossberg
11:43 PM on 01.05.04
Trackback

If you're looking for a tool to log and display referrer information, I recommend Textism's "Refer". It's PHP and free.... Read more in Nice, Simple Referrer Tool »

Joe Grossberg
10:09 AM on 08.04.04
Trackback

update Hey, spammers — I don't have a publicly available Refer page, so stop sending bogus HTTP requests. I've already... Read more in Nice, Simple Referrer Tool »

you tube movie
02:26 AM on 05.25.07
Trackback

you tube movie Read more in you tube movie »

u tubee
02:06 PM on 06.24.07
Trackback

u tubee Read more in u tubee »