Elxis CMS Forum

Support => General => Topic started by: bledi on March 05, 2014, 23:21:13

Title: Robots.txt file (elxis 2009.)
Post by: bledi on March 05, 2014, 23:21:13

Hello!
I don`t know why the last month google is crawling folders like: components/, modules/, (in elxis 2009. site) and it shows a lot of errors in my google webmaster tools pannel.
In order to avoid Goolge to crawl, index, these folders do i do anything wrong if I disallow this folders in robots.txt file:

components/com_maps/
components/com_weather/   
modules/mod_iosn_subscribe/   
modules/mod_iosn_subscribe/   
modules/mod_advcheck_availability/   
components/com_eventcalendar/
modules/mod_eventcalendar/
components/com_reservations/
components/com_eforum/
components/com_eblog/
modules/mod_iosr_roomdetails/
components/com_frontpage/
components/com_weblinks/

appreciate any idea before doing smth wrong.

Title: Re: Robots.txt file (elxis 2009.)
Post by: webgift on March 06, 2014, 10:20:41

No! It's not a big deal. You could disable by adding the following 2 lines to robots.txt file:

Disallow: /components/
Disallow: /modules/

Title: Re: Robots.txt file (elxis 2009.)
Post by: datahell on March 06, 2014, 13:20:40

It is not wise to disallow folders components and modules...

Title: Re: Robots.txt file (elxis 2009.)
Post by: bledi on March 06, 2014, 14:19:34

Hello guys, i was really thinking to disallow only the subfolders like:
components/com_maps/
modules/mod_advcheck_availability/ 
etc.
not the components, modules, folder itself, because google is indexing the subfolders actually, and what is under it.
Does this make any difference, or is still the same?

Title: Re: Robots.txt file (elxis 2009.)
Post by: datahell on March 06, 2014, 14:37:38

My recommendation is to disallow only folder includes, nothing else.

Title: Re: Robots.txt file (elxis 2009.)
Post by: bledi on March 06, 2014, 14:52:10

OK, thanks