Elxis CMS Forum

Support => Security => Topic started by: seadhna on February 20, 2017, 12:45:51

Title: GoogleBot crawl errors
Post by: seadhna on February 20, 2017, 12:45:51: Hi there,
two Elxis sites are currently experiencing crawl errors - all Googlebots are banned in Defender. Is there any way to stop all Googlebots being banned like this?
Title: Re: GoogleBot crawl errors
Post by: seadhna on February 20, 2017, 12:56:20: I replaced an entire Defender_ban.php with a blank file last week, but already today, all Googlebots have been banned again, and receiving 403 Access Denied errors on all pages of the site.
Title: Re: GoogleBot crawl errors
Post by: xmanhattan on February 20, 2017, 14:58:56: Hello seadhna,

Are you using a robot.txt file in the root directory?

see attached.
Title: Re: GoogleBot crawl errors
Post by: datahell on February 20, 2017, 19:56:25: Defender will not ban them except there is a reason for that. Even google is used by hackers to scan sites for exploits. See Defender's security log to find out why it is blocking google robot.
Title: Re: GoogleBot crawl errors
Post by: michalis1984 on February 21, 2017, 01:36:18: Maybe somewhere in your site there is a wronk link ... eg. by typo error at the plugin elxis link.. which drives to a page where is banned by defender. And so bots are banned by follow it. I experience this in the past.
Title: Re: GoogleBot crawl errors
Post by: seadhna on February 21, 2017, 16:23:03: Hi, yes, I have the robots.txt file for both sites.
Title: Re: GoogleBot crawl errors
Post by: seadhna on February 21, 2017, 16:30:42: looking through security.log now.
I have found and remedied this one regarding Bing - an incorrectly coded hyperlink:

2017-02-12 16:39:37 GMT [157.55.39.173 - msnbot-157-55-39-173.search.msn.com]
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
GET /news/issues-in-the-media/international-justice/Afghanistan%20http:/foreignpolicy.com/2016/10/31/exclusive-international-criminal-court-poised-to-open-investigation-into-war-crimes-in-afghanistan/
REFCODE: DEFG-0024 Request blocked, Method: URI, Reason: Remote file inclusion
Title: Re: GoogleBot crawl errors
Post by: seadhna on February 21, 2017, 16:32:48: Here is one related to GoogleBot, is this unusual?

2017-02-16 14:24:23 GMT [66.249.64.94 - crawl-66-249-64-94.googlebot.com]
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
GET /administrator/includes/js/ajax_new.js
REFCODE: DEFG-0031 Request blocked, Method: URI, Reason: Common CMS scan
Title: Re: GoogleBot crawl errors
Post by: seadhna on February 21, 2017, 17:09:20: hmm... I just got banned myself. It seems that just having two hyphens together in an SEO title produces a security warning when you try to navigate to that URL:

SECURITY ALERT
Request dropped!
PHP/SQL Comment
Reference code: SEC-DEFG-0035
Sorry for the inconvenience.
Title: Re: GoogleBot crawl errors
Post by: datahell on February 23, 2017, 18:29:57: Yes, because the 2 hyphens represent a comment in mysql and it is a technique often used by hackers for SQL injection. I also strongly believe that you dont need 2 hyphens side-by-side.
Title: Re: GoogleBot crawl errors
Post by: seadhna on February 23, 2017, 18:53:02: Ok. Crawl errors by Google seem have stopped now, after I resolved the double-hyphen issues in the security log (and some other issues, e.g. incorrectly coded hyperlinks on the site that were interpreted as attacks). I cleared the defender ban file and seems like Google Bots are not getting banned anymore.