GoogleBot crawl errors

[seadhna]

Hi there,
two Elxis sites are currently experiencing crawl errors - all Googlebots are banned in Defender. Is there any way to stop all Googlebots being banned like this?

[seadhna]

I replaced an entire Defender_ban.php with a blank file last week, but already today, all Googlebots have been banned again, and receiving 403 Access Denied errors on all pages of the site.

[xmanhattan]

Hello seadhna,

Are you using a robot.txt file in the root directory?

see attached.

[datahell]

Defender will not ban them except there is a reason for that. Even google is used by hackers to scan sites for exploits. See Defender's security log to find out why it is blocking google robot.

[michalis1984]

Maybe somewhere in your site there is a wronk link ... eg. by typo error at the plugin elxis link.. which drives to a page where is banned by defender. And so bots are banned by follow it. I experience this in the past.

[seadhna]

Hi, yes, I have the robots.txt file for both sites.

[seadhna]

looking through security.log now.
I have found and remedied this one regarding Bing - an incorrectly coded hyperlink:

2017-02-12 16:39:37 GMT [157.55.39.173 - msnbot-157-55-39-173.search.msn.com]
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
GET /news/issues-in-the-media/international-justice/Afghanistan%20http:/foreignpolicy.com/2016/10/31/exclusive-international-criminal-court-poised-to-open-investigation-into-war-crimes-in-afghanistan/
REFCODE: DEFG-0024 Request blocked, Method: URI, Reason: Remote file inclusion

[seadhna]

Here is one related to GoogleBot, is this unusual?

2017-02-16 14:24:23 GMT [66.249.64.94 - crawl-66-249-64-94.googlebot.com]
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
GET /administrator/includes/js/ajax_new.js
REFCODE: DEFG-0031 Request blocked, Method: URI, Reason: Common CMS scan

[seadhna]

hmm... I just got banned myself. It seems that just having two hyphens together in an SEO title produces a security warning when you try to navigate to that URL:

SECURITY ALERT
Request dropped!
PHP/SQL Comment
Reference code: SEC-DEFG-0035
Sorry for the inconvenience.

[datahell]

Yes, because the 2 hyphens represent a comment in mysql and it is a technique often used by hackers for SQL injection. I also strongly believe that you dont need 2 hyphens side-by-side.

[seadhna]

Ok. Crawl errors by Google seem have stopped now, after I resolved the double-hyphen issues in the security log (and some other issues, e.g. incorrectly coded hyperlinks on the site that were interpreted as attacks). I cleared the defender ban file and seems like Google Bots are not getting banned anymore.