How to prevent defender from blocking Googlebot

[komang]

The visitor of my website drops to only 50% in last 2 weeks. When I check Google webmaster I got this notification

Googlebot for smartphones has identified that an increased number of URLs on https://ksmtour.com/ cannot be accessed without permission. Specifically, these pages either present a login page for mobile users or return a forbidden response code (HTTP 403). As a result, mobile users might not be able to access your content at all, and Google might not be able to show your pages in mobile search results.

and

Googlebot has identified that an increased number of URLs on https://ksmtour.com/ cannot be accessed without permission. Specifically, these pages either present a login page or return a forbidden response code (HTTP 403). As a result, users might not be able to access your content at all, and Google might not be able to show your pages.

then I checked the crawl error, there are 21192 pages on access denied for desktop and 829 pages for smartphone.

I tried to fetch some pages as Google and the result was error. I contact the hosting company and they said nothing error. I search on google forum and got no clue.

Then I found a clue here: https://forum.elxis.org/index.php?topic=8833.0

After I clear the "defender bans" I tried to re-fetch some pages as Google and they were fine.

My question then: how to prevent the defender from blocking Googlebot? Since Googlebot is blocked by the defender, Google kicked my pages from their index and I lost a lot of visitors.

Thanks in advanced,

Regards,
Komang

[datahell]

Elxis is secure with or without Defender. Defender is an extra protection shield for your web site to catch the "garbage" before reaching your site. If you see that Defender blocks these requests without reason then disable defender. However you should know that Defender is almost never wrong and there should be a reason why it throws a 403 forbidden error. In many cases the problem is caused by site owners. For example a wrong URL like this: http://www.example.com//something/ is not accepted by Defender. Many site authors make similar mistakes in their links. Google indexes your pages with such errors and then you always get 403 errors for visitors coming from google. But it is not Defender's fault to blame.

I advise you to see the URLs that cause 403 errors and pay special attention to these. I am sure you will understand why you have such errors.

EDIT
I forgot something important. If you use an old Elxis version/revision Defender will block POST requests without referrer and also block requests without user agent (security level above normal). I advise you this:
- Make sure in Elxis configuration that the security level is Normal.
- Make sure you have Elxis 4.5 installed.
- Use the Check file-system tool to find files that needs update
- Download Elxis 4.5 rev1909 (current release) from elxis.org and update the files that needs to be updated.

[komang]

Dear Datahell,

Thank a lot for your fast response.

In many cases the problem is caused by site owners. For example a wrong URL like this: http://www.example.com//something/ is not accepted by Defender. Many site authors make similar mistakes in their links. Google indexes your pages with such errors and then you always get 403 errors for visitors coming from Google. But it is not Defender's fault to blame.

I advise you to see the URLs that cause 403 errors and pay special attention to these. I am sure you will understand why you have such errors.

I'm sure in my case, that is not the problem because many pages that got error status when crawled by Google were popular pages that ever listed on the first page for a certain keyword. I tried to put double slash on the link, then I got a security alert and I can't access the page. In my case, I still can access the page manually when Google got an error.

EDIT
I forgot something important. If you use an old Elxis version/revision Defender will block POST requests without referrer and also block requests without user agent (security level above normal). I advise you this:
- Make sure in Elxis configuration that the security level is Normal.
- Make sure you have Elxis 4.5 installed.
- Use the Check file-system tool to find files that needs update
- Download Elxis 4.5 rev1909 (current release) from elxis.org and update the files that need to be updated.

I already have the 4.5 rev 1909 version, and the file system checking result is Okay. I just updated on Jun 18th, and this case happens since Jun 25th. Is there any correlation between them?

Overall Defender is okay, for almost 4 years I have been using Elxis, this case only happens once. But if there is a way to prevent Defender from blocking Googlebot, that would be better. For example adding a rule to Defender so she never blocks Googlebot. If not, I think I still want to use Defender and try to keep in mind to check and clear the Defender bans list if I got warning grom Google.

After clearing the Defender Bans, it looks like everything is fine.

Thanks Datahell,

Regards,

Komang