Soft 404s - google webmaster

[kyomar]

Hi, I have quite many soft 404 errors. After moving to elxis4 I did 301 redirects for most old urls but not all.

1) Is it possible that although the 404 page shows up there is no actual 404 return from the server. Could it be that the return is a 200 and google indexes these pages again.

2) I just realized that I am getting a fatal error message (Invalid request. Requested language does not exist!) in elxis if a language code is in the url like: website/ja/our-rates.html. I believe this is the problem for the soft 404s.

Thank you for your help!

[datahell]

Elxis 4.x is not translated into japanese (ja) so all /ja/* URLs will return error 404.
Elxis is not wrong, if it shows you 404 it means that the URL does not exist.

[kyomar]

Thanks for your quick response datahell - very appreciated.

A 404 message and code would be okay, but it shows a "fatal error" message. I did a Fetch as Google and the return code is 200 okay, which in this case is not okay us the site doesn't exist. If there is no translation for this language (ja/) then there is obviously no page us well and therefore a 404 code would be appropriate otherwise google indexes the page again.

[datahell]

The "fatal error" message is something else. What does this message says? Elxis may generate fatal error message if it can't login in the database, if it can't find repository and in some other crusial cases.
Have you thought that your redirects might generate the problem?
Provide us the site's URL and/or the error messages/codes and in general more information if you want a full answer.

[kyomar]

Us mentioned in the first post the fatal error message is:
Unrecoverable error!
Invalid request. Requested language does not exist!

Link ja/ (no redirect)
fatal error: link removed

Same link with en/ (no redirect)
404 not found: link removed -> okay

Link ja/ (with redirect)
301 redirect: link removed -> okay

Thank you!

[datahell]

This is a low level alert from the Elxis URI library. The page does not exist in any language and as the language "ja" does not exist the redirection is wrong and the system generates fatal error. Make the re-direction to an existing page not to a non-existing one.


http://www.koyuki-hakuba.com/en/our-rates.html  -> Error 404 (correct, the page does not exist)
http://www.koyuki-hakuba.com/our-rates.html  -> Error 404 (correct, the page does not exist)
http://www.koyuki-hakuba.com/ja/our-rates.html  -> Fatal error (correct, the language is not available)
http://www.koyuki-hakuba.com/ja/anything  -> Fatal error (correct, the language is not available)

http://www.koyuki-hakuba.com/ja/room-rates.html - >301 redirection by you (works due to the redirection you have added, if you hadn't done so you would get a fatal error too)

You can add a generic redirection for any "ja/*" URL to a non-existing "en" URL in order to get a 404 page.

Code: [Select]

RewriteRule ^ja/(.*$) /not-found.html [R=301,L]or to the corresponding page in English language:

Code: [Select]

RewriteRule ^ja/(.*)$ /$1 [R=301,L]
Why Elxis generates a Fatal error page instead of a 404 one for the missing language?
Elxis exit pages are divided in 2 categories, those the system can load and those it can't. The error 404  page is being displayed when the system has successfully get loaded but the requested page does not exist. The fatal error page is being displayed when the system cannot loaded as something in user's request (or in site's settings/environment) is wrong. In your case the wrong thing is the language. In order for the 404 page to be generated Elxis need to load the language but the requested language "ja" does not exist so the system can't proceed further and generates a fatal error message. If you don't want to get this fatal error translate Elxis into japanese and publish that language. After that you will get error 404 message instead of fatal error.

[kyomar]

Thank you very much datahell for your help - awesome support!

The RewriteRule ^ja/(.*$) /not-found.html [R=301,L] didn't work, still fatal error. So created ja language file and changed settings in ja.php. I also changed rtl to ltr as this is much more common in Japan. I haven't translated anything yet due to time issues but will start soon.

One question though. I had to activate (tick) ja in admin, settings because if not activated no 404 message will be triggered, instead the site goes to the homepage (like redirect). The site is not multilingual (yet) and I am worried that search engines will index ja/ now as well and see it as duplicate content. I would be grateful if you could comment on this.

Thanks a lot!

[kyomar]

Sorry I have to restart this thread again as I am still getting more and more Soft 404 crawl errors and this is really bad for search ranking.

Elxis is now showing the correct 404 not found message but the server status code is HTTP/1.1 200 OK (fetch us google) which produces the Soft 404 crawl errors in google instead of 404 not found. How can I make sure that the correct status code is being sent.

Thank you!

[datahell]

If you want to send 404 headers open templates/system/404.php (or create your own in your template folder) and add this line before echoing anything:

header('HTTP/1.0 404 Not Found');

That's all.

.....
header('HTTP/1.0 404 Not Found');
echo $page->doctype."\n";
?>
<html<?php echo $page->htmlattributes; ?>>
<head>
....

[kyomar]

Awesome, thank you very much datahell. Hope this will prevent the soft 404 crawl errors in google.

[kyomar]

Sorry, this drives me nuts  In Fetch as Google still:
HTTP/1.1 200 OK
Date: Fri, 09 Aug 2013 07:33:17 GMT

I have insert the code as explained but still status code 200 OK. What's going on here? I really need to get rid of this stupid soft 404 crawl errors.

Perhaps it's too hot/himid here and my brain is not working properly.

Thanks!

[datahell]

You have done some strange things setting some custom redirections, I believe you better remove them.
Elxis will generate a 404 page if the page does not exist. Remove anything custom you have added.
Elxis will tell google not to index that pages (on 404 page sets the meta tag robots to noindex, follow). That's all you need.
You can also install the Sitemap component and re-submit it to google.
Note, that you might have to wait some time inorder google to refresh your listings (especially if you have set "moved permanent" some pages).