There seems to be an issue with ASP.NET 2.0 and some search engine spiders. Basically if configured certain way, the IIS will return 301 or 302 code back to the browser instead of 200. There are some posts online that claim it affects the indexing of the site by Google and other search engines.
Couple of solutions exists to the problem:
- Change the property in the web.config to cookieless="UseCookies".
- Add a web browser definition for your site that will identify spider browser correctly. Check the solution at: http://todotnet.com/archive/2006/07/01/7472.aspx
Both of this solutions have problems which i will describe below:
Problems with 1: if customer does not support cookies the site might work correctly since the session state won't be able to carry from one request to another. Such features as shopping cart won't work.
Advantage of this solution is that it is pretty simple to implement.
Problem with 2: you need to constantly update the browser definition since search spiders will eventually change user agent definition.
And finally, why does ASP.NET 2.0 works this way. The 302 code is send back to the browser when it's definition is unknown and framework needs to detect if browser supports cookies. It basically saves the cookie in the response object and then redirects to another page. If cookie can be read it marks the session as the one that can accept cookies.
PS: we changes our site to use the 1st method and will let you know if indexing improves