dantor.com - since 1998
[ Log In ]
dantor.com - Est. 1998

Web.config requestFiltering
The requestFiltering section of a web.config can be used to block specific user-agent's from visiting your web site. You can use the requestFiltering section of web.config to block specific browsers, crawlers, robots or other software.

While some web.config sections require that the containing directory is set as an application, this isn't one of them. A simple web.config with a requestFiltering section may be placed in any directory, and the directory does NOT need to be set as an application.

User-agents - what are they?
Each time someone visits your site, their browser software identifies itself by sending a user-agent string. The user-agent string identifies the browser software and the browser version. The user-agent string sometimes includes information on the operating system type, name, and version, as well as information about installed plug-ins.

To help identify their crawlers, search engines (google, bing, yahoo, yandex, baidu, etc) also send user-agent strings when their software crawls your web site.

Example browser user agents
Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:9.0.1)+Gecko/20100101+Firefox/9.0.1 (firefox 9, on Windows 7)
Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.1;) (internet explorer 8, on Windows xp)
Opera/9.80 (Windows NT 6.1; U; es-ES) Presto/2.9.181 Version/12.00 (opera 12, on Windows 7)
Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en) (apple iphone)

Example search engine user agents
Mozilla/5.0+(compatible;+bingbot/2.0;++http://www.bing.com/bingbot.htm) (Bing)
Mozilla/5.0+(compatible;+Googlebot/2.1;++http://www.google.com/bot.html) (Google)
Mozilla/5.0+(compatible;+YandexBot/3.0;++http://yandex.com/bots) (Yandex)

Using web.config and requestFiltering
  • Use a text editor to create a file named web.config
  • Save the web.config file with the appropriate content
  • Place the web.config file in the directory that you wish to protect

How it's done

Example web.config, with requestFiltering. Comments are enclosed in <!-- --> and are not required.
<requestFiltering>
  <filteringRules>
    <!-- name the rule -->
    <filteringRule name="user agent deny" scanUrl="false" scanQueryString="false">
      <scanHeaders>
        <!-- apply rule to user-agent header -->
        <add requestHeader="user-agent" />
      </scanHeaders>
      <appliesTo>
        <clear />
        <!-- only apply rule to php files -->
        <add fileExtension=".php" />
      </appliesTo>
      <denyStrings>
        <clear />
        <!-- block the yandex bot -->
        <add string="yandex" />
      </denyStrings>
    </filteringRule>
  </filteringRules>
</requestFiltering>


Detailed web.config content
  • If there isn't an existing web.config in the directory, your new web.config should look something like this
    <?xml version="1.0"?>
    <configuration>
       <system.webServer>
          <security>
            <requestFiltering>
              <filteringRules>
                <!-- name the rule -->
                <filteringRule name="user agent deny" scanUrl="false" scanQueryString="false">
                  <scanHeaders>
                    <!-- apply rule to user-agent header -->
                    <add requestHeader="user-agent" />
                  </scanHeaders>
                  <appliesTo>
                    <clear />
                    <!-- only apply rule to php files -->
                    <add fileExtension=".php" />
                  </appliesTo>
                  <denyStrings>
                    <clear />
                    <!-- block the yandex bot -->
                    <add string="yandex" />
                  </denyStrings>
                </filteringRule>
              </filteringRules>
            </requestFiltering>
         </security>
       </system.webServer>
    </configuration>
    
  • If there is an existing web config, without a <system.webServer> section... Your new web.config should look like this
    <?xml version="1.0"?>
    <configuration>
       <system.web>
         .. existing text ..
         .. existing text ..
       </system.web>
       <system.webServer>
          <security>
            <requestFiltering>
              <filteringRules>
                <!-- name the rule -->
                <filteringRule name="user agent deny" scanUrl="false" scanQueryString="false">
                  <scanHeaders>
                    <!-- apply rule to user-agent header -->
                    <add requestHeader="user-agent" />
                  </scanHeaders>
                  <appliesTo>
                    <clear />
                    <!-- only apply rule to php files -->
                    <add fileExtension=".php" />
                  </appliesTo>
                  <denyStrings>
                    <clear />
                    <!-- block the yandex bot -->
                    <add string="yandex" />
                  </denyStrings>
                </filteringRule>
              </filteringRules>
            </requestFiltering>
         </security>
       </system.webServer>
    </configuration>
    

  • If your existing web.config already has a <system.webServer> section, just add the <security> and <requestFiltering> sections
    <?xml version="1.0"?>
    <configuration>
       <system.web>
         .. existing text ..
         .. existing text ..
       </system.web>
       <system.webServer>
          <security>
            <requestFiltering>
              <filteringRules>
                <!-- name the rule -->
                <filteringRule name="user agent deny" scanUrl="false" scanQueryString="false">
                  <scanHeaders>
                    <!-- apply rule to user-agent header -->
                    <add requestHeader="user-agent" />
                  </scanHeaders>
                  <appliesTo>
                    <clear />
                    <!-- only apply rule to php files -->
                    <add fileExtension=".php" />
                  </appliesTo>
                  <denyStrings>
                    <clear />
                    <!-- block the yandex bot -->
                    <add string="yandex" />
                  </denyStrings>
                </filteringRule>
              </filteringRules>
            </requestFiltering>
         </security>
          <modules runAllManagedModulesForAllRequests="true"/>
       </system.webServer>
    </configuration>

 

Miscellaneous support articles
Using web.config to return custom 404 and 500 error messages on IIS 7 and IIS 7.5
Using web.config to redirect browsers with 301, 302, and 307 status codes on IIS 7 and IIS 7.5
Using web.config to set a default document for your website
Using web.config to allow or block specific IP addresses
Using web.config to block specific user-agent browser strings