Often, a network administrator will need to block a troublesome user agent that’s causing load or security issues. If you are using junOS firewalls with an IDP subscription, you can craft a custom attack rule that blocks these users at the firewall itself.
Let’s craft a custom attack policy to block Baidu, a common troublesome web spider that doesn’t obey robots.txt rules.
Let’s begin by connecting to your border firewall using CLI. Enter the configuration mode and issue these commands:
set security idp custom-attack Baidu-Spider recommended-action close-client set security idp custom-attack Baidu-Spider severity minor set security idp custom-attack Baidu-Spider attack-type signature context http-header-user-agent set security idp custom-attack Baidu-Spider attack-type signature pattern .*Baidu.* set security idp custom-attack Baidu-Spider attack-type signature direction client-to-server
Next, add your newly crafted custom attack rule to your existing IDP policy that polices your incoming HTTP traffic.
You can modify the signaturepattern using regular expressions to block other user agents as well. If you are comfortable with junOS’s command line interface, this is an easy way to protect all of your webservers from this user agent. Your millage may vary, and you should follow Best Practices when making changes to your firewall configuration.
