Tuesday, September 22, 2009

403 Forbidden with wget

We have few shell scripts, which access some HTTP URLs using wget as part of some task. wget failed to access some of the URLs with error message like below-

Resolving repo1.maven.org...
Connecting to repo1.maven.org||:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
2009-09-22 11:38:53 ERROR 403: Forbidden.

Though all failed URLs were accessible using browsers from same machine / user account. Reason of this failure is that wget does not send any information about itself (agent information) i.e. name and version of the browser etc. and some servers do not entertain requests without agent information. A simple workaround for this problem is to add a dummy agent information as shown below-
wget -U MyBrowser/1.0 URL_TO_DOWNLOAD
Now everything will work as expected.