We have few shell scripts, which access some HTTP URLs using wget as part of some task. wget failed to access some of the URLs with error message like below-
Resolving repo1.maven.org... 188.8.131.52Though all failed URLs were accessible using browsers from same machine / user account. Reason of this failure is that wget does not send any information about itself (agent information) i.e. name and version of the browser etc. and some servers do not entertain requests without agent information. A simple workaround for this problem is to add a dummy agent information as shown below-
Connecting to repo1.maven.org|184.108.40.206|:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
2009-09-22 11:38:53 ERROR 403: Forbidden.
wget -U MyBrowser/1.0 URL_TO_DOWNLOADNow everything will work as expected.