Tuesday, September 22, 2009

403 Forbidden with wget

We have few shell scripts, which access some HTTP URLs using wget as part of some task. wget failed to access some of the URLs with error message like below-

Resolving repo1.maven.org... 38.97.124.18
Connecting to repo1.maven.org|38.97.124.18|:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
2009-09-22 11:38:53 ERROR 403: Forbidden.

Though all failed URLs were accessible using browsers from same machine / user account. Reason of this failure is that wget does not send any information about itself (agent information) i.e. name and version of the browser etc. and some servers do not entertain requests without agent information. A simple workaround for this problem is to add a dummy agent information as shown below-
wget -U MyBrowser/1.0 URL_TO_DOWNLOAD
Now everything will work as expected.

4 Comments:

Unknown said...

I've tried this so many times but I still get a 403/Forbidden error. Any other hints? I've also tried -e robots=off, but to no avail

Gear said...

I too have hit this wall. What is going on guys?

Anonymous said...

It helped me! Without "-U" didn't work. Thanks!

dalilajagow said...

Slots & Casinos 2021 - Mapyro
Check 문경 출장샵 out the top rated slots and casinos in New Jersey. 통영 출장샵 Check out our 양산 출장마사지 list of the best New Jersey online 영주 출장샵 casinos 통영 출장샵 2021. Play slots and try for free!