wget or curl
. In order to prevent crawlers from consuming the server resources, the server selectively screened the request headers according to your request. Therefore, the Agent user-agent of wget and curl needs to be modified for camouflage.I. Modify user-Agent of WGET
1. Temporarily change THE UA of WGET
Before wget, add the parameter
-u
, which means to set User Agent
wget www.google.com -U "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36"
What is User Agent and how to get it, please refer to the following blog. Of course, you can also use the above one directly.
What is the UserAgent and how to view the UserAgent using the browser
2. Permanently change THE UA of Wget
Modify the configuration file /etc/wgetrc to add the following line:
header = User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36
The above configuration will take effect for all users. If you only want the current user to take effect, you can add the above line to ~/.wgetrc. If this file is not available, you can create it manually.
2. Modify the user-Agent of curl
1. Temporary change to Curl’s UA
Use the following parameters:
curl https://www.google.com --user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36"
2. Permanently change Curl’s UA
Modify the profile ~/.curlrc to add the following line:
–user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36
Reference:
1, https://www.linpx.com/p/on-an-interesting-play-wget-use.html
2, https://chaifeng.com/_curl_wget_user-agent/
Read More:
- RCurl error-fatal error: curl/curl.h: No such file or directory
- [Solved] Centos7 Wget Error: nown. wget: unable to resolve host address ‘mirrors.aliyun.com’
- Curl returns empty reply from server. Due to the processing of special characters, curl cannot be accessed and the browser can access it.
- 8、Forbidden (CSRF token missing or incorrect.): /register/
- Urllib2.httperror: http error 403: forbidden solution
- MacOS WGet ‘command not found’ solution
- File under WGet download directory
- wget OpenSSL: error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version
- urllib2.HTTPError: HTTP Error 403: Forbidden
- Shell curl command reported an error: (60) SSL certificate problem: self signed certificate
- cURL error 60: SSL certificate problem…
- Nextcloud error: “access forbidden CSRF check failed”
- wget Error: ERROR: cannot verify nih.at’s certificate, issued by “/C=US/O=Let‘s Encrypt/CN=R3”
- Using curl to generate gnutls_ Handshake() failed: error in the pull function or gnutls recv error: error in the pull function
- GitHub push ErrorThe requested URL returned error: 403 Forbidden while accessing
- On curl: (2) failed initialization
- Forbidden (403) CSRF verification failed. Request aborted. – Django
- 【wget failed】Connecting to 127.0.0.1:7890… failed: Connection refused.
- Solve GitHub push error 403 Forbidden while accessing
- Copy forbidden in C + +++