How to use proxy IP for curl and WGet in Linux shell

Time:2020-1-21

Preface

As you all know, two very practical commands are provided in Linux shell to crawl the web page. They are curl and WGet. This article will give you a detailed introduction about the use of proxy IP by curl and WGet in Linux shell, and share them for your reference and study. Let’s have a look.

Curl and WGet use agents

  • Curl supports HTTP, HTTPS, Socks4, Socks5
  • WGet supports HTTP and HTTPS

Agent example:

#!/bin/bash
#
#Curl supports HTTP, HTTPS, Socks4, Socks5
#WGet supports HTTP and HTTPS
#
#Example of MIPO agent:
# http://proxy.mimvp.com/demo2.php
#
#MIPO agent purchase:
# http://proxy.mimvp.com
#
# mimvp.com
# 2015-11-09
 
 
#HTTP proxy format http_proxy = http: // IP: Port
#HTTPS proxy format https_proxy = http: // IP: Port
 
{'http': 'http://120.77.176.179:8888'}
curl -m 30 --retry 3 -x http://120.77.176.179:8888 http://proxy.mimvp.com/exist.php     # http_proxy
wget -T 30 --tries 3 -e "http_proxy=http://120.77.176.179:8888" http://proxy.mimvp.com/exist.php   # http_proxy
 
{'https': 'http://46.105.214.133:3128'}
curl -m 30 --retry 3 --proxy-insecure -x http://46.105.214.133:3128 -k https://proxy.mimvp.com/exist.php     # https_proxy
wget -T 30 --tries 3 --no-check-certificate -e "https_proxy=http://46.105.214.133:3128" https://proxy.mimvp.com/exist.php # https_proxy
 
  
#Curl supports socks
{'socks4': '101.255.17.145:1080'}
curl -m 30 --retry 3 --socks4 101.255.17.145:1080 http://proxy.mimvp.com/exist.php
  
{'socks5': '82.164.233.227:45454'}
curl -m 30 --retry 3 --socks5 82.164.233.227:45454 http://proxy.mimvp.com/exist.php
 
 
#WGet does not support socks

WGet profile settings proxy


vim ~/.wgetrc
 
http_proxy=http://120.77.176.179:8888:8080
https_proxy=http://12.7.17.17:8888:8080
use_proxy = on
wait = 30
 
wget -T 30 --tries 3 http://proxy.mimvp.com

Shell set temporary local agent

# proxy no auth
export http_proxy=http://120.77.176.179:8888:8080
export https_proxy=http://12.7.17.17:8888:8080
 
# proxy auth
export http_proxy=http://username:[email protected]:8888:8080
export https_proxy=http://username:[email protected]:8888:8080
 
 
#Unset
unset http_proxy
unset https_proxy

Shell set system global agent

#Modify / etc / profile, save and restart the server
Sudo VIM / etc / profile ා valid for everyone
or
Sudo VIM ~ /. Bashrc ා valid for all
or
VIM ~ /. Bash? Profile? Personal effective
  
  
# proxy no auth
export http_proxy=http://120.77.176.179:8888:8080
export https_proxy=http://12.7.17.17:8888:8080
 
# proxy auth
export http_proxy=http://username:[email protected]:8888:8080
export https_proxy=http://username:[email protected]:8888:8080
 
source /etc/profile
or
source ~/.bashrc
or
source ~/.bash_profile
 
 
sudo reboot

MIPO agent example

MIPO agent example, including python, Java, PHP, C ා, go, Perl, ruby, shell, nodejs, phantomjs, groovy, Delphi, Yi language and more than ten programming languages or scripts. Through a large number of running examples, the correct method of using agent IP is explained in detail, which is convenient for web page crawling, data collection, automatic testing and other fields.

MIPO proxy example, the proxy IP used in the test, all from MIPO proxy, covering more than 120 countries, 34 provinces and cities in China, supporting HTTP, HTTPS, Socks4, Socks5, etc

Official website of Mipu proxy: http://proxy.mimvp.com/demo2.php

summary

The above is the whole content of this article. I hope that the content of this article can bring some help to your study or work. If you have any questions, you can leave a message and communicate with us. Thank you for your support for developpaer.

Recommended Today

RCAST 35: add type to currency

– font ALT: Simsun; MSO font charset: 134; mso-generic-font-family:auto; mso-font-pitch:variable; mso-font-signature:3 680460288 22 0 262145 0;} @font-face {font-family:”Cambria Math”; panose-1:2 4 5 3 5 4 6 3 2 4; mso-font-charset:1; mso-generic-font-family:roman; mso-font-format:other; mso-font-pitch:variable; mso-font-signature:0 0 0 0 0 0;} @font-face {font-family:Calibri; Variable; Ose-1: 216301111; mso-font-charset:134; mso-generic-font-family:auto; mso-font-pitch:variable; mso-font-signature:3 680460288 22 0 262145 0;} /\* Style […]