Python 3 Network Crawler Actual Warfare – 8. Installation of App Crawler Related Library: Installation of MitmProxy

Time:2019-10-10

Last article: Python 3 Web Crawler Actual Warfare – 7, Python 3 Web Crawler Actual Warfare – App Crawler Related Library Installation: Charles Installation
Next article: Python 3 Web Crawler Actual Warfare – 9. Installation of App Crawler Related Library: Installation of Appium

MitmProxy is a package grabber that supports HTTP and HTTPS, similar to Fiddler and Charles, but it is a form of console operation.
At the same time, MitmProxy has two related components, one is MitmDump, which is MitmProxy’s command line interface. With it, we can dock Python scripts and use Python to implement the post-listening processing. Another is MitmWeb, which is a Web program through which we can clearly observe the requests captured by MitmProxy.
In this section, we’ll look at the installation of MitmProxy, MitmDump, and MitmWeb.

1. Related links

  • GitHub:https://github.com/mitmproxy/…
  • Official website: https://mitmproxy.org
  • PyPi:https://pypi.python.org/pypi/…
  • Official Document: http://docs.mitmproxy.org
  • MitmDump script: http://docs.mitmproxy.org/en/…
  • Download address: https://github.com/mitmproxy/…
  • DockerHub:https://hub.docker.com/r/mitm…

2. Pip Installation

The simplest way to install is to use Pip, which can be installed by directly executing the following commands:

pip3 install mitmproxy

This is the simplest and most common way to install MitmProxy. After execution, the installation of MitmDump and MitmWeb can be completed. If you do not want to install MitmDump and MitmWeb, you can choose the installation mode listed below, which is specific to each platform or the installation mode of Docker.

3. Installation under Mac

Mac installation is very simple, using HomeBrew, the command is as follows:

brew install mitmproxy

After executing the command, the installation of MitmProxy can be completed.

4. Docker Installation

MitmProxy also supports Docker, whose Docker Hub address is https://hub.docker.com/r/mitm…
The installation command of MitmProxy under Docker is:

docker run --rm -it -p 8080:8080 mitmproxy/mitmproxy mitmdump

This will start MitmProxy and MitmDump on port 8080.
If you want to obtain a CA certificate, you can choose to mount the disk option. The commands are as follows:

docker run --rm -it -v ~/.mitmproxy:/home/mitmproxy/.mitmproxy -p 8080:8080 mitmproxy/mitmproxy mitmdump

This allows CA certificates to be found in the ~/.mitmproxy directory.
In addition, MitmWeb can also be started on port 8081 with the following commands:

docker run --rm -it -p 8080:8080 -p 127.0.0.1:8081:8081 mitmproxy/mitmproxy mitmweb

For more startup options, refer to Docker Hub’s installation instructions.

5. Certificate Configuration

For MitmProxy, if we want to intercept HTTPS requests, we need to set up certificates. MitmProxy will provide a set of CA certificates after installation. As long as the client trusts the certificates provided by MitmProxy, we can obtain the specific content of HTTPS requests through MitmProxy, otherwise MitmProxy can not parse HTTPS requests.
First run the command to generate the CA certificate and start MitmDump.
mitmdump
This will start MitmDump, and then we can find the CA Certificate in the. mitmproxy directory in the user directory, as shown in Figure 1-61:

Python 3 Network Crawler Actual Warfare - 8. Installation of App Crawler Related Library: Installation of MitmProxy

Figure 1-61 Certificate Files
There are five certificates in total. The following is the description of the five certificates:

Python 3 Network Crawler Actual Warfare - 8. Installation of App Crawler Related Library: Installation of MitmProxy

Next, we introduce the certificate configuration process under Mac, iOS and Android platforms.

Mac

Double-click on mitmproxy-ca-cert.pem under Mac to pop up the keystring management page, then find the mitmproxy certificate, click on its settings options, and select Always Trust, as shown in Figure 1-66:

Python 3 Network Crawler Actual Warfare - 8. Installation of App Crawler Related Library: Installation of MitmProxy

Figure 1-66 Certificate Configuration
This completes the configuration of the trust CA certificate under Mac.

iOS

Send the mitmproxy-ca-cert.pem file to the iPhone. It is recommended to send it by email. The attachment can be clicked directly on the iPhone and the installation can be identified. As shown in Figure 1-67:

Python 3 Network Crawler Actual Warfare - 8. Installation of App Crawler Related Library: Installation of MitmProxy

Figure 1-67 Certificate Installation Page
After clicking, it will jump to the page of the installation description file and click the installation button in the upper right corner to install. There will be a warning here, as shown in Figure 1-68:

Python 3 Network Crawler Actual Warfare - 8. Installation of App Crawler Related Library: Installation of MitmProxy

Figure 1-68 Installation Warning Page
Continue to click on the installation in the upper right corner. After successful installation, there will be installation hints, as shown in Figure 1-69:

Python 3 Network Crawler Actual Warfare - 8. Installation of App Crawler Related Library: Installation of MitmProxy

If your iOS version is below 10.3, the process of trusting CA certificates is completed here.
If your iOS version is 10.3 or more, you also need to add full trust to the certificate in Settings – > Universal – > About Local – > Certificate Trust Settings, as shown in Figure 1-70:

Python 3 Network Crawler Actual Warfare - 8. Installation of App Crawler Related Library: Installation of MitmProxy

Figure 1-70 Certificate Trust Settings
Turn on MitmProxy’s full trust switch here.
This completes the process of configuring trust CA certificates on iOS.
Android
Android phones also need to send certificate mitmproxy-ca-cert.pem files to the phone, such as directly copying files.
Next, a prompt window appears when you click on the certificate, as shown in Figure 1-71.

Python 3 Network Crawler Actual Warfare - 8. Installation of App Crawler Related Library: Installation of MitmProxy

Figure 1-71 Certificate Installation Page
Enter the name of the certificate and click OK to complete the installation.

6. Concluding remarks

In this section, we have learned about the installation methods of MitmProxy, MitmDump and MitmWeb, which we will use to capture APP data later.

Last article: Python 3 Web Crawler Actual Warfare – 7, Python 3 Web Crawler Actual Warfare – App Crawler Related Library Installation: Charles Installation
Next article: Python 3 Web Crawler Actual Warfare – 9. Installation of App Crawler Related Library: Installation of Appium

Recommended Today

NET Framework,.net Core and.net Standard

Recently started working on.net Core, and there’s a diagram that shows the relationship.   The diagram above shows that the.net Framework and.net Core implement what is related to the.net Standard, or that the Framework and Core are developed based on the NET Standard. Therefore, we can use the NET Standard project type when building the […]