13、 Mitmproxy mock actual combat

Time:2022-5-23

catalogue

  • Introduction to mitmproxy
  • Mitmproxy tool
  • Mitmdump actual combat

Introduction to mitmproxy

  • As the name suggests, mitmproxy is the proxy for mitm, which is man in the middle attack ­ in ­ the ­ middle attack)。 The agent used for man in the middle attack will first forward the request to the normal agent to ensure the communication between the server and the client. Secondly, it will timely check and record the intercepted data or tamper with the data, causing specific behavior of the server or client.

  • Unlike Fiddler or Wireshark and other packet capture tools, mitmproxy can not only intercept requests to help developers view and analyze, but also carry out secondary development through custom scripts. For example, Fiddler can filter the browser’s request for a specific URL, view and analyze its data, but it can not meet the requirements of high customization, which is similar to: “intercept the browser’s request for the URL, empty the returned content, save the real returned content to a database, and send an email notification in case of exception”. For mitmproxy, this requirement can be easily realized by loading custom Python scripts.

  • However, mitmproxy does not really launch man in the middle attack on innocent people. Because mitmproxy works in the HTTP layer, and the current popularity of HTTPS makes the client have the ability to detect and avoid man in the middle attack. Therefore, in order for mitmproxy to work normally, the client (app or browser) must actively trust mitmproxy’s SSL certificate or ignore certificate exceptions, This means that the app or browser belongs to the developer himself – obviously, this is not doing black production, but developing or testing.

  • What is the practical significance of such a tool? As far as I know, at present, it is widely used to do simulation crawler, that is, use mobile phone simulator and headless browser to crawl the data of app or website. As an agent, mitmpproxy can intercept and store the data obtained by crawler, or modify the data to adjust the behavior of crawler.

  • In fact, the above is only the case that mitmproxy works in the forward proxy mode. By adjusting the configuration, mitmproxy can also be used as transparent proxy, reverse proxy, upstream proxy, socks proxy, etc., but these working modes seem to be not commonly used for mitmproxy, so this paper only discusses the forward proxy mode.

  • reference material:

    13、 Mitmproxy mock actual combat

    image.png

Mock: during the test process, virtual objects are used to replace objects that are not easy to construct and obtain, so as to achieve the same effect. The virtual object is called mock.

1. Installation

  • Mac installation:brew install mitmproxy

  • Install using Python:Python must be at least 3.6

    • In windows, run CMD or power shell command as administrator:
      1. pip install ­­pipx
      2. python3 ­m pipx ensurepath
      3. pipx install mitmproxy
  • Windows installation:

    1. Go to the official website to download an EXE, and then install the next step
    2. Exe installation directly helps you configure environment variables
    3. You can run the mitm web startup proxy service directly
    4. You can run mitmdump directly
    • Environment variables need to be configured after installation: the prompt of successful installation will give the directory to be configured. Configure it to the path environment variable and restart CMD

      13、 Mitmproxy mock actual combat

      image.png

2. Environmental verification

  • mitmdump --version

    13、 Mitmproxy mock actual combat

    image.png

Mitmproxy tool

  • Mitmproxy – > command line tool (not supported by windows)

  • Mitmdump – > Python script can be loaded

  • Mitmweb – > Web interface tool

  • Common parameters:

    • -hEnter the help interface
    • -pSpecify the port number, which is 8080 by default
    • -sLoad Python script

1. mitmproxy

  • implementmitimproxyEnter the interface

  • Use the mouse to scroll up and down, and click the mouse to select the request for analysis

  • Exit the interface: Pressqkey

  • Capture the package after installing the agent and certificate

    13、 Mitmproxy mock actual combat

    image.png

    13、 Mitmproxy mock actual combat

    image.png

2. Set up agent

  • Method 1:

    • Browser switchomega adds a new mitmproxy proxy

      13、 Mitmproxy mock actual combat

      image.png
  • Method 2:

    • Set system agent:
      • mac:

        13、 Mitmproxy mock actual combat

        image.png
      • Windows:

        13、 Mitmproxy mock actual combat

        image.png

3. Installation certificate

4. Mitmweb is far from Charles. Just understand it

  • Start: inputmitmweb -p 8999, automatically jump to the page
    13、 Mitmproxy mock actual combat

    image.png

    13、 Mitmproxy mock actual combat

    image.png

5. mitmdump

  • Default listening 8080
    • mitmdump
  • -pParameter modification listening port
  • -S Python fileLoad Python script

6. Differences between the three tools

  • To start mitmproxy, you can use any of the three commands: mitmproxy, mitmdump and mitmweb. The functions of these three commands are the same, and you can load custom scripts. The only difference is the difference in the interactive interface.
  • After the mitmproxy command is started, a command line interface will be provided. Users can see the requests in real time, filter the requests through the command and view the request data.
  • After the mitmweb command is started, a web interface will be provided. Users can see the requests in real time, filter the requests and view the request data through GUI interaction
  • After the mitmdump command is started – you should have guessed that there is no interface and the program runs silently. Therefore, mitmdump cannot provide the function of filtering requests and viewing data. It can only work silently in combination with custom scripts.
  • Because the interactive operation of mitmproxy command is slightly complicated and does not support Windows system, and our main use method is to load custom scripts without interaction, in principle, we only need mitmdump.

Mitmdump actual combat

1. Recording and playback

  • Recording:mitmdump -w tmp-wwrite filetmp
    13、 Mitmproxy mock actual combat

    image.png
  • Filtering:mitmdump -nr tmp -w tmp2 "~s Insane"~sFilter only the content in the response,~qFilter only the content in the request;-nMeans that the agent is not started and only the files are filtered,-rreadtmpContents in the document,-wwrite filetmp2

    13、 Mitmproxy mock actual combat

    image.png

    13、 Mitmproxy mock actual combat

    image.png
  • Playback:mitmdump -nC tmp2-nAgent value does not start,-CPlayback interface request

    13、 Mitmproxy mock actual combat

    image.png

Implement maplocal and modify the request and response data without the server

  1. Modify Python script of header
from mitmproxy import http

def request(flow: http.HTTPFlow):
    flow.request.headers["myheader"] = "feier"
    print(flow.request.headers)
  • Execute in terminalmitmdump -p 8999 -s D:\Programs\DevOps\Python_Practice\Exercises\mitm\request_demo.py
    13、 Mitmproxy mock actual combat

    image.png
  1. Python script for modifying request and response information
from mitmproxy import http

def request(flow: http.HTTPFlow) -> None:
    #Initiate a request to determine whether the URL is the expected value
    if flow.request.pretty_url == "https://www.baidu.com/":
        #Create a response 
        flow.response = http.HTTPResponse.make(
            200,  # (optional) status code
            b"Hello World",  # (optional) content
            {"Content-Type": "text/html"}  # (optional) headers
        )
  • Execute in terminalmitmdump -p 8999 -s D:\Programs\DevOps\Python_Practice\Exercises\mitm\maplocal_baidu.py --ssl-insecure

among--ssl-insecureTo prevent<< Cannot establish TLS with 10.66.253.120:443 (sni: None): TlsException('Cannot validate certificate hostname without SNI')This error will cause the status code to report 502.

13、 Mitmproxy mock actual combat

image.png
  1. Modify snowball response information
  • Python file
from mitmproxy import http

def request(flow: http.HTTPFlow) -> None:
    #Initiate a request to determine whether the URL is the expected value
    if "quote.json" in flow.request.pretty_url:
        #Open a file saved locally
        with open("path/quote.json") as f:
        #Create a response
            flow.response = http.HTTPResponse.make(
                200,  # (optional) status code
                f.read(),  # (optional) content
                {"Content-Type": "application/json"}  # (optional) headers
            )
  • Execute script in terminalmitmdump -p 8999 -s D:\Programs\DevOps\Python_Practice\Exercises\mitm\maplocal_xueqiu.py
  • usecurl http://ww.baidu.com/xxx/queto.json -x 127.0.0.1:8999, simulate a request to see if the return is queto JSON content
    13、 Mitmproxy mock actual combat

    image.png
  • Refresh the page to view the response information

    13、 Mitmproxy mock actual combat

    image.png
  • Modify queto.com locally JSON file, and then refresh the page to realize maplocal

    13、 Mitmproxy mock actual combat

    image.png

Implement rewrite and modify the response data obtained from the server

from pprint import pprint
from mitmproxy import http


def response(flow: http.HTTPFlow):
    pprint(flow.response)
  • Execute script in terminal

    13、 Mitmproxy mock actual combat

    image.png
  1. Modify “China Ping An” in snowball as the target value, and the dictionary structure is as follows

    13、 Mitmproxy mock actual combat

    image.png
  2. Python files:

from mitmproxy import http
import json

def response(flow: http.HTTPFlow):
    #Add filter conditions
    if "quote.json" in flow.request.pretty_url and "x=" in flow.request.pretty_url:
        #Convert the response content to dictionary format
        data = json.loads(flow.response.content)
        #Modify the value of the corresponding field
        data["data"]["items"][0]["queto"]["name"] = "rewrite_hogwarts"
        #The modified data is converted into a string and assigned to the original data
        flow.response.text = json.dumps(data)
  1. Execute script in terminalmitmdump -p 8999 -s D:\Programs\DevOps\Python_Practice\Exercises\mitm\rewrite_xueqiu.py

  2. Refresh the page to view the results

    13、 Mitmproxy mock actual combat

    image.png

When assigning the modified data to the original data, you must useflow.response.textBecausetextIs the binary stream format, otherwise the script will report the following error:Message content must be bytes, not string.

13、 Mitmproxy mock actual combat

image.png

Recommended Today

Storage and use of user tag data in Clickhouse

Basic ideas Offline part: importing offline data of HDFS into Clickhouse Real time part: write directly to Clickhouse and useAggregatingMergeTreeThe table engine merges the latest data by sort key Queries: UsingfinalKeyword to query only the latest merged data Timed executionoptimize, press sort key to merge the whole table data Data distribution: split the conditions submitted […]