Mobile data capture of crawler

Time:2020-9-20

Mobile data capture

  • Third party bag grabbing tool

    • Fiddler(window)
    • blue and white porcelain(mac)
    • miteproxy
  • The packet capture tool is the proxy server

    • Intercept and forward requests & responses
  • FiddlerBasic use of

    • install
      • Installed by defaultFiddlerOnly HTTP protocol requests can be caught
    • to configure
      • conductFiddlerNative installation of certificates
        • tools→options→HTTPS→Decrypt HTTPS traffic→Check for certiicate revocation
      • to configureFiddlerLet it capture requests from other devices
        • tools→options→Connections→Allow remote computers to connect
      • takeFiddlerrestart
  • Install it on the mobile phoneFiddlerCertificate for

    • It is necessary to set the fiddler’s machine and mobile phone to a network segment (PC on hotspot, mobile phone link)

    • Visit the IP: Fiddler port (192.168.252.1:9999) of the machine where Fiddler is located in the mobile phone browser to see the following page (a certificate download function is provided)

    • Trust and install the downloaded certificate into the mobile phone

      • Baidu, Android, IOS can be found
    • Configuring agents in mobile phones

      • Baidu, Android, IOS can be found
      • Configure proxy IP: the IP of the machine where Fiddler is located
      • Proxy port: Fiddler’s own port

Through Fiddler to locate the request URL, you can crawl the data requested by the mobile terminal