The server nodejs grabs the jsonp interface data

Time:2022-4-27

as everyone knows,jsonpThe interface returns a segmentjsScripts, using in browsersscriptAfter the tag is imported and loaded successfully, the script method will be executed directly, and the really useful interface data will be returned directly in the image of parameters, so as to achieve the purpose of cross domain request. But how to crawl in the non browser environment (node execution environment) is mainly introduced in this article,The server nodejs grabs the jsonp interface dataIdeas, methods and pits.

Node+Axios Jsonp

preface

Out of curiosity, it has recently been used in researchnodeThe program analyzes the stock data to see if it can find some rules.
But the premise is to obtain a batch of data, so I checked several relevant website platforms through the developer toolsnetworkView, in which the interface data is basically usedjsonpFormatted.

reminder:Do not crawl data too frequently, which may involve network security issues.

Methods and ideas

  1. staynodeServer useaxiosobtainjsonpReturned content
  2. usetypeofView the type of returned content
  3. analysisjsonpReturned content, executioncallbackmethod

The key lies in this third step, innodeHow to dynamically execute this paragraph in the environmentStringCode of type.

Node environment dynamic execution script

  1. eval()

Called directly, using the local scope.

function test() {
  let x = 10, y = 20;
  let sum = eval('x + y')
  console.log(sum);  // 30
}
  1. Function

AndevalThe difference is,FunctionFunctions can only be created inglobal scopeRun in.
test()Error will be reported during execution:ReferenceError: x is not defined

//Error reporting
function test() {
  let x = 10, y = 20;
  let fn = new Function('return x + y')
  let sum = fn()
  console.log(sum) // ReferenceError: x is not defined
}

//Normal execution
global.x = 10
global.y = 20
function test() {
  let fn = new Function('return x + y')
  let sum = fn()
  console.log(sum) // 30
}
  1. vm.runInThisContext(recommended)

vmyesnodeThe core module of,vmhave access tov8ofVirtual Machine contextsThe code is compiled and executed dynamically, and the execution context of the code is isolated from the current process.
The executed code cannot obtain the local scope, and can only be executed in the current scopeglobalObjectcode

const vm = require('vm')

global.x = 10
global.y = 20
function test() {
  let sum = vm.runInThisContext('x + y')
  console.log(sum) // 30
}

Complete code implementation

// index.js

const vm = require('vm')
const axios = require('axios')

//Mount the corresponding callback method on the global object
global.jQuery11230971606670044967_1649312313646 = function(res) {
  //Data returned by jsonp interface
  console.log(res)
  // do something
}

async function getData() {
  let { data } = await axios({
    url: 'https://push2.eastmoney.com/api/qt/clist/get?cb=jQuery11230971606670044967_1649312313646&fid=f62&po=1&pz=10&pn=1&np=1&fltt=2&invt=2&fs=m%3A90+t%3A3&stat=1&fields=f12%2Cf14%2Cf2%2Cf3%2Cf62%2Cf184%2Cf66%2Cf69%2Cf72%2Cf75%2Cf78%2Cf81%2Cf84%2Cf87%2Cf204%2Cf205%2Cf124&ut=b2884a393a59ad64002292a3e90d46a5'
  })
  
  //View the type of returned content
  console.log(typeof data)
  
  //Execute callback
  vm.runInThisContext(data)
}

//Call the method to grab data
getData()

Special reminder:
Browser environment andNodeThe execution environment is different from mounting global objects.
In the browser environment, global scopes can be usedvarTo declare a variable or method, which will be automatically mounted towindowOn the object;
But inNodeIn the environment, each file or module will be enclosed in a separate scope. Therefore, no matter where the variables declared are in the local scope relative to the current module or file, they will not be automatically mounted to the global scopeglobalObject, if you want to use it, you must mount it manually.

Welcome to:Tianwen blog