Tag：Reptile
-
Time：2019-11-17
What is an agent? When will agents be usedIP？ Proxy server（Proxy Server）Its function is to obtain network information on behalf of users and then return it to users. The image says: it is the network information transit station. Through agencyIPVisit the target station to hide the user’s realityIP。 For example, if you want to […]
-
Time：2019-11-2
Preface How are so many modules of scrapy combined? It is clear that the XPath helper plug-in on chrome has written the XPath. Why does it read none when it comes to the program? Can scratch write multi-level response directly? Do you have to use the requests library again?? It’s OK. This article is a […]
-
Time：2019-10-25
Needs encountered Some time ago, we need to quickly make a static display page, which requires responsive and beautiful. Due to the short time, it’s a bit troublesome to write by yourself, so I plan to go online to find the ready-made one. I found a few pages in the middle and found that it […]
-
Time：2019-10-12
On the afternoon of September 6, several insiders said that Hangzhou Magic Scorpion Data Technology Co., Ltd., a well-known big data service company in Hangzhou, was suspected to be under the control of relevant law enforcement officers. One of the core executives surnamed Zhou was taken away by the police. Above is a news spread […]
-
Time：2019-10-11
Reptiles will always be sealed up after they have been used for a long time. — Lu Xun Some websites, especially some old ones, have not done anti-crawler mechanism. We can climb freely and happily and put their underwear on it. The data all crawled down. For emotional reasons at most, we crawl slowly […]
-
Time：2019-10-6
Before we said that a common method of anti crawler is to detect IP and limit access frequency. So we need to bypass this limitation by setting up proxy ip. There are many websites that offer free proxy ip, such as https://www.xicidaili.com/nt/. We can get a lot of proxy IP from the website. But not […]
-
Time：2019-10-5
Introduction to Python Python is a high-level scripting language that combines interpretability, compilation, interaction and object-oriented. Python’s design is very readable. Compared with other languages which often use English keywords, some punctuation symbols in other languages have more distinctive grammatical structures than other languages. Python is an interpretative language: that means there is no compilation […]
-
Time：2019-9-28
The learning textbook is “Python Network Data Acquisition”, most of the code came to this book. Network crawler crawl data first is to have the right to crawl, no matter how good the code can not run without the right to crawl. So first of all, we should disguise our own crawler, so that the […]
-
Time：2019-9-9
The purpose of learning Python for oneself is relatively clear – Crawler and quantification. At present, some resources have been found for learning, first for quantitative learning, and the crawler lags behind. The current goal is to “master as much quantitative capacity as possible in 180 days”! In the future, regularly send their own learning […]
-
Time：2019-8-25
Scrapy middleware is divided into downloader middleware and crawler Middleware Downloader middlewares: Mainly deals with callbacks sent out by request requests and returned by response responses. Method: process_request(self,request,spider)： Called when request request passes through downloader Middleware Return to None: Continue the request Return to the Request object, hand the request object to the scheduler […]
-
Time：2019-8-13
The night before last, a friend of Xiaobian’s analyst whispered that the website of the US Treasury Department had been finalized at last. What route is this? Are you in the current situation?Be brave enough to destroy? Xiaobian’s face was frightened. Don’t be afraid. I wrote a crawler with Python and crawled down the data […]
-
Time：2019-8-12
I. What is a reptile? A crawler, also known as a “web crawler”, is a program that automatically accesses the Internet and downloads the content of a website. It is also the basis of search engines, such as Baidu and GOOGLE, which rely on powerful web crawlers to retrieve huge amounts of Internet information and […]
