{"id":1732,"date":"2025-04-28T11:18:05","date_gmt":"2025-04-28T02:18:05","guid":{"rendered":"https:\/\/www.yilus5.com\/blog\/?p=1732"},"modified":"2025-04-28T14:06:55","modified_gmt":"2025-04-28T05:06:55","slug":"scrapy%e4%bb%a3%e7%90%86ip%e6%b1%a0%e6%90%ad%e5%bb%ba%e6%95%99%e7%a8%8b%ef%bc%9a%e9%ab%98%e6%95%88%e7%a8%b3%e5%ae%9a%e7%88%ac%e5%8f%96%e6%95%b0%e6%8d%ae%e7%9a%84%e5%85%b3%e9%94%ae%e4%b8%80%e6%ad%a5","status":"publish","type":"post","link":"https:\/\/www.yilus5.com\/blog\/1732.html","title":{"rendered":"Scrapy\u4ee3\u7406IP\u6c60\u642d\u5efa\u6559\u7a0b\uff1a\u9ad8\u6548\u7a33\u5b9a\u722c\u53d6\u6570\u636e\u7684\u5173\u952e\u4e00\u6b65"},"content":{"rendered":"\n<p>\u5728\u7f51\u7edc\u4fe1\u606f\u7206\u70b8\u7684\u4eca\u5929\uff0c\u6570\u636e\u5df2\u7ecf\u6210\u4e3a\u9a71\u52a8\u4e1a\u52a1\u589e\u957f\u7684\u6838\u5fc3\u52a8\u529b\u3002\u5bf9\u4e8e\u9700\u8981\u5927\u89c4\u6a21\u3001\u81ea\u52a8\u5316\u6293\u53d6\u7f51\u9875\u6570\u636e\u7684\u5f00\u53d1\u8005\u548c\u4f01\u4e1a\u800c\u8a00\uff0cScrapy\u65e0\u7591\u662f\u4e00\u6b3e\u5f3a\u5927\u4e14\u7075\u6d3b\u7684Python\u722c\u866b\u6846\u67b6\u3002\u7136\u800c\uff0c\u5728\u5b9e\u9645\u722c\u53d6\u8fc7\u7a0b\u4e2d\uff0c\u6211\u4eec\u5e38\u5e38\u4f1a\u9047\u5230\u7f51\u7ad9\u7684\u53cd\u722c\u866b\u673a\u5236\uff0c\u5176\u4e2d\u6700\u5e38\u89c1\u7684\u5c31\u662fIP\u5c01\u9501\u3002\u4e3a\u4e86\u5e94\u5bf9\u8fd9\u4e00\u6311\u6218\uff0c\u6784\u5efa\u4e00\u4e2a\u7a33\u5b9a\u9ad8\u6548\u7684Scrapy\u4ee3\u7406IP\u6c60\u663e\u5f97\u81f3\u5173\u91cd\u8981\u3002<\/p>\n\n\n\n<p>\u672c\u6587\u5c06\u6df1\u5165\u63a2\u8ba8\u5982\u4f55\u5229\u7528\u6613\u8def\u4ee3\u7406\uff08YiLu Proxy\uff09\u63d0\u4f9b\u7684\u5168\u7403\u9ad8\u533f\u540d\u4f4f\u5b85\u4e0e\u6570\u636e\u4e2d\u5fc3IP\u4ee3\u7406\u670d\u52a1\uff0c\u642d\u5efa\u4e00\u4e2a\u53ef\u9760\u7684Scrapy\u4ee3\u7406IP\u6c60\uff0c\u4ece\u800c\u6709\u6548\u907f\u514dIP\u5c01\u9501\uff0c\u63d0\u5347\u722c\u866b\u6548\u7387\uff0c\u52a9\u529b\u60a8\u7684\u8de8\u5883\u7535\u5546\u3001\u793e\u4ea4\u5a92\u4f53\u8fd0\u8425\u3001SEO\u4f18\u5316\u7b49\u4e1a\u52a1\u573a\u666f\u3002<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">\u4e3a\u4ec0\u4e48Scrapy\u9700\u8981\u4ee3\u7406IP\u6c60\uff1f<\/h2>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter\"><img decoding=\"async\" src=\"http:\/\/www.yilus5.com\/blog\/wp-content\/uploads\/image-2025-04-24T092353.726.jpg\" alt=\"\"\/><\/figure>\n<\/div>\n\n\n<p>Scrapy\u4f5c\u4e3a\u4e00\u4e2a\u9ad8\u6548\u7684\u722c\u866b\u6846\u67b6\uff0c\u80fd\u591f\u5feb\u901f\u5730\u53d1\u9001\u5927\u91cf\u8bf7\u6c42\u3002\u7136\u800c\uff0c\u8bb8\u591a\u7f51\u7ad9\u4e3a\u4e86\u4fdd\u62a4\u81ea\u8eab\u6570\u636e\u548c\u670d\u52a1\u5668\u8d44\u6e90\uff0c\u4f1a\u91c7\u53d6\u53cd\u722c\u866b\u7b56\u7565\uff0c\u4f8b\u5982\u9650\u5236\u5355\u4e2aIP\u5728\u4e00\u5b9a\u65f6\u95f4\u5185\u7684\u8bbf\u95ee\u9891\u7387\u3002\u5f53Scrapy\u722c\u866b\u7684\u8bf7\u6c42\u9891\u7387\u8d85\u8fc7\u8fd9\u4e2a\u9650\u5236\u65f6\uff0c\u60a8\u7684IP\u5730\u5740\u5f88\u53ef\u80fd\u4f1a\u88ab\u76ee\u6807\u7f51\u7ad9\u6682\u65f6\u751a\u81f3\u6c38\u4e45\u5c01\u9501\uff0c\u5bfc\u81f4\u722c\u866b\u4efb\u52a1\u4e2d\u65ad\uff0c\u6570\u636e\u6293\u53d6\u5931\u8d25\u3002<\/p>\n\n\n\n<p>\u4f7f\u7528\u4ee3\u7406IP\uff0c\u7279\u522b\u662f\u9ad8\u8d28\u91cf\u7684\u533f\u540d\u4ee3\u7406IP\uff0c\u53ef\u4ee5\u6709\u6548\u5730\u9690\u85cf\u60a8\u7684\u771f\u5b9eIP\u5730\u5740\uff0c\u5c06\u8bf7\u6c42\u901a\u8fc7\u4ee3\u7406\u670d\u52a1\u5668\u53d1\u9001\u51fa\u53bb\uff0c\u4ece\u800c\u7ed5\u8fc7\u76ee\u6807\u7f51\u7ad9\u7684IP\u9650\u5236\u3002\u800c\u6784\u5efa\u4e00\u4e2a\u4ee3\u7406IP\u6c60\uff0c\u5219\u610f\u5473\u7740\u60a8\u62e5\u6709\u591a\u4e2a\u53ef\u7528\u7684\u4ee3\u7406IP\uff0cScrapy\u722c\u866b\u53ef\u4ee5\u4ece\u4e2d\u968f\u673a\u9009\u62e9\u6216\u6839\u636e\u7b56\u7565\u8f6e\u6362\u4f7f\u7528\uff0c\u8fdb\u4e00\u6b65\u964d\u4f4e\u88ab\u5c01\u9501\u7684\u98ce\u9669\uff0c\u63d0\u9ad8\u722c\u866b\u7684\u7a33\u5b9a\u6027\u548c\u6548\u7387\u3002<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">\u6613\u8def\u4ee3\u7406\uff1a\u6784\u5efa\u9ad8\u8d28\u91cf\u4ee3\u7406IP\u6c60\u7684\u7406\u60f3\u9009\u62e9<\/h2>\n\n\n\n<p><a href=\"https:\/\/www.yilus5.com\/\">\u6613\u8def\u4ee3\u7406<\/a>\uff08YiLu Proxy\uff09\u4f5c\u4e3a\u5168\u7403\u9886\u5148\u7684IP\u4ee3\u7406\u670d\u52a1\u63d0\u4f9b\u5546\uff0c\u4e3a\u7528\u6237\u63d0\u4f9b\u4e86\u4e30\u5bcc\u7684IP\u8d44\u6e90\u548c\u7075\u6d3b\u7684\u4ee3\u7406\u65b9\u6848\uff0c\u662f\u6784\u5efaScrapy\u4ee3\u7406IP\u6c60\u7684\u7406\u60f3\u9009\u62e9\u3002\u5176\u4e3b\u8981\u7279\u70b9\u5305\u62ec\uff1a<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>\u5168\u7403\u9ad8\u533f\u540d\u4f4f\u5b85\u4e0e\u6570\u636e\u4e2d\u5fc3IP:<\/strong> \u6613\u8def\u4ee3\u7406\u63d0\u4f9b\u8986\u76d6\u5168\u7403\u591a\u4e2a\u5730\u533a\u7684\u4f4f\u5b85IP\u548c\u6570\u636e\u4e2d\u5fc3IP\u3002\u4f4f\u5b85IP\u5177\u6709\u66f4\u9ad8\u7684\u533f\u540d\u6027\u548c\u771f\u5b9e\u6027\uff0c\u66f4\u4e0d\u5bb9\u6613\u88ab\u76ee\u6807\u7f51\u7ad9\u8bc6\u522b\u4e3a\u4ee3\u7406\uff1b\u6570\u636e\u4e2d\u5fc3IP\u5219\u62e5\u6709\u66f4\u9ad8\u7684\u901f\u5ea6\u548c\u7a33\u5b9a\u6027\uff0c\u9002\u7528\u4e8e\u5bf9\u901f\u5ea6\u8981\u6c42\u8f83\u9ad8\u7684\u573a\u666f\u3002\u60a8\u53ef\u4ee5\u6839\u636e\u5b9e\u9645\u9700\u6c42\u7075\u6d3b\u9009\u62e9\u3002<\/li>\n\n\n\n<li><strong>\u652f\u6301HTTP\/SOCKS5\u534f\u8bae:<\/strong> \u6613\u8def\u4ee3\u7406\u540c\u65f6\u652f\u6301HTTP\u548cSOCKS5\u4e24\u79cd\u4ee3\u7406\u534f\u8bae\uff0c\u60a8\u53ef\u4ee5\u6839\u636e\u76ee\u6807\u7f51\u7ad9\u7684\u8981\u6c42\u548cScrapy\u7684\u914d\u7f6e\u8fdb\u884c\u9009\u62e9\uff0c\u786e\u4fdd\u6700\u4f73\u7684\u517c\u5bb9\u6027\u548c\u6027\u80fd\u3002SOCKS5\u534f\u8bae\u901a\u5e38\u5177\u6709\u66f4\u597d\u7684\u901a\u7528\u6027\u548c\u5b89\u5168\u6027\u3002<\/li>\n\n\n\n<li><strong>\u52a8\u6001\u4e0e\u9759\u6001\u72ec\u4eabIP:<\/strong> \u6613\u8def\u4ee3\u7406\u63d0\u4f9b\u52a8\u6001\u548c\u9759\u6001\u4e24\u79cd\u72ec\u4eabIP\u3002\u52a8\u6001\u72ec\u4eabIP\u6bcf\u6b21\u8fde\u63a5\u90fd\u4f1a\u83b7\u53d6\u4e00\u4e2a\u65b0\u7684IP\u5730\u5740\uff0c\u8fdb\u4e00\u6b65\u63d0\u9ad8\u533f\u540d\u6027\uff1b\u9759\u6001\u72ec\u4eabIP\u5219\u4e3a\u60a8\u63d0\u4f9b\u4e00\u4e2a\u56fa\u5b9a\u7684IP\u5730\u5740\uff0c\u9002\u7528\u4e8e\u9700\u8981\u957f\u671f\u7a33\u5b9a\u8bbf\u95ee\u7684\u573a\u666f\uff0c\u4f8b\u5982\u67d0\u4e9b\u9700\u8981IP\u767d\u540d\u5355\u7684API\u63a5\u53e3\u3002<\/li>\n\n\n\n<li><strong>\u7a33\u5b9a\u9ad8\u901f\u7684\u7f51\u7edc:<\/strong> \u6613\u8def\u4ee3\u7406\u62e5\u6709\u5f3a\u5927\u7684\u670d\u52a1\u5668 infrastructure \u548c\u4f18\u5316\u7684\u7f51\u7edc\u7ebf\u8def\uff0c\u4fdd\u8bc1\u4e86\u4ee3\u7406IP\u7684\u7a33\u5b9a\u6027\u548c\u9ad8\u901f\u8bbf\u95ee\uff0c\u8fd9\u5bf9\u4e8eScrapy\u722c\u866b\u7684\u9ad8\u6548\u8fd0\u884c\u81f3\u5173\u91cd\u8981\u3002<\/li>\n\n\n\n<li><strong>\u9002\u7528\u4e8e\u591a\u79cd\u4e1a\u52a1\u573a\u666f:<\/strong> \u65e0\u8bba\u662f\u8de8\u5883\u7535\u5546\u7684\u6570\u636e\u91c7\u96c6\u3001\u793e\u4ea4\u5a92\u4f53\u8d26\u53f7\u7684\u8fd0\u8425\u7ba1\u7406\u3001\u8fd8\u662fSEO\u5173\u952e\u8bcd\u7684\u4f18\u5316\u76d1\u63a7\uff0c\u6613\u8def\u4ee3\u7406\u90fd\u80fd\u63d0\u4f9b\u7a33\u5b9a\u53ef\u9760\u7684IP\u652f\u6301\uff0c\u52a9\u60a8\u8f7b\u677e\u5e94\u5bf9\u5404\u79cd\u7f51\u7edc\u6311\u6218\u3002<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">\u4f7f\u7528Scrapy\u7ed3\u5408\u6613\u8def\u4ee3\u7406\u642d\u5efaIP\u6c60\u7684\u6b65\u9aa4<\/h2>\n\n\n\n<p>\u4e0b\u9762\u6211\u4eec\u5c06\u8be6\u7ec6\u4ecb\u7ecd\u5982\u4f55\u4f7f\u7528Scrapy\u7ed3\u5408\u6613\u8def\u4ee3\u7406\u63d0\u4f9b\u7684IP\u670d\u52a1\uff0c\u642d\u5efa\u4e00\u4e2a\u57fa\u672c\u7684\u4ee3\u7406IP\u6c60\u3002<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">\u6b65\u9aa4\u4e00\uff1a\u6ce8\u518c\u5e76\u83b7\u53d6\u6613\u8def\u4ee3\u7406IP<\/h3>\n\n\n\n<p>\u9996\u5148\uff0c\u60a8\u9700\u8981\u5728\u6613\u8def\u4ee3\u7406\u5b98\u7f51\uff08\u8bf7\u81ea\u884c\u641c\u7d22\u6613\u8def\u4ee3\u7406\u5b98\u65b9\u7f51\u7ad9\uff09\u6ce8\u518c\u8d26\u53f7\u5e76\u8d2d\u4e70\u9002\u5408\u60a8\u9700\u6c42\u7684IP\u5957\u9910\u3002\u6839\u636e\u60a8\u7684\u4e1a\u52a1\u91cf\u548c\u5bf9IP\u8d28\u91cf\u7684\u8981\u6c42\uff0c\u9009\u62e9\u5408\u9002\u7684IP\u7c7b\u578b\uff08\u4f4f\u5b85\/\u6570\u636e\u4e2d\u5fc3\uff09\u3001\u534f\u8bae\uff08HTTP\/SOCKS5\uff09\u3001\u4ee5\u53caIP\u6570\u91cf\u3002<\/p>\n\n\n\n<p>\u8d2d\u4e70\u6210\u529f\u540e\uff0c\u60a8\u53ef\u4ee5\u5728\u6613\u8def\u4ee3\u7406\u7684\u540e\u53f0\u7ba1\u7406\u754c\u9762\u83b7\u53d6\u60a8\u7684API\u5bc6\u94a5\u3001IP\u5217\u8868\u3001\u7aef\u53e3\u4fe1\u606f\u4ee5\u53ca\u8ba4\u8bc1\u65b9\u5f0f\u7b49\u3002\u8bf7\u52a1\u5fc5\u59a5\u5584\u4fdd\u7ba1\u8fd9\u4e9b\u4fe1\u606f\uff0c\u5b83\u4eec\u5c06\u7528\u4e8eScrapy\u722c\u866b\u7684\u914d\u7f6e\u3002<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">\u6b65\u9aa4\u4e8c\uff1a\u5728Scrapy\u9879\u76ee\u4e2d\u914d\u7f6e\u4e2d\u95f4\u4ef6<\/h3>\n\n\n\n<p>Scrapy\u7684\u4e2d\u95f4\u4ef6\uff08Middleware\uff09\u673a\u5236\u5141\u8bb8\u60a8\u5728\u8bf7\u6c42\u53d1\u9001\u524d\u548c\u54cd\u5e94\u5230\u8fbe\u540e\u63d2\u5165\u81ea\u5b9a\u4e49\u7684\u5904\u7406\u903b\u8f91\u3002\u6211\u4eec\u9700\u8981\u521b\u5efa\u4e00\u4e2a\u81ea\u5b9a\u4e49\u7684\u4e0b\u8f7d\u5668\u4e2d\u95f4\u4ef6\uff08Downloader Middleware\uff09\u6765\u5904\u7406\u4ee3\u7406IP\u7684\u8bbe\u7f6e\u548c\u8f6e\u6362\u3002<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>\u6253\u5f00\u60a8\u7684Scrapy\u9879\u76ee\uff0c\u627e\u5230<code>settings.py<\/code>\u6587\u4ef6\u3002<\/strong><\/li>\n\n\n\n<li><strong>\u53d6\u6d88\u6ce8\u91ca<code>DOWNLOADER_MIDDLEWARES<\/code>\u8bbe\u7f6e\uff0c\u5e76\u6dfb\u52a0\u60a8\u7684\u81ea\u5b9a\u4e49\u4e2d\u95f4\u4ef6\u3002<\/strong> \u4f8b\u5982\uff0c\u521b\u5efa\u4e00\u4e2a\u540d\u4e3a<code>ProxyMiddleware<\/code>\u7684\u4e2d\u95f4\u4ef6\uff0c\u5e76\u8bbe\u7f6e\u5176\u4f18\u5148\u7ea7\uff1a Python<code>DOWNLOADER_MIDDLEWARES = { 'your_project_name.middlewares.ProxyMiddleware': 750, }<\/code> \u8bf7\u5c06<code>your_project_name<\/code>\u66ff\u6362\u4e3a\u60a8\u7684\u5b9e\u9645\u9879\u76ee\u540d\u79f0\u3002<\/li>\n\n\n\n<li><strong>\u5728\u60a8\u7684Scrapy\u9879\u76ee\u76ee\u5f55\u4e0b\uff08\u901a\u5e38\u4e0e<code>spiders<\/code>\u6587\u4ef6\u5939\u540c\u7ea7\uff09\u7684<code>middlewares.py<\/code>\u6587\u4ef6\u4e2d\uff0c\u521b\u5efa<code>ProxyMiddleware<\/code>\u7c7b\u3002<\/strong> Python<code>import base64 import random from scrapy import signals class ProxyMiddleware: def __init__(self): # \u4ece\u6613\u8def\u4ee3\u7406\u540e\u53f0\u83b7\u53d6\u60a8\u7684\u4ee3\u7406IP\u5217\u8868\u548c\u8ba4\u8bc1\u4fe1\u606f self.proxy_list = [ {'ip_port': 'ip1:port1', 'username': 'user1', 'password': 'password1'}, {'ip_port': 'ip2:port2', 'username': 'user2', 'password': 'password2'}, # ... \u66f4\u591a\u4ee3\u7406IP ] # \u5982\u679c\u60a8\u7684\u4ee3\u7406\u9700\u8981\u8ba4\u8bc1\uff0c\u8bf7\u8bbe\u7f6eauth\u4e3aTrue self.auth = True def process_request(self, request, spider): if self.proxy_list: proxy = random.choice(self.proxy_list) request.meta['proxy'] = f\"http:\/\/{proxy['ip_port']}\" if request.url.startswith('http') else f\"socks5:\/\/{proxy['ip_port']}\" if self.auth and proxy['username'] and proxy['password']: # \u5bf9\u9700\u8981\u8ba4\u8bc1\u7684\u4ee3\u7406\u8fdb\u884cBase64\u7f16\u7801 auth = base64.b64encode(f\"{proxy['username']}:{proxy['password']}\".encode()).decode() request.headers['Proxy-Authorization'] = f'Basic {auth}' def process_response(self, request, response, spider): # \u53ef\u9009\uff1a\u5728\u8fd9\u91cc\u5904\u7406\u88ab\u5c01\u9501\u7684\u54cd\u5e94\uff0c\u4f8b\u5982\u66f4\u6362\u4ee3\u7406IP\u91cd\u8bd5 if response.status in [403, 503]: # \u4ece\u4ee3\u7406\u5217\u8868\u4e2d\u79fb\u9664\u5f53\u524d\u5931\u6548\u7684\u4ee3\u7406\uff08\u53ef\u9009\uff09 if 'proxy' in request.meta: print(f\"\u4ee3\u7406 {request.meta['proxy']} \u8bbf\u95ee\u5931\u8d25\uff0c\u72b6\u6001\u7801\uff1a{response.status}\") # self.proxy_list = [p for p in self.proxy_list if p['ip_port'] not in request.meta['proxy']] # \u91cd\u65b0\u53d1\u8d77\u8bf7\u6c42\uff0c\u5c1d\u8bd5\u4f7f\u7528\u65b0\u7684\u4ee3\u7406 new_request = request.copy() new_request.dont_filter = True return new_request return response def process_exception(self, request, exception, spider): # \u53ef\u9009\uff1a\u5728\u8fd9\u91cc\u5904\u7406\u8bf7\u6c42\u5f02\u5e38\uff0c\u4f8b\u5982\u66f4\u6362\u4ee3\u7406IP\u91cd\u8bd5 if 'proxy' in request.meta: print(f\"\u4ee3\u7406 {request.meta['proxy']} \u53d1\u751f\u5f02\u5e38\uff1a{exception}\") # self.proxy_list = [p for p in self.proxy_list if p['ip_port'] not in request.meta['proxy']] # \u91cd\u65b0\u53d1\u8d77\u8bf7\u6c42\uff0c\u5c1d\u8bd5\u4f7f\u7528\u65b0\u7684\u4ee3\u7406 new_request = request.copy() new_request.dont_filter = True return new_request<\/code><strong>\u8bf7\u6ce8\u610f\u66ff\u6362\u4ee3\u7801\u4e2d\u7684\u4ee5\u4e0b\u5185\u5bb9\uff1a<\/strong>\n<ul class=\"wp-block-list\">\n<li><code>your_project_name<\/code>: \u60a8\u7684Scrapy\u9879\u76ee\u540d\u79f0\u3002<\/li>\n\n\n\n<li><code>ip1:port1<\/code>, <code>user1<\/code>, <code>password1<\/code>\u7b49: \u60a8\u4ece\u6613\u8def\u4ee3\u7406\u540e\u53f0\u83b7\u53d6\u7684\u5b9e\u9645\u4ee3\u7406IP\u5730\u5740\u3001\u7aef\u53e3\u3001\u7528\u6237\u540d\u548c\u5bc6\u7801\u3002\u6839\u636e\u60a8\u8d2d\u4e70\u7684IP\u7c7b\u578b\u548c\u8ba4\u8bc1\u65b9\u5f0f\u8fdb\u884c\u586b\u5199\u3002\u5982\u679c\u60a8\u7684\u4ee3\u7406\u4e0d\u9700\u8981\u7528\u6237\u540d\u548c\u5bc6\u7801\uff0c\u53ef\u4ee5\u5c06<code>self.auth<\/code>\u8bbe\u7f6e\u4e3a<code>False<\/code>\uff0c\u5e76\u79fb\u9664<code>process_request<\/code>\u65b9\u6cd5\u4e2d\u5173\u4e8e<code>Proxy-Authorization<\/code>\u7684\u903b\u8f91\u3002<\/li>\n\n\n\n<li>\u6839\u636e\u60a8\u4f7f\u7528\u7684\u4ee3\u7406\u534f\u8bae\uff08HTTP\u6216SOCKS5\uff09\u4fee\u6539<code>request.meta['proxy']<\/code>\u7684\u8d4b\u503c\u3002\u6613\u8def\u4ee3\u7406\u652f\u6301HTTP\u548cSOCKS5\uff0c\u8bf7\u6839\u636e\u60a8\u7684\u9700\u6c42\u9009\u62e9\u3002<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n<h3 class=\"wp-block-heading\">\u6b65\u9aa4\u4e09\uff1a\u914d\u7f6eScrapy\u8bbe\u7f6e<\/h3>\n\n\n\n<p>\u5728<code>settings.py<\/code>\u6587\u4ef6\u4e2d\uff0c\u786e\u4fdd\u4ee5\u4e0b\u8bbe\u7f6e\u6ca1\u6709\u88ab\u7981\u7528\uff1a<\/p>\n\n\n\n<p>Python<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># \u542f\u7528\u4e0b\u8f7d\u5668\u4e2d\u95f4\u4ef6\nDOWNLOADER_MIDDLEWARES = {\n    'your_project_name.middlewares.ProxyMiddleware': 750,\n}\n\n# \u8bbe\u7f6e\u4e0b\u8f7d\u5ef6\u8fdf\uff0c\u907f\u514d\u8bf7\u6c42\u8fc7\u5feb\u88ab\u5c01\u9501\uff08\u5373\u4f7f\u4f7f\u7528\u4e86\u4ee3\u7406\uff09\nDOWNLOAD_DELAY = 0.25  # \u53ef\u4ee5\u6839\u636e\u76ee\u6807\u7f51\u7ad9\u7684\u60c5\u51b5\u8c03\u6574\n\n# \u5982\u679c\u76ee\u6807\u7f51\u7ad9\u9700\u8981User-Agent\uff0c\u8bf7\u8bbe\u7f6e\u4e00\u4e2a\u5408\u9002\u7684User-Agent\nUSER_AGENT = 'Mozilla\/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/114.0.0.0 Safari\/537.36'\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">\u6b65\u9aa4\u56db\uff1a\u8fd0\u884c\u60a8\u7684Scrapy\u722c\u866b<\/h3>\n\n\n\n<p>\u5b8c\u6210\u4ee5\u4e0a\u914d\u7f6e\u540e\uff0c\u60a8\u53ef\u4ee5\u50cf\u5f80\u5e38\u4e00\u6837\u8fd0\u884c\u60a8\u7684Scrapy\u722c\u866b\u3002Scrapy\u5c06\u4f1a\u901a\u8fc7\u60a8\u914d\u7f6e\u7684<code>ProxyMiddleware<\/code>\u81ea\u52a8\u9009\u62e9\u548c\u8f6e\u6362\u4ee3\u7406IP\u53d1\u9001\u8bf7\u6c42\u3002<\/p>\n\n\n\n<p>Bash<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>scrapy crawl your_spider_name\n<\/code><\/pre>\n\n\n\n<p>\u8bf7\u5c06<code>your_spider_name<\/code>\u66ff\u6362\u4e3a\u60a8\u7684\u722c\u866b\u540d\u79f0\u3002<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">\u8fdb\u9636\u6280\u5de7\uff1a\u6784\u5efa\u66f4\u667a\u80fd\u7684\u4ee3\u7406IP\u6c60<\/h2>\n\n\n\n<p>\u4ee5\u4e0a\u662f\u4e00\u4e2a\u57fa\u672c\u7684\u4ee3\u7406IP\u6c60\u642d\u5efa\u65b9\u6cd5\u3002\u4e3a\u4e86\u6784\u5efa\u66f4\u667a\u80fd\u3001\u66f4\u7a33\u5b9a\u7684\u4ee3\u7406IP\u6c60\uff0c\u60a8\u53ef\u4ee5\u8003\u8651\u4ee5\u4e0b\u8fdb\u9636\u6280\u5de7\uff1a<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>\u52a8\u6001\u83b7\u53d6\u4ee3\u7406IP:<\/strong> \u5c06\u4ee3\u7406IP\u5217\u8868\u7684\u83b7\u53d6\u903b\u8f91\u4ece\u786c\u7f16\u7801\u6539\u4e3a\u4ece\u6613\u8def\u4ee3\u7406\u7684API\u63a5\u53e3\u52a8\u6001\u83b7\u53d6\uff0c\u8fd9\u6837\u53ef\u4ee5\u5b9e\u65f6\u66f4\u65b0\u53ef\u7528\u7684IP\uff0c\u907f\u514d\u56e0\u90e8\u5206IP\u5931\u6548\u800c\u5bfc\u81f4\u722c\u866b\u9519\u8bef\u3002\u6613\u8def\u4ee3\u7406\u901a\u5e38\u4f1a\u63d0\u4f9bAPI\u63a5\u53e3\uff0c\u65b9\u4fbf\u7528\u6237\u7a0b\u5e8f\u5316\u5730\u83b7\u53d6\u548c\u7ba1\u7406IP\u3002<\/li>\n\n\n\n<li><strong>\u4ee3\u7406IP\u8d28\u91cf\u68c0\u6d4b:<\/strong> \u5728\u4f7f\u7528\u4ee3\u7406IP\u4e4b\u524d\uff0c\u53ef\u4ee5\u5148\u5bf9\u5176\u8fdb\u884c\u53ef\u7528\u6027\u6d4b\u8bd5\u548c\u533f\u540d\u6027\u6d4b\u8bd5\uff0c\u4f8b\u5982\u901a\u8fc7\u8bbf\u95ee\u4e00\u4e2a\u5df2\u77e5\u7684IP\u67e5\u8be2\u7f51\u7ad9\uff0c\u68c0\u67e5\u4ee3\u7406IP\u662f\u5426\u53ef\u7528\u4ee5\u53ca\u662f\u5426\u9690\u85cf\u4e86\u771f\u5b9eIP\u3002Scrapy\u53ef\u4ee5\u96c6\u6210\u76f8\u5173\u7684\u6d4b\u8bd5\u903b\u8f91\uff0c\u53ea\u4f7f\u7528\u9ad8\u8d28\u91cf\u7684\u4ee3\u7406IP\u3002<\/li>\n\n\n\n<li><strong>\u5931\u8d25\u91cd\u8bd5\u673a\u5236:<\/strong> \u5f53\u4f7f\u7528\u67d0\u4e2a\u4ee3\u7406IP\u8bf7\u6c42\u5931\u8d25\u65f6\uff08\u4f8b\u5982\u8fd4\u56de403\u6216503\u72b6\u6001\u7801\uff09\uff0c\u53ef\u4ee5\u5c1d\u8bd5\u66f4\u6362\u5176\u4ed6\u4ee3\u7406IP\u5e76\u91cd\u65b0\u53d1\u9001\u8bf7\u6c42\u3002Scrapy\u7684<code>RetryMiddleware<\/code>\u53ef\u4ee5\u8fdb\u884c\u7b80\u5355\u7684\u91cd\u8bd5\u914d\u7f6e\uff0c\u60a8\u53ef\u4ee5\u81ea\u5b9a\u4e49\u66f4\u590d\u6742\u7684\u91cd\u8bd5\u7b56\u7565\uff0c\u7ed3\u5408\u4ee3\u7406IP\u7684\u66f4\u6362\u3002<\/li>\n\n\n\n<li><strong>IP\u4fe1\u8a89\u7ba1\u7406:<\/strong> \u8bb0\u5f55\u6bcf\u4e2a\u4ee3\u7406IP\u7684\u6210\u529f\u7387\u548c\u5931\u8d25\u7387\uff0c\u5bf9\u4e8e\u5931\u8d25\u7387\u8fc7\u9ad8\u7684IP\uff0c\u53ef\u4ee5\u6682\u65f6\u6216\u6c38\u4e45\u5730\u4eceIP\u6c60\u4e2d\u79fb\u9664\uff0c\u4ee5\u63d0\u9ad8\u6574\u4f53\u7684\u722c\u866b\u8d28\u91cf\u3002<\/li>\n\n\n\n<li><strong>\u667a\u80fdIP\u9009\u62e9\u7b56\u7565:<\/strong> \u6839\u636e\u76ee\u6807\u7f51\u7ad9\u7684\u4e0d\u540c\u53cd\u722c\u866b\u7b56\u7565\uff0c\u91c7\u7528\u4e0d\u540c\u7684IP\u9009\u62e9\u7b56\u7565\u3002\u4f8b\u5982\uff0c\u5bf9\u4e8e\u53cd\u722c\u4e25\u683c\u7684\u7f51\u7ad9\uff0c\u4f18\u5148\u4f7f\u7528\u9ad8\u533f\u540d\u7684\u4f4f\u5b85IP\uff1b\u5bf9\u4e8e\u901f\u5ea6\u8981\u6c42\u9ad8\u7684\u573a\u666f\uff0c\u53ef\u4ee5\u5c1d\u8bd5\u4f7f\u7528\u901f\u5ea6\u8f83\u5feb\u7684\u6570\u636e\u4e2d\u5fc3IP\u3002<\/li>\n\n\n\n<li><strong>\u7ed3\u5408\u6613\u8def\u4ee3\u7406\u7684\u72ec\u4eabIP\u4f18\u52bf:<\/strong> \u5bf9\u4e8e\u9700\u8981\u957f\u671f\u7a33\u5b9a\u8bbf\u95ee\u7684\u573a\u666f\uff0c\u4f8b\u5982API\u63a5\u53e3\u8c03\u7528\u6216\u9700\u8981\u7ef4\u6301\u767b\u5f55\u72b6\u6001\u7684\u64cd\u4f5c\uff0c\u53ef\u4ee5\u8003\u8651\u4f7f\u7528\u6613\u8def\u4ee3\u7406\u63d0\u4f9b\u7684\u9759\u6001\u72ec\u4eabIP\uff0c\u907f\u514dIP\u9891\u7e41\u66f4\u6362\u5e26\u6765\u7684\u95ee\u9898\u3002<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">\u603b\u7ed3<\/h2>\n\n\n\n<p>\u6784\u5efa\u4e00\u4e2a\u7a33\u5b9a\u9ad8\u6548\u7684Scrapy\u4ee3\u7406IP\u6c60\u662f\u6210\u529f\u8fdb\u884c\u7f51\u7edc\u6570\u636e\u722c\u53d6\u7684\u5173\u952e\u4e00\u6b65\u3002\u901a\u8fc7\u7ed3\u5408\u6613\u8def\u4ee3\u7406\u63d0\u4f9b\u7684\u5168\u7403\u9ad8\u533f\u540d\u4f4f\u5b85\u4e0e\u6570\u636e\u4e2d\u5fc3IP\u8d44\u6e90\uff0c\u60a8\u53ef\u4ee5\u8f7b\u677e\u5730\u642d\u5efa\u4e00\u4e2a\u6ee1\u8db3\u5404\u79cd\u4e1a\u52a1\u9700\u6c42\u7684\u4ee3\u7406IP\u6c60\u3002\u8bb0\u4f4f\uff0c\u6839\u636e\u60a8\u7684\u5b9e\u9645\u722c\u53d6\u76ee\u6807\u548c\u7f51\u7ad9\u7684\u53cd\u722c\u866b\u7b56\u7565\uff0c\u7075\u6d3b\u5730\u914d\u7f6e\u548c\u4f18\u5316\u60a8\u7684\u4ee3\u7406IP\u6c60\uff0c\u624d\u80fd\u6700\u5927\u9650\u5ea6\u5730\u63d0\u9ad8\u722c\u866b\u7684\u6548\u7387\u548c\u7a33\u5b9a\u6027\uff0c\u6700\u7ec8\u83b7\u53d6\u60a8\u6240\u9700\u7684\u6570\u636e\uff0c\u52a9\u529b\u60a8\u7684\u4e1a\u52a1\u53d1\u5c55\u3002\u6613\u8def\u4ee3\u7406\u7684\u7a33\u5b9a\u9ad8\u901f\u548c\u591a\u6837\u5316\u7684IP\u9009\u62e9\uff0c\u65e0\u7591\u4e3aScrapy\u722c\u866b\u63d0\u4f9b\u4e86\u4e00\u4e2a\u5f3a\u5927\u7684\u540e\u76fe\uff0c\u8ba9\u60a8\u80fd\u591f\u66f4\u4e13\u6ce8\u4e8e\u6570\u636e\u5206\u6790\u548c\u4e1a\u52a1\u903b\u8f91\u7684\u5b9e\u73b0\uff0c\u800c\u65e0\u9700\u8fc7\u591a\u62c5\u5fc3IP\u5c01\u9501\u5e26\u6765\u7684\u56f0\u6270\u3002<\/p>\n","protected":false},"excerpt":{"rendered":"<p>\u5728\u7f51\u7edc\u4fe1\u606f\u7206\u70b8\u7684\u4eca\u5929\uff0c\u6570\u636e\u5df2\u7ecf\u6210\u4e3a\u9a71\u52a8\u4e1a\u52a1\u589e\u957f\u7684\u6838\u5fc3\u52a8\u529b\u3002\u5bf9\u4e8e\u9700\u8981\u5927\u89c4\u6a21\u3001\u81ea\u52a8\u5316\u6293\u53d6\u7f51\u9875\u6570\u636e\u7684\u5f00\u53d1\u8005\u548c\u4f01\u4e1a\u800c\u8a00 &#8230; <a title=\"Scrapy\u4ee3\u7406IP\u6c60\u642d\u5efa\u6559\u7a0b\uff1a\u9ad8\u6548\u7a33\u5b9a\u722c\u53d6\u6570\u636e\u7684\u5173\u952e\u4e00\u6b65\" class=\"read-more\" href=\"https:\/\/www.yilus5.com\/blog\/1732.html\" aria-label=\"\u9605\u8bfb Scrapy\u4ee3\u7406IP\u6c60\u642d\u5efa\u6559\u7a0b\uff1a\u9ad8\u6548\u7a33\u5b9a\u722c\u53d6\u6570\u636e\u7684\u5173\u952e\u4e00\u6b65\">\u9605\u8bfb\u66f4\u591a<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[18],"tags":[],"class_list":["post-1732","post","type-post","status-publish","format-standard","hentry","category-yiluproxy16"],"_links":{"self":[{"href":"https:\/\/www.yilus5.com\/blog\/wp-json\/wp\/v2\/posts\/1732","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.yilus5.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.yilus5.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.yilus5.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.yilus5.com\/blog\/wp-json\/wp\/v2\/comments?post=1732"}],"version-history":[{"count":1,"href":"https:\/\/www.yilus5.com\/blog\/wp-json\/wp\/v2\/posts\/1732\/revisions"}],"predecessor-version":[{"id":1733,"href":"https:\/\/www.yilus5.com\/blog\/wp-json\/wp\/v2\/posts\/1732\/revisions\/1733"}],"wp:attachment":[{"href":"https:\/\/www.yilus5.com\/blog\/wp-json\/wp\/v2\/media?parent=1732"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.yilus5.com\/blog\/wp-json\/wp\/v2\/categories?post=1732"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.yilus5.com\/blog\/wp-json\/wp\/v2\/tags?post=1732"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}