小白求助，这种爬虫问题怎么解决？？？？

作者站长头像
站长
· 阅读数 5

代码能运行，显示框没有任何文字小白求助，这种爬虫问题怎么解决？？？？

import requests
import lxml
from lxml import etree
def douban(bianma,n):
    for i in range(0,n,20):
        url = f"https://movie.douban.com/subject{bianma}/comments?start={i}&limit=20&status=P&sort=new_score"
        headers = {
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36',
            'cookie': 'll="108309"; bid=LGZ_GB9T9uY; __utmc=30149280; __utmz=30149280.1700787245.1.1.utmcsr=baidu|utmccn=(organic)|utmcmd=organic; _pk_id.100001.4cf6=c7613f3b72957e4e.1700787247.; __utmz=223695111.1700787247.1.1.utmcsr=douban.com|utmccn=(referral)|utmcmd=referral|utmcct=/; __utmc=223695111; __yadk_uid=mdIdeZjXoC0wsL4IPuQ4wdNGg5UmEDAY; _vwo_uuid_v2=DE11183014D98803B79CC7A320DB26A19|fa5bf1d7e9a9db0f272915f4dc3c782c; _pk_ref.100001.4cf6=%5B%22%22%2C%22%22%2C1700796132%2C%22https%3A%2F%2Fwww.douban.com%2F%22%5D; _pk_ses.100001.4cf6=1; __utma=30149280.2062835829.1700787245.1700787245.1700796132.2; __utmb=223695111.0.10.1700796132; __utma=223695111.1955455109.1700787247.1700787247.1700796132.2; ap_v=0,6.0; __utmt=1; ct=y; __utmb=30149280.5.10.1700796132; dbcl2="268931642:WgYrVWjADts"; ck=peuD; push_noty_num=0; push_doumail_num=0'
        }
        response=requests.get(url=url,headers=headers)
        # print(response.status_code)
        xinxi=etree.HTML(response.text)
        data=xinxi.xpath("/html/body/div[3]/div[1]/div/div[1]/div[4]/div/div[2]/p/span/text()")
        author=xinxi.xpath("/html/body/div[3]/div[1]/div/div[1]/div[4]/div/div[2]/h3/span[2]/a/text()")
        time=xinxi.xpath("/html/body/div[3]/div[1]/div/div[1]/div[4]/div/div[2]/h3/span[2]/span[3]/text()")
        diyu=xinxi.xpath("/html/body/div[3]/div[1]/div/div[1]/div[4]/div/div[2]/h3/span[2]/span[4]/text()")
        for a,b,c in zip(data,author,diyu):
            print("评论:"+a+"\n"+"作者:"+b+"\n"+"省份:"+c+"\n")
            with open("douban.txt","a+",encoding="utf-8")as file:
                file.write("评论:"+a+"\n"+"作者:"+b+"\n"+"时间:"+c+"\n")

# douban(35633650,600)#35633650对应堕落的审判
# douban(36283000,600)#36283000对应进击的巨人 最终季 完结篇 后篇
# douban(35918457,500)#35918457 对应快乐再出发
douban(35633650,500)#36563539 对应快乐老友记

怎么样能解决这个问题

回复
1个回答
avatar
test
2024-06-23

两个建议

  • 学一下HTTP知识，学一下Xpath语法
  • 学会断点调试

douban.com/subject{bianma}/comments 这边 subject/{bianma}少东西，URL不对

response = requests.get(url=url, headers=headers)
if not response.ok:
    # 对响应做个校验
    raise Exception(f'url: {url}, status_code: {response.status_code}, content: {response.content.decode()}')
回复
likes
适合作为回答的
  • 经过验证的有效解决办法
  • 自己的经验指引，对解决问题有帮助
  • 遵循 Markdown 语法排版，代码语义正确
不该作为回答的
  • 询问内容细节或回复楼层
  • 与题目无关的内容
  • “赞”“顶”“同问”“看手册”“解决了没”等毫无意义的内容