热门网络流行语查询历史数据爬虫

1. 还是先分析一下网站结构，网站地址

可见列表页就在首页，且下面又加载更多，是做翻页使用的，盲猜是ajax渲染，数据因该是在接口中而不是html。 抓下包看一下，打开F12

额.哈哈，居然在json接口中又放了html源码，这个老六，这种情况一般都是ajax先拿出来数据然后前端js渲染到页面上。看一下参数也只有一个page作为翻页使用的，那这个列表页就没有什么问题了，然后我们看一下详解页数据接口是什么样子的。

点击之后直接打开了新的页面，并且有文章id，这包页不用抓了，八成是在html中的数据，ok整个流程就明了了，需要先...然后...最后...就ok了

1. 先请求一下列表页并解析出来所有的梗题目和id

headers = {
    'authority': 'www.shenmegeng.cn',
    'accept': 'application/json, text/javascript, */*; q=0.01',
    'accept-language': 'zh-CN,zh;q=0.9',
    'cache-control': 'no-cache',
    # 'content-length': '0',
    'cookie': 'PHPSESSID=6qjhh5momeu5eseueho13drhid; __51uvsct__JSHvsSrYOkXClLjw=1; __51vcke__JSHvsSrYOkXClLjw=a7620899-bb09-5d4b-807d-04797197187b; __51vuft__JSHvsSrYOkXClLjw=1682472158732; _tcnyl=1; __vtins__JSHvsSrYOkXClLjw=%7B%22sid%22%3A%20%224aeeee8a-a8d5-5443-8378-7be81ebe73b5%22%2C%20%22vd%22%3A%203%2C%20%22stt%22%3A%2080196%2C%20%22dr%22%3A%2074124%2C%20%22expires%22%3A%201682474038925%2C%20%22ct%22%3A%201682472238925%7D',
    'origin': 'https://www.shenmegeng.cn',
    'pragma': 'no-cache',
    'referer': 'https://www.shenmegeng.cn/',
    'sec-ch-ua': '" Not A;Brand";v="99", "Chromium";v="100", "Google Chrome";v="100"',
    'sec-ch-ua-mobile': '?0',
    'sec-ch-ua-platform': '"Windows"',
    'sec-fetch-dest': 'empty',
    'sec-fetch-mode': 'cors',
    'sec-fetch-site': 'same-origin',
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.60 Safari/537.36',
    'x-requested-with': 'XMLHttpRequest',
}

params = {
    'page': '3',
}

response = requests.post('https://www.shenmegeng.cn/', params=params, headers=headers).json()['data']
res = etree.HTML(response)
data_list = res.xpath("//div/article/div/div/h2/a")
for data in data_list:
    title = data.xpath("./text()")[0]
    url = data.xpath("./@href")[0]
    print(title,url)

没问题,先清洁一下链接再请求详情页就ok了

headers = {
    'authority': 'www.shenmegeng.cn',
    'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
    'accept-language': 'zh-CN,zh;q=0.9',
    'cache-control': 'no-cache',
    # 'cookie': 'PHPSESSID=6qjhh5momeu5eseueho13drhid; __51uvsct__JSHvsSrYOkXClLjw=1; __51vcke__JSHvsSrYOkXClLjw=a7620899-bb09-5d4b-807d-04797197187b; __51vuft__JSHvsSrYOkXClLjw=1682472158732; _tcnyl=1; __vtins__JSHvsSrYOkXClLjw=%7B%22sid%22%3A%20%224aeeee8a-a8d5-5443-8378-7be81ebe73b5%22%2C%20%22vd%22%3A%204%2C%20%22stt%22%3A%20934245%2C%20%22dr%22%3A%20854049%2C%20%22expires%22%3A%201682474892974%2C%20%22ct%22%3A%201682473092974%7D',
    'pragma': 'no-cache',
    'sec-ch-ua': '" Not A;Brand";v="99", "Chromium";v="100", "Google Chrome";v="100"',
    'sec-ch-ua-mobile': '?0',
    'sec-ch-ua-platform': '"Windows"',
    'sec-fetch-dest': 'document',
    'sec-fetch-mode': 'navigate',
    'sec-fetch-site': 'none',
    'sec-fetch-user': '?1',
    'upgrade-insecure-requests': '1',
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.60 Safari/537.36',
}

response = requests.get(url, headers=headers)
res = etree.HTML(response.text)
con = ''
ccc = res.xpath("//div[@class='article-text']//text()")
for c in ccc:
    con += c
print(title)
print(con)
print('========================')

ok，保护一下眼睛，数数有几株草