爬蟲與反爬蟲 | 土法破解字元映射反爬策略及高頻詞可視化

tool.chinaz.com

通過這兩步我們就可以知道ue232對應數字1，ue4ec對應數字5，以此類推。該網站每天會大概在晚上23點左右更新一次對應規則。

當然此處最自動化的方法是使用圖像識別對Unicode和數字進行對應，但這樣未免太過複雜，畢竟我們只需要找到10個對應關係。（其實是因為圖像識別太複雜我懶啦）

選擇中文轉Unicode就可以看到每個數字對應的Unicode碼了，將它們寫入轉換字典後換成數字即可。

import reimport requestsfrom bs4 import BeautifulSoupfrom lxml import etreekv = {user-agent:Mozilla/5.0 (Windows NT 6.0) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.120 Safari/535.2 ,"Referer":"https://www.shixiseng.com"}#此處是隨便找了個網頁哈r = requests.get(https://www.shixiseng.com/intern/inn_lp7s67llfyoh, headers=kv)soup = BeautifulSoup(r.content, html.parser, from_encoding="utf-8")text = soup.prettify()replace_dict = {"ue02b" : "0", "ue232" : "1", "uecf3" : "2", "ue553" : "3", "uf005" : "4", "ue4ec" : "5", "uef28" : "6", "uf386" : "7", "uf78a" : "8", "ue8e8" : "9" }for key, value in replace_dict.items(): text = txt.replace(key, value)

替換成功。

爬取職位信息

# -*- coding: utf-8 -*-"""Created on Tue Apr 3 19:54:46 2018@author: Jun Gao"""import reimport requestsfrom bs4 import BeautifulSoupimport numpy as npimport pandas as pdimport oskv = {user-agent:Mozilla/5.0 (Windows NT 6.0) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.120 Safari/535.2 ,"Referer":"https://www.shixiseng.com"}search_keyword = "數據"replace_dict = {"ue471" : "0", "uf530" : "1", "ue79d" : "2", "ue1ca" : "3", "uef99" : "4", "uf77e" : "5", "uf3d5" : "6", "uf8f2" : "7", "uf30e" : "8", "ue9a3" : "9"}result = np.array([[job_name, job_benefit, job_detail, salary, day_per_week, address]]) for page in np.arange(0,5): url1 = "https://www.shixiseng.com/interns/c-110100_?k={}&p={}".format(search_keyword, page) r = requests.get(url1, headers = kv) if r.status_code == 200: print("第{}頁索引頁抓取成功".format(page)) else: print("第{}頁索引頁抓取失敗".format(page)) break soup = BeautifulSoup(r.content, html.parser) text = soup.prettify() url2_behind_list = re.findall(r"intern_uuid": "(.*?)", text) for index,i in enumerate(url2_behind_list): url2 = "https://www.shixiseng.com/intern/" + i r = requests.get(url2, headers = kv) if r.status_code == 200: print("第{}頁-{}-{}抓取成功".format(page, index, i)) else: print("第{}頁-{}-{}抓取失敗".format(page, index, i)) break soup = BeautifulSoup(r.content, "html.parser") text = soup.prettify() for key, value in replace_dict.items(): text = text.replace(key, value) job_name = re.findall(r<div class="new_job_name" title="(.*?)">, text)[0].replace( ,).replace(
,).replace( ,) job_benefit = re.findall(r職位誘惑：(.*?)
, text)[0].replace( ,).replace(
,).replace( ,) job_detail = re.findall(r<div class="job_detail">([sS]*?)<div class="job_til">,text)[0].replace( ,).replace(
,).replace( ,) job_detail = re.sub(r<[sS]*?>, "", job_detail) salary = re.findall(r(.*?)／天, text)[0].strip() day_per_week = re.findall(r(.?)天／周, text)[0].strip() address = re.findall(r<span class="com_position">([sS]*?)</span>, text)[0].strip() print("第{}頁-{}-{}正在寫入".format(page, index, i)) data = [job_name, job_benefit, job_detail, salary, day_per_week, address] result = np.row_stack((result, data))

高頻詞可視化

下面我們實現功能——根據搜索「數據」獲得的前5頁結果職位的相關信息，對在任職要求中出現頻次高的關鍵詞做可視化。

首先安裝輔助工具，詞雲圖生成庫wordcloud，可以在CMD中輸入

pip install wordcloud

此處很可能出現一個問題就是提示安裝wordcloud由於缺乏VS組件而失敗，可以直接到Python Extension Packages for Windows下載whl文件，這樣就不用使用VS編譯，可以直接安裝成功了。

from wordcloud import WordCloud, ImageColorGeneratorimport matplotlib.pyplot as plttext = result[1:,2]words = "".join(str(i) for i in text)words = words.lower()words = re.findall(r[a-zA-Z]+,words)words = /.join(str(i) for i in words)wc = WordCloud(background_color=black, max_font_size=100, min_font_size=10, height=500, width_=500)wc.generate(words)wc.to_file(r"C:UsersAlbus GaoDesktopword_cloud.png")

運行代碼即可獲得下圖。

這裡也可以使用一些底圖來做出詞雲的效果，由於我們這裡詞數量有限就不實操了。有興趣的旁友請看這兩篇文章。第一篇是對各函數和參數的詳細講解，第二篇是WordCloud官方的使用指南。