爬取兩個知乎大V的關注者信息,保存至MySQL,並做簡單比較分析

04-21

需求:爬取知乎倆大V 甲大@路人甲和大頭死變態@王尼瑪的關注者信息

目的:通過簡單分析比較,看看隔著屏幕關注兩位的,究竟是牛鬼蛇神還是妖魔鬼怪,是女裝大佬還是大D萌妹,是東北老鐵還是華南老仙...

(話說,甲大爬別人爬了不少,不知道有沒有爬過自己的數據呢?)

手段:requests + cookies + 多進程 + 協程

前戲:知己知彼,百戰不殆.先去到知乎路人甲的關注者主頁 ,F12 接 F5 一氣呵成,不難找到一個followers?include= 開頭的數據,preview 一下,發現這是個json數據,data下有20個值

但是知乎耍了點壞,因為此時的信息並非當前頁關注者的信息,觀察請求鏈接最後部分,

果然,offset=20,一般來說offset應該是0開頭,所以這位叫"Boboan"的胖友應該就在第二頁的第一位,不信請看:

所以推斷是正確的.

篩選:我們需要的是關注者的昵稱(name),性別(gender),回答數(answer),文章數(articlescount),關注者數(follower_count),頭像鏈接(avatar_url),個性簽名(headline),url_token(url_token),住址,行業,學校,專業,職業住址,行業,學校,專業,職業

除後五個外,我們都可以在上面的json文件找到,後五個需通過url_token構造形如:

https://www.zhihu.com/api/v4/members/"url_token"?include=locations,business,employments,educations

的鏈接獲取

P.S.url_token可用於生成唯一的用戶主頁鏈接

運籌:我們大方向是,先以遊客的形式獲取前面部分的數據,後面五個數據需登錄後通過cookies訪問獲取.

具體思路是,requests構造請求獲取響應-->jsonpath解析響應,提取目標數據-->發送目標數據至協程-->由協程執行儲存數據至MySQL的操作

開爬:

我這裡是開了4個進程,其實可以適當增多提高抓取效率,同時我準備了4個許可權,每次都更換瀏覽器頭,每個進程爬取間隔為0.7-1.2s之間的隨機數企圖能起到一點反爬的作用

#!/usr/bin/env python# -*- coding: utf-8 -*-9#運用多進程爬去路人甲的關注者信息import timefrom random import uniformimport multiprocessingimport requestsimport jsonpathimport pymysql#我定製了一個有四個不同許可權的頭放在browser_head中import browser_head#用於往資料庫發送信息def insert_data(res): #用戶id,用作主鍵 ids = jsonpath.jsonpath(res,"$.data.*.id") #用戶昵稱 name = jsonpath.jsonpath(res,"$.data.*.name") #用戶頭像鏈接 avatar_url = jsonpath.jsonpath(res,"$.data.*.avatar_url") #用戶性別,1為男,0為女,-1為未知 gender = jsonpath.jsonpath(res,"$..gender") #用戶個簽 headline = jsonpath.jsonpath(res,"$..headline") #用戶被關注人數 follower = jsonpath.jsonpath(res,"$..follower_count") #用戶回答數 answer = jsonpath.jsonpath(res,"$..answer_count") #用戶文章數 articles = jsonpath.jsonpath(res,"$..articles_count") #用戶定製域名欄位 url_token = jsonpath.jsonpath(res,"$..url_token") #用戶首頁 user_url = [] for n in range(len(url_token)): user_url.append("https://www.zhihu.com/people/{}/activities".format(url_token[n])) data_list = [] for n in range(len(ids)): data_list.append( { id:ids[n], name:name[n], avatar_url:avatar_url[n], gender:str(gender[n]), headline:headline[n], follower:follower[n], answer_count:answer[n], articles:articles[n], url_token:url_token[n], user_url:user_url[n] } ) c = insert_one_page_data() next(c) c.send(data_list) return#獲取一頁內容,生產者def get_one_page_info(page,header): #定義一個計數君 jsj = 0 while 1: #如果頁數超出最大值,終止函數 if (page + 4) > 31628: print("進程%s完成!已退出!" %multiprocessing.current_process().name) return offset = page * 20 base_url = https://www.zhihu.com/api/v4/members/sgai/followers? param = { include:data[*].answer_count,articles_count,gender,follower_count,is_followed,is_following,badge[?(type=best_answerer)].topics, offset:offset, limit:20 } try: time.sleep(uniform(0.7,1.2)) res = requests.get(base_url,headers=header,params=param,timeout=7).json() except: jsj += 1 #ip列表不為空,則繼續當前循環 if jsj >5: print("連續5次請求錯誤,請排除錯誤後再試!") path = "D:\lurenjia_log\fail.txt" with open(path,"a") as f: f.write("offset:{}錯誤,請另行獲取.".format(offset)) #嘗試5次依然失敗後,記錄下當前錯誤的offset,往下進行 page += 4 continue #對offset進行反向運算,繼續循環 else: print("第%s頁解析錯誤!正在重試..."%page) time.sleep(jsj) offset = offset / 20 continue #如果一切順利,則把數據交給協程進行入庫,繼續當前循環 else: jsj = 0 insert_data(res) path = "D:\lurenjia_log\{}.txt".format(multiprocessing.current_process().name) output = "第{}頁已完成! ".format(page) with open(path,"a") as f: f.write(output) page += 4爬取數據前,我已經創建好一個lurenjia的資料庫,並用以下代碼在此資料庫下創建了table_1~table_4的表格但事實證明url_token欄位設計不周,因為沒想到有的url_token能那麼長...創建表格for n in range(1,5): db = pymysql.connect(host=localhost, user=root, password=******, port=3306, db=lurenjia) cursor = db.cursor() sql = "CREATE TABLE IF NOT EXISTS table_{}(id CHAR(100) NOT NULL PRIMARY KEY,name CHAR(20) NOT NULL,avatar_url CHAR(255), gender CHAR(2),headline VARCHAR(255),follower CHAR(10),answer_count CHAR(10),articles CHAR(10),url_token CHAR(40),user_url VARCHAR(255))DEFAULT CHARSET=utf8".format(n) cursor.execute(sql) db.close()#儲存一頁內容,消費者def insert_one_page_data(): while 1: data_list = yield if not data_list : break n = multiprocessing.current_process().name db = pymysql.connect(host=localhost, user=root, password=******, port=3306, db=lurenjia,charset=utf8) cursor = db.cursor() keys = ,.join(data_list[0].keys()) values = ,.join([%s] * len(data_list[0])) for data in data_list: sql = "INSERT INTO datou_{n} ({keys}) VALUES ({values})".format(n=n, keys=keys, values=values) try: if cursor.execute(sql,tuple(data.values())): db.commit() except: db.rollback() print("insert data unsuccessfully!") print("insert data successfully from Process %s !"%multiprocessing.current_process().name) db.close()if __name__ == "__main__": start = time.time() header_list = browser_head.header_list proccess_list = [] for n in range(4): p = multiprocessing.Process(name=str(n+1),target=get_one_page_info,args=(n,header_list[n])) proccess_list.append(p) for p in proccess_list: p.daemon = True p.start() for p in proccess_list: p.join() end = time.time() path = "D:\lurenjia_log\takentime.txt" with open(path, "w") as f: f.write("大吉大利,任務完成,共耗時{}s!".format(end-start))#關注該用戶的url,此處不實現#follow_url = "https://www.zhihu.com/api/v4/members/{}/followers".format(url_token)

小睡一蛤,那麼我們應該就能得到table_1~table_4這四個表格,在資料庫上把四個表格合併成一個,通過添加新的列num並設置主鍵遞增,再把重複的數據去除,就得到一個新的完全表格table_all:

看圖可知,共218頁,每頁1000條數據,共217456條數據!應該是沒什麼問題的了.

但是還有住址,行業,學校,專業,職業這五項信息.

關於這個API實在是不好找,首先你要登陸,然後到每個用戶的主頁去分析,有時還不一定能順利找到,我就試過今天找到,但沒留意那個就是,然後第二天就找不到了,後來看了 @Deserts X 的文章才恍然大悟,但也恍如隔世,因為我再也找不到那個被他成為薛定諤的API了.但我知道它長這樣:路人甲的詳細信息

我們發現它非常的...長,也很詳細,但我們需要的是住址,行業,學校,專業,職業這五個而已,所以可以縮寫成這樣,以下是根據url_token爬取每個用戶詳細信息的代碼:

#!/usr/bin/env python# -*- coding: utf-8 -*-###知乎用戶詳細信息import randomimport multiprocessingfrom time import time,sleep,strftimefrom multiprocessing import Processimport requestsimport pymysqlfrom jsonpath import jsonpath#我的頭信息和cookies都封裝在broswer_head中from browser_head import header_list,cookie_listdef get_url_token(n): ##從資料庫獲取1000個url_token,生產者 #:param n: n是進程名數字量-1的值,比如第3個進程,那麼n就是2 #獲取當前進程名稱 pn = multiprocessing.current_process().name n = 1 + 1000 * n db = pymysql.connect(host=localhost, user=root, password=******, port=3306, db=lurenjia,charset=utf8) cursor = db.cursor() while 1: if n // 1000 > 218: db.close() with open("D:\lurenjia_log\run_log.txt","a") as f: f.write("Process %s was done! --%s " %(pn,strftime("%Y-%m-%d %H:%M:%S"))) return else: #從資料庫中獲取第n到第n+1000個url_token sql = "select url_token from table_all where num >= {} and num <= {}".format(n,n+1000) cursor.execute(sql) url_token_list = cursor.fetchall() c = paser_urls() next(c) c.send(url_token_list) #由於有16個進程,每個進程獲取1000個,所以同一個進程第二次獲取的偏移量是16000 n += 16000def paser_urls(): while 1: pn = multiprocessing.current_process().name url_token_list = yield data_list = [] for item in url_token_list: url = "https://www.zhihu.com/api/v4/members/{}?".format(item[0]) param = {"include":"locations,business,employments,educations"} jsj = 0 while 1: try: sleep(random.uniform(0.50, 0.90)) res = requests.get(url,headers=header_list[0],cookies=cookie_list[0],params=param).json() except: if jsj > 5: with open("D:\lurenjia_log\parse_error.txt","a") as f: f.write("link of url_token {} error! --{} ".format(item[0],strftime("%Y-%m-%d %H:%M:%S"))) jsj = 0 break else: jsj += 1 sleep(2) continue else: print("parse url_token : %s successfully! inserting... from Process %s" %(item[0], pn)) #地址 address = jsonpath(res,"$.locations..name") #行業 business = jsonpath(res,"$.business.name") #學校 educations = jsonpath(res,"$.educations..school.name") #專業 major = jsonpath(res,"$.educations..major.name") #職業 employments = jsonpath(res,"$.employments..job.name") data = {} one_data_list = [address,business,educations,major,employments] name_list = ["address","business","educations","major","employments"] data["url_token"] = item[0] for n in range(5): if one_data_list[n]: data[name_list[n]] = one_data_list[n][0] else: data[name_list[n]] = 未知 data_list.append(data) break c = insert_one_page_data() next(c) c.send(data_list)同樣地,我事先在lurenjia資料庫用以下代碼創建了else_1~else_16的表格:db = pymysql.connect(host=localhost, user=root, password=******, port=3306, db=lurenjia)cursor = db.cursor()for n in range(1,17): sql = "CREATE TABLE IF NOT EXISTS else_{}(url_token VARCHAR(40),address VARCHAR(40),business VARCHAR(40), educations VARCHAR(40),major VARCHAR(40),employments VARCHAR(40))DEFAULT CHARSET=utf8".format(n) cursor.execute(sql) db.commit()db.close()#儲存一頁內容,消費者def insert_one_page_data(): n = multiprocessing.current_process().name while 1: data_list = yield if not data_list: break db = pymysql.connect(host=localhost, user=root, password=*******, port=3306, db=lurenjia,charset=utf8) cursor = db.cursor() keys = ,.join(data_list[0].keys()) values = ,.join([%s] * len(data_list[0])) for data in data_list: sql = "INSERT INTO else_{n} ({keys}) VALUES ({values})".format(n=n, keys=keys, values=values) try: if cursor.execute(sql,tuple(data.values())): db.commit() except: db.rollback() print("insert data unsuccessfully!") with open("D:\lurenjia_log\run_log.txt","a") as f: f.write("insert data %s error! --%s " %(data,strftime("%Y-%m-%d %H:%M:%S"))) with open("D:\lurenjia_log\insert_log.txt", "a") as f: f.write("insert data sucsessfully! from Process %s--%s " %(n, strftime("%Y-%m-%d %H:%M:%S"))) db.close()def run(): start = time() process_list = [] for n in range(16): #這裡我開了16個進程,每個進程名分別就叫1~16 p = Process(name=str(n+1),target=get_url_token,args=(n,)) process_list.append(p) for p in process_list: p.daemon = True p.start() for p in process_list: p.join() run_time = time() - start with open("D:\lurenjia_log\parse_error.txt","a") as f: f.write("大吉大利!任務完成!總共耗時{}s! --{} ".format(run_time,strftime("%Y-%m-%d %H:%M:%S")))if __name__ == "__main__": run()

如無意外,我們得到了else_1~else_16的表格,我們先把這16個表格合併成一個else_all.