分享一個簡單的多進程小爬蟲

分享一個簡單的多進程小爬蟲，爬取某加盟創業網上所有加盟項目；

使用requests請求頁面，re和beautifulSoup解析網頁，multiprocessing進行多進程，pymongo數據入庫；

（打開網站時發現密密麻麻、琳琅滿目的加盟項目，但是爬完之後，發現數據也就只有那麼一點點，甚至不到一千條數據，這就有點囧了）

# coding:utf-8nn"""n auther:zmister.comn"""nnimport requestsnfrom bs4 import BeautifulSoupnfrom multiprocessing import Poolnimport renimport pymongonn# 設置MongoDB連接信息nclient = pymongo.MongoClient(localhost,27017)ncn_78 = client[cn_78]nproject_info = cn_78[project_info]nn# 瀏覽器頭信息nheader = {n User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36,n Connection: keep-alive,n}nn# 獲取分類類目名字和鏈接ndef get_all_big_category():n url = http://www.78.cnn wbdata = requests.get(url,headers=header).contentn soup = BeautifulSoup(wbdata,lxml)n cates = soup.select(div.xmFl_left > ul > li > p > a)n cate_list = []n for c in cates:n cname = c.get_text()n curl = c.get("href")n cate_list.append(curl)n # print(cate_list)n return cate_listnn# 從分類頁面獲取具體項目信息ndef get_project_info(cates):n url = http://www.78.cn+catesn wbdata = requests.get(url,headers=header).contentn soup = BeautifulSoup(wbdata,lxml)n cate = soup.select(div.dhy_cont3 > dl.cont3_top > dt > span)[0].get_text()n subcate = soup.select(div.dhy_cont3 > dl.cont3_top > dd.check > a)[0].get_text()n p_name = re.findall(rtarget="_blank">(.*?)</a></h2>,str(soup))n p_mony = re.findall(r<dt>投資：<em class="red">￥<strong>(.*?)</strong>,str(soup))n p_aear = re.findall(r<dd>所屬區域：(.*?)</dd>,str(soup))n p_time = re.findall(r<dd>累計時間：(.*?)</dd>,str(soup))n for name,mony,aear,times in zip(p_name,p_mony,p_aear,p_time):n data = {n cate:cate,n subcate:subcate,n name:name,n mony:mony,n aear:aear,n time:times,n minmoney:mony.replace(萬,).split(-)[0] if - in mony else None,n maxmoney:mony.replace(萬,).split(-)[-1] if - in mony else mony[0]n }n print(data)n project_info.insert_one(data)n print("插入數據成功")nnif __name__ == __main__:n pool = Pool(processes=4)n pool.map_async(get_project_info,get_all_big_category())n pool.close()n pool.join()n

不知對各位有沒有幫助？

推薦閱讀：