python生成器處理大文本文件的代碼

運維經常要讀寫大文本文件，幾個G用生成器讀寫，速度很快。

幾十G的先切分為N個小文件，再處理。

# coding:utf-8n"""n黃哥python遠程視頻培訓班nhttps://github.com/pythonpeixun/article/blob/master/index.mdnn黃哥python培訓試看視頻播放地址nhttps://github.com/pythonpeixun/article/blob/master/python_shiping.mdn"""nimport timenstart_time = time.time()ndef find_ip(path):n # urllist = []nn for line in open(path):n s = line.find("Sogou web spider)n if s >=0 :n yield line[:s].strip()nnp = find_ip("bigfile.txt")np = list(set(list(p)))nfor item in p:n print itemnnprint time.time() - start_time, "seconds"n

推薦閱讀：