某測試模擬器性能優化-選對你的庫

01-27

另一個模擬器組件

前面好幾篇文章寫都是模擬器裡面的一個叫做log_generator的組件的調優過程，這次我們講講另外一個組件client_simulator。

client_simulator這個組件內在邏輯非常簡單，希望模擬大量的客戶端，同時按照一定的頻率向一台HTTP伺服器請求編碼過的chunk數據，這個一定的頻率指的是，每秒發送10次請求，每次請求一個不斷變化的url。編寫client_simulator這個組件的目的是驗證該HTTP伺服器的內存置換策略的有效性。

問題描述

組裡負責該組件的同事，寫完Python代碼後上機一跑，性能不忍直視，大致只能模擬100個客戶端。要知道，這個結果可是在一台8核4G的虛擬機上跑的啊，總共也就打出了130M左右的帶寬，離我們的目標700節點，差距太大。

重構和分析

看了下該組件的代碼，好嘛，又是坑爹的多線程。直接改成多進程+多線程的模式。進程和CPU數一致，每進程啟動線程數= 期望節點數/進程數。

重構完上機一跑，嗯，有所改善，CPU都吃的差不多滿了，但是整體性能提高不大，大概能模擬300個客戶端了。

這可咋辦，再看一次代碼？代碼很簡單，就是兩層循環，外層生成進程，內層生成線程，線程使用網路神庫requests請求數據。再沒有其他複雜邏輯了~

既然又是一個IO密集的程序，那麼自然系統資源都消耗在網路數據傳輸上了，難道要重構requests？我可沒這能力。

數據來幫忙

死馬當作活馬醫，再次上yappi分析，得到下面的結果：

Clock type: CPUnOrdered by: totaltime, descnnname ncall tsub ttot tavgn/usr/lib64/python2.7/threading.py:754 Thread.run 10 0.000704 14.011758 1.401176n/home/admin/comrade/comrade/faker.py:23 fake_sdk 10 0.124799 14.010940 1.401094n/usr/lib/python2.7/site-packages/requests/api.py:61 get 2070 0.045381 13.767441 0.006651n/usr/lib/python2.7/site-packages/requests/api.py:16 request 2070 0.041756 13.719202 0.006628n/usr/lib/python2.7/site-packages/requests/sessions.py:441 Session.request 2070 0.059906 11.679531 0.005642n/usr/lib/python2.7/site-packages/requests/sessions.py:589 Session.send 2070 0.089826 7.909129 0.003821n/usr/lib/python2.7/site-packages/requests/adapters.py:388 HTTPAdapter.send 2070 0.064437 6.610118 0.003193n/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:447 HTTPConnectionPool.urlopen 2070 0.085589 3.781414 0.001827n/usr/lib/python2.7/site-packages/requests/sessions.py:401 Session.prepare_request 2070 0.078741 3.018275 0.001458n/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:322 HTTPConnectionPool._make_request 2070 0.077418 2.430796 0.001174n/usr/lib/python2.7/site-packages/requests/models.py:299 PreparedRequest.prepare 2070 0.038006 1.528752 0.000739n/usr/lib/python2.7/site-packages/requests/adapters.py:290 HTTPAdapter.get_connection 2070 0.024181 1.526116 0.000737n/usr/lib/python2.7/site-packages/urllib3/poolmanager.py:266 PoolManager.connection_from_url 2070 0.017157 1.346354 0.000650n/usr/lib/python2.7/site-packages/urllib3/poolmanager.py:206 PoolManager.connection_from_host 2070 0.015797 1.236867 0.000598n/usr/lib/python2.7/site-packages/requests/structures.py:42 CaseInsensitiveDict.__init__ 12420 0.108715 1.226514 0.000099n/usr/lib/python2.7/site-packages/urllib3/poolmanager.py:229 PoolManager.connection_from_context 2070 0.018828 1.213145 0.000586n/usr/lib64/python2.7/httplib.py:1053 HTTPConnection.getresponse 2070 0.034232 1.161452 0.000561n/usr/lib/python2.7/site-packages/requests/sessions.py:340 Session.__init__ 2070 0.060470 1.106475 0.000535n/usr/lib/python2.7/site-packages/requests/sessions.py:50 merge_setting 14490 0.103324 1.102030 0.000076n/usr/lib/python2.7/site-packages/urllib3/poolmanager.py:242 PoolManager.connection_from_pool_key 2070 0.034751 1.082293 0.000523n/usr/lib64/python2.7/httplib.py:1015 HTTPConnection.request 2070 0.010323 1.073503 0.000519n/usr/lib64/python2.7/httplib.py:1036 HTTPConnection._send_request 2070 0.074046 1.063180 0.000514n/usr/lib64/python2.7/httplib.py:437 HTTPResponse.begin 2070 0.056034 1.041176 0.000503n/usr/lib64/python2.7/_abcoll.py:526 update 22770 0.277791 1.022363 0.000045n/usr/lib/python2.7/site-packages/requests/adapters.py:253 HTTPAdapter.build_response 2070 0.050085 1.008311 0.000487n/usr/lib/python2.7/site-packages/requests/sessions.py:398 Session.__exit__ 2070 0.007221 0.889536 0.000430n/usr/lib/python2.7/site-packages/requests/sessions.py:705 Session.close 2070 0.016573 0.882314 0.000426n/usr/lib/python2.7/site-packages/urllib3/poolmanager.py:170 PoolManager._new_pool 2070 0.038002 0.854147 0.000413n/usr/lib/python2.7/site-packages/requests/adapters.py:313 HTTPAdapter.close 4140 0.021252 0.841980 0.000203n/usr/lib/python2.7/site-packages/requests/models.py:810 Response.content 2070 0.023343 0.821442 0.000397n

果然和預期的一樣，最佔用CPU資源的是requests庫。咦，為何有那麼多sessions.py模塊的調用，我們的需求，明明只是簡單的HTTP GET請求。

替換大法好

啊，啊，啊，我們是不是殺雞用牛刀了，趕緊的，把requests換成urllib2吧。

替換完，上機一跑，性能杠杠滴，帶寬瞬間打滿。

如果覺得本文對您有幫助，敬請點贊。