數據集列歸一化與聚類示例

數據集列歸一化與聚類示例

file = "userid_data.csv";data0 = Import@file;{head, data} = {First@data0, Rest@data0};asso = GroupBy[data, First -> Rest, First];c = FindClusters[asso[[1 ;; 100]], 10];Length /@ c

{87,1,4,1,2,1,1,1,1,1}

data1 = AssociationThread[head -> #] & /@ data;dataset = Dataset@data1;

看數據

data[[All, 2]] // Histogram

data[[All, 3]] // Histogram

Log[data[[All, 2]]] // Histogram

Log[data[[All, 3]]] // Histogram

Dataset

fields = Rest@head

{paidordercnt,paidorderamt}

(minMax[#] = Normal@dataset[MinMax, #]) & /@ fields

{{1,2340},{0.,2.24879*10^6}}

minMaxManul = Association@{"paidordercnt" -> {1, 60}, "paidorderamt" -> {0, 20000}};minMaxRescale[x_, minMax_] := Block[{min, max},{min, max} = minMax;(*(最大值截斷)*)If[x > max, 1, N@((x - min)/(max - min))]](*(minMax[#]=Normal@dataset[[1;;10]][MinMax,#])&/@fields;*)dataFinal = Normal@dataset[All, {#@"userid", minMaxRescale[#@"paidordercnt", minMaxManul["paidordercnt"]],minMaxRescale[#@"paidorderamt", minMaxManul["paidorderamt"]]} &];

上面的運算元中輸入多次minMaxRescale，即對於每個變數分別處理，也沒什麼不好的, 因為不同的列可能會有不同的處理方式.

改進是輸入一個fields，自動處理，但是簡單嘗試了一下，發現有點小問題，原因是會遇到兩個不同層級的匿名變數[純函數] 的嵌套問題，有空再更新一下

(*dataset[All,minMaxNormalizeByRow@fields]*)assoFinal = GroupBy[dataFinal, First -> Rest, First];

加一個Log

log1p := Log[1 + #] &minMaxManulLog = Association@{"paidordercnt" -> {log1p@1, log1p@60}, "paidorderamt" -> {log1p@0, log1p@20000}};dataFinalLog = Normal@dataset[All, {#@"userid", minMaxRescale[#@"paidordercnt", minMaxManul["paidordercnt"]],minMaxRescale[log1p@#@"paidorderamt", minMaxManulLog["paidorderamt"]]} &];assoFinalLog = GroupBy[dataFinalLog, First -> Rest, First];minMaxManulLog = Association@{"paidordercnt" -> {log1p@1, log1p@60}, "paidorderamt" -> {log1p@0, log1p@20000}};dataFinalLog1 = Normal@dataset[All, {#@"userid", minMaxRescale[log1p@#@"paidordercnt", minMaxManulLog["paidordercnt"]],minMaxRescale[log1p@#@"paidorderamt", minMaxManulLog["paidorderamt"]]} &];assoFinalLog1 = GroupBy[dataFinalLog1, First -> Rest, First];

聚類

clusters = FindClusters[assoFinal, 10];Length /@ clusters

{6333,2571,2432,45451,18001,6269,8112,4364,2557,3910}

這個結果比不歸一化的時候要好，部分也反應了特徵的分布情況，當然並沒有達到預期的效果，聚類結果不夠均勻。

理想情況下我們可以用外部指標度量[比如當成一個分類問題時的標記的類別], 如果暫時沒有,則定義一些輔助指標人肉檢測效果

clustersLog = FindClusters[assoFinalLog, 10];Length /@ clustersLog

{8258,4414,7017,16650,7310,8568,18482,9776,11034,8491}

clustersLog1 = FindClusters[assoFinalLog1, 10];Length /@ clustersLog1

{7963,11085,11337,10080,12099,9810,11728,8371,9036,8491}

列相關性

按一個維度的分箱結果的類別去跟聚類的類別雲做比較,想像一個退化情況,只有一維特徵做聚類.

ListPlot[Values@KeyTake[assoFinal, #] & /@ clusters]

ListPlot[Values@KeyTake[assoFinalLog, #] & /@ clustersLog]

ListPlot[Values@KeyTake[assoFinalLog1, #] & /@ clustersLog1]

其他

對特徵進行分箱,平滑操作,聚類方法的選擇等,有空再更新新例子.

本來就沒幾個人看，喜歡的/有所幫助的，記得點個贊加個星@github

<< "/Users/hypergroups/Documents/Wolfram Mathematica/DeployProjects/MyMarkDown.m"Notebook2Markdown[EvaluationNotebook[], "title" -> "ColumnNormalize", "dirOutput" -> "/Users/hypergroups/Documents/Wolfram Mathematica/MyMarkdown/知乎專欄@Mathematica機器學習實戰/"]

推薦閱讀：