學習筆記（Udacity）- 簡單的車道識別

github.com

2 代碼環境

我是通過下載anaconda的jupyter notebook來做的。具體步驟請參考網上的安裝資料。（之後可能會更新一下如何安裝tensorflow和opencv2，安裝過程其實我認為是比較痛苦的）

3 project 目的

其實project的目的很單純，就是給演算法一段視頻輸入，然後由構建的演算法通過計算，最後輸出車道兩邊的線（當然是通過x，y坐標輸出）的圖片或者視頻。

4 涉及到的知識點

• opencv庫的用法（參考官網）

cv2.inRange() #for color selectioncv2.fillPoly() #for regions selectioncv2.line()#to draw lines on an image given endpointscv2.addWeighted()#to coadd / overlay two imagescv2.cvtColor()to grayscale or change colorcv2.imwrite()to output images to filecv2.bitwise_and()#to apply a mask to an image

• RGB的基本理解
• 圖像中的xy坐標理解
• 灰度圖
• ROI（region of interest）
• 邊緣檢測
• hough transfrom（檢測直線演算法）
• 理解好各個operator的意義
• 理解x<threshold && x> threshold 這種代碼的意義
• 雖然Udacity給出的jupyter notebook的template也涉及到HTML庫，但是我們只需要知道簡單的用法就可以了，不用太過在意

5 代碼解析

5.1 import 庫

首先最開始的是，import 庫。 庫的import跟c語言的include是一個意思。就是把我們所需要的所有的函數包拿來備用。這裡我們import了一下庫。matplotlib，numpy其實都很好安裝及import，但是吧這個cv2真得是特別難裝。。。。尤其是以後基於tensorlfow GPU版本安裝cv2更費勁。。又跑題了。

#importing some useful packagesimport matplotlib.pyplot as pltimport matplotlib.image as mpimgimport numpy as npimport cv2%matplotlib inline

這裡%matplotlib inline是一種jupyter notebook的特別的用法。叫magic mthods。這是幹啥的呢？ matplot 他本身默認是不會在jupyter notebook代碼cell之間打開plot的。所以%matplotlib inline 就是命令matplot在cell之間打開plot。

import 。。。as 。。 就是把特定函數包單獨按照我們想要的簡稱命名的。

Python中的代碼包是按照a.b.c這種方式來的。也就說a的函數包裡面包含b的函數包。b的函數包里包含c的函數包。嗯。就是所有語言中過的class的那種結構。

5.2 讀取圖片

為什麼項目目的是輸入視頻，但是我們讀取的卻是圖片呢？ 這裡需要解釋一下，所有的視頻都是連續的圖片。FPS是指 frame per second ，也就是說一分鐘播放幾個圖片。如果FPS30的話，就是一分鐘播放30個圖片。所以，處理圖片和處理視頻基本上是一個事情。其實就是我們的演算法1秒鐘處理30個圖片，然後通過其他代碼把這個再挨個播放或者合成成一個視頻就好了。

這裡用到了mpimg。值得注意的是，mpimg可以讀取圖片，cv2也可以的。但是他們讀取圖片之後的數據存儲序列是不一樣的。mpimg讀取的是，RGB順序的數據。而cv2讀取的是BGR順序的數據。數據本身並沒有什麼變化，除了順序。RGB指的是 red，green，blue。 那麼BGR指的是Blue， Green，Red。一般這三種數據類型稱為三個channel。為啥是RGB這三種顏色？ 因為他們是三原色。他們三個通過不同的組合，得到所有的顏色。

#reading in an imageimage = mpimg.imread(test_images/solidWhiteRight.jpg)#printing out some stats and plottingprint(This image is:, type(image), with dimensions:, image.shape)plt.imshow(image) #call as plt.imshow(gray, cmap=gray) to show a grayscaled image

5.4 構建helper functions

什麼是helper functions？

其實就是一堆函數。用處是簡化代碼。

內容如下

• 灰度圖轉化。把彩色的轉換成黑白的。這裡需要注意的是，黑白並不是0或者1，而是0~255.因為黑白的還有灰色等顏色。數字越大，圖像越白。也就是說，0就是完全黑的，125是灰的，255就是完全白的。所以他叫灰度圖，而不是黑白圖。關鍵詞：彩色轉換為黑白
• canny operator 用於邊緣檢測 。關鍵詞：邊緣檢測
• 高斯blur（給pixel添加noise的部分）。關鍵詞：添加noise
• region of interest 簡稱ROI ，顧名思義就是我們只考慮及只計算我們關心範圍內的東西。關鍵詞：關心領域
• draw lines 就是給定兩個點的坐標（x1，x2，y1，y2）在圖片上畫出一條線。關鍵詞：畫線
• weighted img 的作用就是為了可視化。我們最終算出來的車道線，所以要把標記好的車道線覆蓋到原理的圖片上。關鍵詞：覆蓋

import mathdef grayscale(img): """Applies the Grayscale transform This will return an image with only one color channel but NOTE: to see the returned image as grayscale you should call plt.imshow(gray, cmap=gray)""" return cv2.cvtColor(img, cv2.COLOR_RGB2GRAY) # Or use BGR2GRAY if you read an image with cv2.imread() # return cv2.cvtColor(img, cv2.COLOR_BGR2GRAY) def canny(img, low_threshold, high_threshold): """Applies the Canny transform""" return cv2.Canny(img, low_threshold, high_threshold)def gaussian_blur(img, kernel_size): """Applies a Gaussian Noise kernel""" return cv2.GaussianBlur(img, (kernel_size, kernel_size), 0)def region_of_interest(img, vertices): """ Applies an image mask. Only keeps the region of the image defined by the polygon formed from `vertices`. The rest of the image is set to black. """ #defining a blank mask to start with mask = np.zeros_like(img) #defining a 3 channel or 1 channel color to fill the mask with depending on the input image if len(img.shape) > 2: channel_count = img.shape[2] # i.e. 3 or 4 depending on your image ignore_mask_color = (255,) * channel_count else: ignore_mask_color = 255 #filling pixels inside the polygon defined by "vertices" with the fill color cv2.fillPoly(mask, vertices, ignore_mask_color) #returning the image only where mask pixels are nonzero masked_image = cv2.bitwise_and(img, mask) return masked_imagedef draw_lines(img, lines, color=[255, 0, 0], thickness=10): """ NOTE: this is the function you might want to use as a starting point once you want to average/extrapolate the line segments you detect to map out the full extent of the lane (going from the result shown in raw-lines-example.mp4 to that shown in P1_example.mp4). Think about things like separating line segments by their slope ((y2-y1)/(x2-x1)) to decide which segments are part of the left line vs. the right line. Then, you can average the position of each of the lines and extrapolate to the top and bottom of the lane. This function draws `lines` with `color` and `thickness`. Lines are drawn on the image inplace (mutates the image). If you want to make the lines semi-transparent, think about combining this function with the weighted_img() function below """ for line in lines: for x1,y1,x2,y2 in line: cv2.line(img, (x1, y1), (x2, y2), color, thickness)def hough_lines(img, rho, theta, threshold, min_line_len, max_line_gap): """ `img` should be the output of a Canny transform. Returns an image with hough lines drawn. """ lines = cv2.HoughLinesP(img, rho, theta, threshold, np.array([]), minLineLength=min_line_len, maxLineGap=max_line_gap) line_img = np.zeros((*img.shape, 3), dtype=np.uint8) draw_lines(line_img, lines) return line_img# Python 3 has support for cool math symbols.def weighted_img(img, initial_img, α=0.8, β=1., λ=0.): """ `img` is the output of the hough_lines(), An image with lines drawn on it. Should be a blank image (all black) with lines drawn on it. `initial_img` should be the image before any processing. The result image is computed as follows: initial_img * α + img * β + λ NOTE: initial_img and img must be the same shape! """ return cv2.addWeighted(initial_img, α, img, β, λ)#Removing noise slopes from the averaging performed below in lane_linesdef remove_noise(slopes, m = 2): mean_value = np.mean(slopes) stand_deviation = np.std(slopes) for slope in slopes: if abs(slope - mean_value) > (m * stand_deviation): slopes.remove(slope) return slopes

5.5 主程序

下面就是主程序process image（輸入）。

這個函數裡面所有代碼都是要在1/FPS的時間裡完成的。說實話，因為這個時候是第一次入門cv，所以我想原來計算機可以在這麼短的時間裡處理這麼多的事情。只有在做更複雜的cv代碼的的時候，我才知道，原來計算機可以處理更多。。。不過計算機視覺的處理時間確實是計算機視覺應用在無人駕駛上的障礙。

def process_image(img): #find the size of image xsize,ysize = [image.shape[1],image.shape[0]] #copy the image to modify origin_image = np.copy(img) #make a gray image gray = grayscale(img) #Smooth with guassian blur kernel_size = 9 blur_gray = gaussian_blur(gray, kernel_size) #Use canny operator to extract edeges low_threshold = 90 high_threshold = 180 edges = canny(blur_gray, 90,180) #define the region of the interest imshape = image.shape vertices = np.array([[(0,imshape[0]),(470, 320), (550, 320), (imshape[1],imshape[0])]], dtype=np.int32) masked_edges = region_of_interest(edges, vertices) # Define the Hough transform parameters # Make a blank the same size as our image to draw on rho = 6 # distance resolution in pixels of the Hough grid theta = np.pi/180 # angular resolution in radians of the Hough grid threshold = 50 # minimum number of votes (intersections in Hough grid cell) min_line_len = 25 #minimum number of pixels making up a line max_line_gap = 25 # maximum gap in pixels between connectable line segments line_image = np.copy(img)*0 # creating a blank to draw lines on # Run Hough on edge detected image # Output "lines" is an array containing endpoints [x1,y1,x2,y2] of detected line segments lines = cv2.HoughLinesP(masked_edges, rho, theta, threshold, np.array([]), min_line_len, max_line_gap) #Make lists of the lines and slopes for averaging left_lines = [] left_slopes = [] right_slopes = [] right_lines = [] for line in lines: for x1,y1,x2,y2 in line: slope = (y2 - y1) / (x2-x1) if slope < 0: left_lines.append(line) left_slopes.append(slope) else: right_lines.append(line) right_slopes.append(slope) #Average line positions,zip function can generate the all the column elements in a list . * stands for unpacked lists mean_left_pos = [sum(column)/len(column) for column in zip(*left_lines)] mean_right_pos = [sum(column)/len(column) for column in zip(*right_lines)] #Remove slope outliers, and take the average mean_left_slope = np.mean(remove_noise(left_slopes)) mean_right_slope = np.mean(remove_noise(right_slopes)) #Extrapolate to our mask boundaries - up to 325, down to 539 #Exrapoplate the left line, right line to boundary up to y_top = 320, down to y_bottom = 540 mean_left_line = [] mean_right_line = [] for x1,y1,x2,y2 in mean_left_pos: x = int(np.mean([x1, x2])) #Midpoint x y = int(np.mean([y1, y2])) #Midpoint y slope = mean_left_slope #base on y = mx + b calculate the b = y-mx b = y -(slope * x) #Solving y=mx+b for b mean_left_line = [int((320-b) / slope), 320, int((540-b)/slope), 540] for x1,y1,x2,y2 in mean_right_pos: x = int(np.mean([x1, x2])) y = int(np.mean([y1, y2])) slope = mean_right_slope b = y - (slope * x) mean_right_line = [int((320-b)/slope), 320, int((540 - b)/slope), 540] #The final lines of the lane lines = [[mean_left_line], [mean_right_line]] #Draw the lines to the line_image draw_lines(line_image, lines) # Transparent the processed lines image to original weighted_image = weighted_img(line_image, img) #return the weighted_image to the fucntion process_image return weighted_image

亂七八糟的，看不懂。是的。所以需要拆分重要的部分進行解釋。（沒涉及的內容多看幾遍代碼就可以知道了。如果還是不懂可以留言給我）

下面一行代碼是為了獲取圖片的x，y方向的個數。這樣我們才能通過指定位置來定位那個像素點並編輯那個像素點

xsize,ysize = [image.shape[1],image.shape[0]]

下面這行代碼利用我們在helper function裡面定義的函數，將彩色圖變成灰度圖。

#make a gray image gray = grayscale(img)

也是利用helper function的函數，給像素點添加雜訊。其實可以想像一下，添加雜訊的圖片會變得怎麼樣？ （變得模糊）這裡kernal_size就是人為設定的值。我是憑感覺設定的。

#Smooth with guassian blur kernel_size = 9 blur_gray = gaussian_blur(gray, kernel_size)

下面是利用canny operator提取邊緣的。為什麼要檢測邊緣？ 因為理解物體的邊緣是我們識別物體的最基本的方法。計算機視覺也是一樣的。canny 其實就是利用特定的operator，也就是一種3*3的矩陣，通過卷積對圖片上的每一個點進行計算。計算後值如果在我們定義的low_threshold和high_threshold之間， 那麼我們就認為他是有效的邊緣點。可以用來識別物體。所以low_threshold和high_threshold也是認為調的。調的效果好，那麼就是好的。

#Use canny operator to extract edeges low_threshold = 90 high_threshold = 180 edges = canny(blur_gray, 90,180)

下面代碼是用來定義ROI的。對於無人車來說，他不需要關注整個攝像機拍到的所有的東西。無人車只需要關心自己前面的道路及自己周圍的目標就可以了。所以通過設定ROI區域，來減少計算量。設定ROI是通過給定vertices（就是頂點）的坐標，讓演算法排除一切在ROI區域之外的像素點。

換成大白話，就是我只關心我關心的，別人愛咋咋的。

#define the region of the interest imshape = image.shape vertices = np.array([[(0,imshape[0]),(470, 320), (550, 320), (imshape[1],imshape[0])]], dtype=np.int32) masked_edges = region_of_interest(edges, vertices)

下面是hough 變換的代碼。邊緣點，ROI已經定義好了，那麼我們就要在圖片上找找車道了。車道是直的，人類一眼就能看出來。但是想沒想過人類是如何判斷直的呢？計算機視覺的演算法又應該怎麼落實呢？人類是通過透視和車道大部分是直的這種假設來判斷的。那對與計算機視覺也是一樣。計算機需要找到圖片里有一定規律的點，然後把他們都標記出來。how？ 所有有類似的斜率且像素間的距離不大的兩點，認為其是直線。hough transfrom就是做這個事情的。 xy坐標系裡的直線在hough space里，可以用一個點來表示。如下面的圖片。其實就是把y=mx+b用斜率和截距來表示。那麼在hough space裡面，聚集在一定範圍內的點們就是一條直線。這個一定範圍就是用rho來表示，theta就是指hough space里點構成的直線的斜率。有點說不明白，建議在網上找個動圖或者看看這個鏈接第三章 霍夫變換(Hough Transform)（作者看到了如果覺得不妥可以告訴我）

# Define the Hough transform parameters # Make a blank the same size as our image to draw on rho = 6 # distance resolution in pixels of the Hough grid theta = np.pi/180 # angular resolution in radians of the Hough grid threshold = 50 # minimum number of votes (intersections in Hough grid cell) min_line_len = 25 #minimum number of pixels making up a line max_line_gap = 25 # maximum gap in pixels between connectable line segments line_image = np.copy(img)*0 # creating a blank to draw lines on # Run Hough on edge detected image # Output "lines" is an array containing endpoints [x1,y1,x2,y2] of detected line segments lines = cv2.HoughLinesP(masked_edges, rho, theta, threshold, np.array([]), min_line_len, max_line_gap)

剩下的就簡單了。因為有ROI我們只會看到車道裡面的直線。那麼有很多小的直線，但是都是不連續怎麼辦？ 我們先通過挨個定義左右兩邊的直線們來計算出連續的直線。

先通過各個直線（通過點表示）的斜率分成兩個部分。左右兩邊。

#Make lists of the lines and slopes for averaging left_lines = [] left_slopes = [] right_slopes = [] right_lines = [] for line in lines: for x1,y1,x2,y2 in line: slope = (y2 - y1) / (x2-x1) if slope < 0: left_lines.append(line) left_slopes.append(slope) else: right_lines.append(line) right_slopes.append(slope)

最終左右兩邊的斜率通過取平均確定下來，然後通過這個確定的斜率，在圖像的坐標軸系裡求出相應的直線。（斜率確定下來了，ROI也給出了x，y的最大值，所以可以得到相應的xy頂點）

#Average line positions,zip function can generate the all the column elements in a list . * stands for unpacked lists mean_left_pos = [sum(column)/len(column) for column in zip(*left_lines)] mean_right_pos = [sum(column)/len(column) for column in zip(*right_lines)] #Remove slope outliers, and take the average mean_left_slope = np.mean(remove_noise(left_slopes)) mean_right_slope = np.mean(remove_noise(right_slopes)) #Extrapolate to our mask boundaries - up to 325, down to 539 #Exrapoplate the left line, right line to boundary up to y_top = 320, down to y_bottom = 540 mean_left_line = [] mean_right_line = [] for x1,y1,x2,y2 in mean_left_pos: x = int(np.mean([x1, x2])) #Midpoint x y = int(np.mean([y1, y2])) #Midpoint y slope = mean_left_slope #base on y = mx + b calculate the b = y-mx b = y -(slope * x) #Solving y=mx+b for b mean_left_line = [int((320-b) / slope), 320, int((540-b)/slope), 540] for x1,y1,x2,y2 in mean_right_pos: x = int(np.mean([x1, x2])) y = int(np.mean([y1, y2])) slope = mean_right_slope b = y - (slope * x) mean_right_line = [int((320-b)/slope), 320, int((540 - b)/slope), 540] #The final lines of the lane lines = [[mean_left_line], [mean_right_line]] #Draw the lines to the line_image draw_lines(line_image, lines) # Transparent the processed lines image to original weighted_image = weighted_img(line_image, img) #return the weighted_image to the fucntion process_image

得到直線的坐標後，通過draw_lines的函數，我們就可以在原圖上覆蓋車道識別結果了。

這樣所有的代碼就結束了。

6 結果

最終就可以得到如下的結果。（圖片）

通過其他編碼器，就可以把所有的處理過的圖片合成最終得到有車道標記後的視頻文件。

7 總結

本次項目通過邊緣檢測，霍普變換實現簡單的車道識別。但是實際應用中，根據環境不同，車道識別的識別率也會很不同。比如顏色啊，標記模糊啊等等。在之後的項目中，我們會利用更好的演算法得到更加穩定的輸出。

相關內容也會在未來更新的。

謝謝支持，各位看官的關注就是持續更新的動力~

看完就別吝嗇點贊加關注啦~

同時也希望朋友往咱們專欄投稿，讓我們在無人車演算法的造詣上不停的成長~！

2018.06.02 家裡 林明

推薦閱讀：