OpenCV在車道線查找中的使用

本篇是自動駕駛系列的第二篇，在後台留言索取代碼會提供源碼鏈接。這次的目標是編寫一個軟體流水線來識別汽車前置攝像頭的視頻中的車道邊界。攝像機標定圖像，試驗路圖像和視頻項目都可以在這裡儲存。

這次試驗的目標/步驟如下：

• 計算相機校準矩陣和給定一組棋盤圖像的失真係數。
• 對原始圖像應用畸變校正。
• 使用顏色變換，漸變等創建閾值二值圖像。
• 應用透視變換來糾正二值圖像（「鳥瞰」）。
• 檢測車道像素，找到車道邊界。
• 確定車道和車輛相對於中心的曲率。
• 將檢測到的車道邊界轉回到原始圖像上。
• 輸出車道邊界的視覺顯示和車道曲率和車輛位置的數值估計。

相機校準矩陣和失真係數

當照相機查看真實世界中的3D對象並將其轉換為2D圖像時，會發生圖像失真; 這個轉變並不完美。失真實際上改變了這些3D對象的形狀和大小。因此，分析相機圖像的第一步是消除這種失真，以便從中獲得正確和有用的信息。

真實的相機使用彎曲的鏡頭來形成圖像，而光線在這些鏡頭的邊緣往往會彎曲得太多或太少。這會產生扭曲圖像邊緣的效果，使線條或物體看起來或多或少比實際彎曲。這被稱為徑向失真，這是最常見的失真類型。

另一種失真是切向失真。當相機的鏡頭沒有完全平行於相機膠片或感測器所在的成像平面時，會發生這種情況。這使圖像看起來傾斜，使一些物體看起來比實際距離或距離更近。

有三個係數需要校正徑向失真：k1，k2和k3，以及2對於切向失真：p1，p2。在這個項目中，使用OpenCV和具有9×6角的棋盤面板來執行相機校準。

import os, glob, pickleimport numpy as npimport cv2import matplotlib.pyplot as pltclass CameraCalibrator: Compute the camera calibration matrix and distortion coefficients given a set of chessboard images. def __init__(self, image_directory, image_filename, binary_filename, nx, ny): print (Initializing CameraCalibrator ...) self.__image_directory = image_directory self.__image_filename = image_filename self.__binary_filename = binary_filename self.__nx = nx # the number of inside corners in x self.__ny = ny # the number of inside corners in y self.mtx = None self.dist = None self.rvecs = None self.tvecs = None self.__calibrated = False def __calibrate(self): # Read in and make a list of calibration images calibration_filenames = glob.glob(self.__image_directory+/+self.__image_filename) # Arrays to store object points and image points from all the images object_points = [] # 3D points in real world space image_points = [] # 2D points in image plane # Prepare object points, like (0,0,0), (1,0,0), (2,0,0), ...,(7,5,0) object_p = np.zeros((self.__ny*self.__nx,3),np.float32) object_p[:,:2] = np.mgrid[0:self.__nx,0:self.__ny].T.reshape(-1,2) # s,y coordinates # Extract the shape of any image image = cv2.imread(calibration_filenames[1]) gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY) shape = gray.shape[::-1] # (width,height) # Process each calibration image for image_filename in calibration_filenames: # Read in each image image = cv2.imread(image_filename) image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) # RGB is standard in matlibplot # Convert to grayscale gray = cv2.cvtColor(image, cv2.COLOR_RGB2GRAY) # Find the chessboard corners ret, corners = cv2.findChessboardCorners(gray, (self.__nx, self.__ny), None) # If found, draw corners if ret == True: # Store the corners found in the current image object_points.append(object_p) # how it should look like image_points.append(corners) # how it looks like # Draw and display the corners cv2.drawChessboardCorners(image, (self.__nx, self.__ny), corners, ret) plt.figure() plt.imshow(image) plt.show() # Do the calibration ret, mtx, dist, rvecs, tvecs = cv2.calibrateCamera(object_points, image_points, shape, None, None) #print(ret, mtx, dist, rvecs, tvecs) # Pickle to save time for subsequent runs binary = {} binary["mtx"] = mtx binary["dist"] = dist binary["rvecs"] = rvecs binary["tvecs"] = tvecs pickle.dump(binary, open(self.__image_directory + / + self.__binary_filename, "wb")) self.mtx = mtx self.dist = dist self.rvecs = rvecs self.tvecs = tvecs self.__calibrated = True def __load_binary(self): Load previously computed calibration binary data with open(self.__image_directory + / + self.__binary_filename, mode=rb) as f: binary = pickle.load(f) self.mtx = binary[mtx] self.dist = binary[dist] self.rvecs = binary[rvecs] self.tvecs = binary[tvecs] self.__calibrated = True def get_data(self): Getter for the calibration data. At the first call it gerenates it. if os.path.isfile(self.__image_directory + / + self.__binary_filename): self.__load_binary() else: self.__calibrate() return self.mtx, self.dist, self.rvecs, self.tvecs def undistort(self, image): if self.__calibrated == False: self.get_data() return cv2.undistort(image, self.mtx, self.dist, None, self.mtx) def test_undistort(self, image_filename, plot=False): A method to test the undistort and to plot its result. image = cv2.imread(image_filename) image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) # RGB is standard in matlibplot image_undist = self.undistort(image) # Ploting both images Original and Undistorted f, (ax1, ax2) = plt.subplots(1, 2, figsize=(20,10)) ax1.set_title(Original/Distorted) ax1.imshow(image) ax2.set_title(Undistorted) ax2.imshow(image_undist) plt.show()

開始準備「對象點」，這將是世界上棋盤角的（x，y，z）坐標。在這裡，我假設棋盤固定在z = 0處的（x，y）平面上，使得每個校準圖像的目標點是相同的。因此，objp只是一個複製的坐標數組，每當我成功檢測到測試圖像中的所有棋盤角時，objpoints都會附加一個副本。每個成功的棋盤檢測將會在圖像平面中的每個角落附加（x，y）像素位置。

然後，我使用輸出對象和imgpoint來使用OpenCV cv2.calibrateCamera（）函數來計算相機校準和失真係數。我使用cv2.undistort（）函數將此畸變校正應用於測試圖像，並獲得了以下結果：

該步驟的代碼包含在文件「./camera_calibration.py」中。將這一步應用於一個示例圖像，你會得到這樣的結果：

使用顏色變換，漸變等創建閾值二值圖像

使用顏色和漸變閾值的組合來生成二進位圖像，方法compute_binary_image（）可以在lane_detection.py中找到。有各種顏色和梯度閾值的組合來生成車道線清晰可見的二值圖像。

def compute_binary_image(self, color_image, plot=False): # Convert to HLS color space and separate the S channel # Note: img is the undistorted image hls = cv2.cvtColor(color_image, cv2.COLOR_RGB2HLS) s_channel = hls[:,:,2] # Grayscale image # NOTE: we already saw that standard grayscaling lost color information for the lane lines # Explore gradients in other colors spaces / color channels to see what might work better gray = cv2.cvtColor(color_image, cv2.COLOR_RGB2GRAY) # Sobel x sobelx = cv2.Sobel(gray, cv2.CV_64F, 1, 0) # Take the derivative in x abs_sobelx = np.absolute(sobelx) # Absolute x derivative to accentuate lines away from horizontal scaled_sobel = np.uint8(255*abs_sobelx/np.max(abs_sobelx)) # Threshold x gradient thresh_min = 20 thresh_max = 100 sxbinary = np.zeros_like(scaled_sobel) sxbinary[(scaled_sobel >= thresh_min) & (scaled_sobel <= thresh_max)] = 1 # Threshold color channel s_thresh_min = 170 s_thresh_max = 255 s_binary = np.zeros_like(s_channel) s_binary[(s_channel >= s_thresh_min) & (s_channel <= s_thresh_max)] = 1 # Combine the two binary thresholds combined_binary = np.zeros_like(sxbinary) combined_binary[(s_binary == 1) | (sxbinary == 1)] = 1 if (plot): # Ploting both images Original and Binary f, (ax1, ax2) = plt.subplots(1, 2, figsize=(20,10)) ax1.set_title(Undistorted/Color) ax1.imshow(color_image) ax2.set_title(Binary/Combined S channel and gradient thresholds) ax2.imshow(combined_binary, cmap=gray) plt.show() return combined_binary

這是我為這一步輸出的一個例子。

應用透視變換來糾正二值圖像（「鳥瞰」）

接下來，我們要為透視變換確定四個源點。在這種情況下，我們可以假設道路是平面的。這不是嚴格的，但它可以作為這個項目的近似值。我們希望從上面俯瞰道路時，選取四個梯形（類似於區域遮擋）的點，這個梯形代表一個矩形。

要做到這一點，最簡單的方法是調查車道線是直線的圖像，並找到沿線的四個點，在透視變換之後，從鳥瞰視角使線看起來筆直且垂直。

我的透視變換的代碼包括2個函數調用compute_perspective_transform()和apply_perspective_transform()，這出現在文件中lane_detection.py。所述compute_perspective_transform()構建的變換矩陣M通過使用apply_perspective_transform()變換的二進位圖像。

def compute_perspective_transform(self, binary_image): # Define 4 source and 4 destination points = np.float32([[,],[,],[,],[,]]) shape = binary_image.shape[::-1] # (width,height) w = shape[0] h = shape[1] transform_src = np.float32([ [580,450], [160,h], [1150,h], [740,450]]) transform_dst = np.float32([ [0,0], [0,h], [w,h], [w,0]]) M = cv2.getPerspectiveTransform(transform_src, transform_dst) return Mdef apply_perspective_transform(self, binary_image, M, plot=False): warped_image = cv2.warpPerspective(binary_image, M, (binary_image.shape[1], binary_image.shape[0]), flags=cv2.INTER_NEAREST) # keep same size as input image if(plot): # Ploting both images Binary and Warped f, (ax1, ax2) = plt.subplots(1, 2, figsize=(20,10)) ax1.set_title(Binary/Undistorted and Tresholded) ax1.imshow(binary_image, cmap=gray) ax2.set_title(Binary/Undistorted and Warped Image) ax2.imshow(warped_image, cmap=gray) plt.show() return warped_image

以下面的方式選擇硬編碼的源和目標點：

transform_src = np.float32([ [580,450], [160,h], [1150,h], [740,450]]) transform_dst = np.float32([ [0,0], [0,h], [w,h], [w,0]])

這兩個方法的輸出如下所示：

檢測車道像素，找到車道邊界

現在有一個閾值扭曲的圖像，我們準備繪製出車道線！有很多方法可以解決這個問題，但是在直方圖中使用峰值效果很好。

在對道路圖像進行校準，閾值處理和透視變換之後，我們應該有一個二值圖像，車道線清晰可見。但是，我們仍然需要明確地確定哪些像素是線的一部分，哪些屬於左邊線，哪些屬於右邊線。

我首先沿著圖像下半部分的所有列採用直方圖，如下所示：

import numpy as nphistogram = np.sum(img[img.shape[0]/2:,:], axis=0)plt.plot(histogram)

使用這個直方圖，我將圖像中每列的像素值相加。在我的閾值二進位圖像中，像素是0或1，所以這個直方圖中最突出的兩個峰值將成為車道線底部x坐標的良好指標。我可以用它作為尋找線條的起點。從這一點上，我可以使用一個滑動的窗口，放置在線條中心周圍，找到並遵循框架頂部的線條。

def extract_lanes_pixels(self, binary_warped): # Take a histogram of the bottom half of the image histogram = np.sum(binary_warped[binary_warped.shape[0]/2:,:], axis=0) #plt.plot(histogram) #plt.show() # Create an output image to draw on and visualize the result out_img = np.dstack((binary_warped, binary_warped, binary_warped))*255 # Find the peak of the left and right halves of the histogram # These will be the starting point for the left and right lines midpoint = np.int(histogram.shape[0]/2) leftx_base = np.argmax(histogram[:midpoint]) rightx_base = np.argmax(histogram[midpoint:]) + midpoint # Choose the number of sliding windows nwindows = 9 # Set height of windows window_height = np.int(binary_warped.shape[0]/nwindows) # Identify the x and y positions of all nonzero pixels in the image nonzero = binary_warped.nonzero() nonzeroy = np.array(nonzero[0]) nonzerox = np.array(nonzero[1]) # Current positions to be updated for each window leftx_current = leftx_base rightx_current = rightx_base # Set the width of the windows +/- margin margin = 100 # Set minimum number of pixels found to recenter window minpix = 50 # Create empty lists to receive left and right lane pixel indices left_lane_inds = [] right_lane_inds = [] # Step through the windows one by one for window in range(nwindows): # Identify window boundaries in x and y (and right and left) win_y_low = binary_warped.shape[0] - (window+1)*window_height win_y_high = binary_warped.shape[0] - window*window_height win_xleft_low = leftx_current - margin win_xleft_high = leftx_current + margin win_xright_low = rightx_current - margin win_xright_high = rightx_current + margin # Draw the windows on the visualization image cv2.rectangle(out_img,(win_xleft_low,win_y_low),(win_xleft_high,win_y_high),(0,255,0), 2) cv2.rectangle(out_img,(win_xright_low,win_y_low),(win_xright_high,win_y_high),(0,255,0), 2) # Identify the nonzero pixels in x and y within the window good_left_inds = ((nonzeroy >= win_y_low) & (nonzeroy < win_y_high) & (nonzerox >= win_xleft_low) & (nonzerox < win_xleft_high)).nonzero()[0] good_right_inds = ((nonzeroy >= win_y_low) & (nonzeroy < win_y_high) & (nonzerox >= win_xright_low) & (nonzerox < win_xright_high)).nonzero()[0] # Append these indices to the lists left_lane_inds.append(good_left_inds) right_lane_inds.append(good_right_inds) # If you found > minpix pixels, recenter next window on their mean position if len(good_left_inds) > minpix: leftx_current = np.int(np.mean(nonzerox[good_left_inds])) if len(good_right_inds) > minpix: rightx_current = np.int(np.mean(nonzerox[good_right_inds])) # Concatenate the arrays of indices left_lane_inds = np.concatenate(left_lane_inds) right_lane_inds = np.concatenate(right_lane_inds) # Extract left and right line pixel positions leftx = nonzerox[left_lane_inds] lefty = nonzeroy[left_lane_inds] rightx = nonzerox[right_lane_inds] righty = nonzeroy[right_lane_inds] return leftx, lefty, rightx, righty, left_lane_inds, right_lane_indsdef poly_fit(self, leftx, lefty, rightx, righty, left_lane_inds, right_lane_inds, binary_warped, plot:False): # Fit a second order polynomial to each left_fit = np.polyfit(lefty, leftx, 2) right_fit = np.polyfit(righty, rightx, 2) # Generate x and y values for plotting ploty = np.linspace(0, binary_warped.shape[0]-1, binary_warped.shape[0] ) left_fitx = left_fit[0]*ploty**2 + left_fit[1]*ploty + left_fit[2] right_fitx = right_fit[0]*ploty**2 + right_fit[1]*ploty + right_fit[2] # Identify the x and y positions of all nonzero pixels in the image nonzero = binary_warped.nonzero() nonzeroy = np.array(nonzero[0]) nonzerox = np.array(nonzero[1]) out_img = np.dstack((binary_warped, binary_warped, binary_warped))*255 out_img[nonzeroy[left_lane_inds], nonzerox[left_lane_inds]] = [255, 0, 0] out_img[nonzeroy[right_lane_inds], nonzerox[right_lane_inds]] = [0, 0, 255] if(plot): plt.imshow(out_img) plt.plot(left_fitx, ploty, color=yellow) plt.plot(right_fitx, ploty, color=yellow) plt.xlim(0, 1280) plt.ylim(720, 0) plt.show() return left_fit, right_fit, ploty, left_fitx, right_fitx

輸出應該是這樣的：

確定車道和車輛相對於中心的曲率

自駕車需要被告知正確的轉向角度，左轉或右轉。如果我們知道汽車的速度和動力以及車道彎曲的程度，我們就可以計算出這個角度。計算車道線曲率的一種方法是將二次多項式擬合到該線上，由此我們可以容易地提取有用的信息。

對於接近垂線的車道線，我們可以用這個公式擬合一條直線：f（y）= Ay ^ 2 + By + C，其中A，B和C是係數。A給出了車道線的曲率，B給出了該線所指向的標題或方向，並且C根據距離圖像的最左邊多遠來給出線的位置（y = 0 ）。

執行：

def compute_curvature(self, left_fit, right_fit, ploty, left_fitx, right_fitx, leftx, lefty, rightx, righty): # Define conversions in x and y from pixels space to meters ym_per_pix = 30/720 # meters per pixel in y dimension xm_per_pix = 3.7/700 # meters per pixel in x dimension y_eval = np.max(ploty) fit_cr_left = np.polyfit(ploty * ym_per_pix, left_fitx * xm_per_pix, 2) curverad_left = ((1 + (2 * left_fit[0] * y_eval / 2. + fit_cr_left[1]) ** 2) ** 1.5) / np.absolute(2 * fit_cr_left[0]) fit_cr_right = np.polyfit(ploty * ym_per_pix, right_fitx * xm_per_pix, 2) curverad_right = ((1 + (2 * left_fit[0] * y_eval / 2. + fit_cr_right[1]) ** 2) ** 1.5) / np.absolute(2 * fit_cr_right[0]) return (curverad_left + curverad_right) / 2

輸出車道邊界的視覺顯示和車道曲率和車輛位置的數值估計

lane_detection.py中的函數render_curvature_and_offset用於將檢測到的車道線返回到原始圖像上，並使用填充的多邊形繪製檢測到的車道。它還繪製了圖像或視頻幀的左上角和底部的曲率和位置。

所有六個測試圖像的結果：

輕輕一掃 歡迎關注~

http://weixin.qq.com/r/IkROVq-E3-Kgres19xEw (二維碼自動識別)