Removing the pectoral muscle from the mammogram images is a very important step for computer aided cancer diagnosis methods due to the fact that pectoral muscle having similar properties with the abnormal tissue. This is a hard task since the pectoral muscle can be in various shapes, sizes, and densities. In this paper; firstly Canny edge detection method was used to identify a reference area that covered a certain amount of the pectoral muscle. Afterwards, the average color values of the image remaining in the reference area is found which is converted to the L*a*b* color space before and the distance of all pixels from the this average value is computed. In this new image consisting of color differences, pixels that are close to the mean value are marked as the pectoral muscle. The study was done on 40 mammograms images taken from the Mini-MIAS database. The results obtained were evaluated by an expert radiologist and borders of pectoral muscle taking place in 36 mammograms were determined as acceptable. © 2012 IEEE.