Human capital accumulation through college education has become an important measure to narrow the income gap. Based on the micro-data of China Family Panel Studies, using the instrumental variable quantile regression method, the heterogeneity of returns to college education in China under the conditional income distribution was systematically analyzed from the perspective of full sample, gender, urban and rural, and regions. The results show that returns to college education in China is L-shaped according to the conditional income distribution, which means the low-income and high-income groups have higher yields. This means that low-income groups can narrow the income gap with other groups through college education. The “Matthew effect” of high-income groups to expand the income gap through college education exists at the same time. For all income groups, returns to college education for women are higher than men; the annual return rate for low-income groups is 4.59% for men and 10.62% for women. In addition to the low-income group of 10 points, the returns of urban are greater than of rural areas. The returns in the east increases with income levels, while the central and western regions are declining.