Overgrazing has driven degradation and desertification of semi-arid grasslands in Northern China over recent decades. Selective grazing by sheep influences sward structure by inducing heterogeneous vegetation patterns comprising overgrazed hotspots and areas rejected by grazing sheep. In this study, we examined the effects of grazing intensity (ungrazed, light and heavy grazing in 2008 and 2010) and grazing system (a mixed system involving continuous grazing alternating annually with hay making vs. a continuous system involving continuous utilization of the same area for grazing) on plant biomass distribution and ecosystem functioning after 4 years and 6 years of controlled grazing in a semi-arid steppe of Inner Mongolia, China. The spatial biomass distribution was determined by sward height measurements converted to biomass and afterwards visualized in biomass distribution semivariograms. Within each of the different areas: grazed (i.e., areas frequently grazed by sheep), rejected (i.e., areas largely avoided by grazing sheep) and fenced (i.e., areas from which grazing had been excluded by fencing), plant species and soil parameters were sampled in order to analyze the mechanisms and effects of grazing patterns on ecosystem functioning. The results revealed a more homogeneous biomass distribution in the ungrazed and heavily grazed plots compared to lightly grazed plots, in which heterogeneous biomass distribution patterns included both overgrazed hotspots and rejected areas. The patch vegetation patterns were consistent between years only under light grazing intensity. However, patch vegetation patterns in the continuous system did not necessarily indicate negative effects on grassland ecosystem functioning. Within the 6 years of grazing experiment, it appears that patchy structure rather than homogeneous patterns showed higher biodiversity, significant variations in litter, soil water content and soil temperature and smaller effects on belowground biomass and carbon storage. Therefore, heterogeneous patchy vegetation patterns are likely to moderate grassland recovery and optimize ecosystem functioning by forming resource islands with sufficient water and nutrients in the short run.