Height loss starting in middle age is reported to be an independent risk factor for cardiovascular mortality. Recent studies have revealed an inverse association between height and hypertension, but the influence of hypertension on height loss is unknown. Since hypertension is an established cardiovascular risk factor, clarifying the association between baseline hypertension and height loss could lead to an efficient tool to estimate the risk of mortality. A retrospective study of 11,154 Japanese aged 40–74 years was conducted. Height loss was defined as being in the highest quintile of annual height decrease (≥2.015 mm/year for men and ≥1.756 mm/year). Baseline height was significantly inversely associated with incident hypertension for men only. The adjusted odds ratio (OR) and 95% confidence interval (CI) for incident hypertension for each 1 standard deviation increment of height (5.9 cm for men and 5.6 cm for women) was 0.90 (0.84, 0.97) for men and 1.07 (0.91, 1.26) for women, respectively. We also found that baseline hypertension is independently positively associated with height loss for men only. The adjusted OR was 1.25 (1.11, 1.42) for men and 0.93 (0.71, 1.21) for women. These results might lead to an efficient tool for estimating the risk of height loss, which has been reported to be associated with a higher risk of mortality in adults.