We report results from a search for gravitational waves produced by perturbed intermediate mass black holes (IMBH) in data collected by LIGO and Virgo between 2005 and 2010. The search was sensitive to astrophysical sources that produced damped sinusoid gravitational wave signals, also known as ringdowns, with frequency $50\le f_{0}/\mathrm{Hz} \le 2000$ and decay timescale $0.0001\lesssim \tau/\mathrm{s} \lesssim 0.1$ characteristic of those produced in mergers of IMBH pairs. No significant gravitational wave candidate was detected. We report upper limits on the astrophysical coalescence rates of IMBHs with total binary mass $50 \le M/\mathrm{M}_\odot \le 450$ and component mass ratios of either 1:1 or 4:1. For systems with total mass $100 \le M/\mathrm{M}_\odot \le 150$, we report a 90\%-confidence upper limit on the rate of binary IMBH mergers with non-spinning and equal mass components of $6.9\times10^{-8}\,$Mpc$^{-3}$yr$^{-1}$. We also report a rate upper limit for ringdown waveforms from perturbed IMBHs, radiating 1\% of their mass as gravitational waves in the fundamental, $\ell=m=2$, oscillation mode, that is nearly three orders of magnitude more stringent than previous results.