In theory, this should be determined by how important different sized errors are to you, or in other words, your loss function.
In the real world, people put ease of use first. So RMS deviations (or the related variances) are easier to combine, and easier to calculate in a single pass, while average absolute deviations are more robust to outliers and exist for more distributions. Basic linear regression and many of its offshoots are based on minimsing RMS errors.
Another point is that the mean will minimise RMS deviations while the median will minimise absolute deviations, and you may prefer one of these.