Visualforce: apex:inputText – restrict input to numbers only

One way you could make this work is HTML5-ifying your Visualforce page and use <apex:input type="number" /> instead of apex:inputText.

Otherwise, you could implement some javascript validation or send it back to your controller and do the validation in Apex.

Note: To HTML5-ify your page, the easiest way is using the page
element with the docType attribute <apex:page docType="html-5.0">.

See:
http://blogs.developerforce.com/developer-relations/2011/12/visualforce-html5-mobile.html