Welcome,
Have you tried creating a language model which uses the number-related words you’d like to recognize? e.g. “ONE”, “TWO”, [….] , “TWENTY”, [….], “HUNDRED EIGHTY”, [….]
This is what would be necessary if you wanted to mix in numbers with other speech. If you want to recognize numbers exclusively, you’d want to replace your acoustic model with the tidigits acoustic model (and also create a language model that contains the spoken versions of the numbers you want to be able to recognize).