I haven't got anything more concrete than this at the moment, but I thought I would show where my thought process is at the moment.
I think that the basic idea is still sound, but I need to split up the dice provision, the number reduction, and the value adder mechanisms. I'm thinking of having the dice come from several sources, based on your action, the reduction come from a skill of some sort, and the adder come from your attribute, so I can keep the value low. I also thin I need ones to remove dice, starting with your highest first. I did some test rolls last night, and this seemed to curb things nicely. I am gaming with my crew tonight, but I will have time to throw my mind against the issue tomorrow.