Reps. Tim Huelskamp, R-Kan., and Justin Amash, R-Mich., huddled with conservatives after the Republican Steering Committee removed them from their seats on the House Budget Committee.

"It's petty, it's vindictive and if you have any conservative principle, you'll be punished," Huelskamp said at a briefing for conservatives at the Heritage Foundation.

Huelskamp and Amash, along with Reps. David Schweikert, R-Ariz., and Walter Jones, R-N.C., lost their seats on the budget panel and House Financial Services Committee after the GOP conference determined they were "not team players," in the words of one Republican aide.

That action has prompted a minor outcry among conservatives, who fear that lawmakers who cross the GOP leadership will be punished for their transgressions. That fear coincides with mounting concern on the right that Republican leaders will cut a "fiscal cliff" deal with President Barack Obama that results in higher taxes, through either increased rates or eliminated deductions.