Where did any of them claim that "only" Tea Party groups were targeted?

As I recall,"tea party" and other keywords common to those groups were among those used improperly according to the IRS Inspector General.