Because special interest groups have more influence due to the fact that a lot of them are affiliated with politicians and sometimes have been politicians themselves. This relates to the iron triangle and revolving door.
I believe interest groups are more damaging because they use public opinion and in our society , the public as a significant amount of power and can change policy based on population petitions or voting .
