Saying NO to a "friend's" child's BIRTHDAY party?

I have a son about to turn three. For his first birthday party, I invited close family and only really close friends who I keep in touch with and see a lot. For his second, I invited fam only and I plan to do the same for his third because finances don't allow for anything extravagant right now. Anyway, I always have these brand new friends or acquaintances inviting me to their child's BD party. I find it kind of annoying because it seems like they're just looking to get another gift, but maybe I'm wrong. I don't want to go to non-close friends' children's BD parties because I can't afford to buy gifts for everyone's kids. How do I say NO?! What are your thoughts?