Nope. They will still brute force, they'll just fail. And while they're doing it, your system will be slow/sluggish.
I like my whitelist only version.
But I thought I would warn others.
Remember that once the Hackers find out that there is a live machine there, they will log your IP and come back time and again with a new attack method. phpMyAdmin flaws, SIP account, anything and everything. White list: no chance to guess and they don't even know there is a server there.
So just like using an alternate port, you're getting away with it "for now", but eventually they Will Try Again.