You're right, caliber doesn't necessarily matter. But there were rounds designed to be anti-tank rounds, and a lot of them became obsolete within 2 years of introduction. The only anti-tank round I can think of still in use is 14.5x115mm and that's because Russia will never replace anything they can find a use for. So now said round is an anti-material round because if you tried using it for it's original intended role, you might as well use your sidearm pistol, it'll be equally as effective.
You can say that anti-tank rifles are just very specialized anti-material rifles but that's the point, it's specialized and unique, and that particular role is dead, so dedicated anti-tank guns don't exist anymore outside of museums.