You can do this, but I never have. If your IP changes, or you temporarily need to access from a different IP, it becomes a hassle. I'm not convinced that is a weak spot. If you use good passwords and limit who can access administrator, it would be uncommon for anyone to "break in" via Administrator.
It's far more likely that someone will manage to inject bad code into the database. This can cause pages to be redirected to malware sites. The best way to prevent this is to ensure both the core and all extensions are kept up to date.