Depends on where the rust is and what that item is.
If it lives in the tank: Get it out. So a powerhead magnet, frag rack magnet, screw, hose clamp, etc - if it stays in the water it's not good to keep in. This is less because rust itself is bad, but more because we don't know what metals may be contained in those items, and so we could be damaging our livestock without knowing it.
If it's a tool? So long as its usable and is mostly used outside of the water you're not going to see much in the way of negatives from it. Tongs, cutters, blades, scrapers - if they live outside of the water and only interact with your tank when you're using it, the main downside with rust is just if it makes the tool harder to use.
You can usually clean off rust with citric acid, which you should have on hand as a pump cleaner anyhow.