Loaded question and one that is often asked...
As mentioned in a post above, the best reference as to whether long term low PO4 and NO3 is an issue is how your corals look. In a mature system that is fed regularly, this is often a non-issue as the PO4 and NO3 that are being measured by test kits are just residuals in the water column and the rest of the regular nutrient input is being assimilated by the tank's organisms (and in addition your PO4 test kit is only reading the inorganic form of phosphate and not the organic forms). Kind of like if I give you a cheeseburger for lunch and you eat all but a few crumbs that fall on the floor you'll still have received enough nutrients for maintenance and likely growth :)
If you have a newer and therefore less stable system it typically can't yet handle larger food input without becoming eutrophic ('more polluted'). When overfed and in response to algae and cyano outbreaks that thrive with the excess nutrients, many reef keepers tend to overcompensate by cutting the nutrients way down much too fast by feeding much less, adding more nutrient absorbing media, vigorous cleaning/detritus removal and/or larger water changes. All of which can then lead to nutrient levels that don't provide enough nutrients for the animals to even maintain a healthy metabolism and they start to loose color, tissue and eventually die off.