I think you are observing (at least) two phenomena:
- When something consumes phosphate it shifts the equilibrium between bound and freely dissolved phosphorus. This means a fraction of the consumed phosphate will slowly leech back into the water through the rock and substrate making phosphate assimilation look inefficient. This can also occur when performing water changes.
- The carbon, nitrogen, and phosphorus (C:N:P) assimilation ratio is always skewed towards N, and especially C. This means you need a lot of carbon and nitrogen to get rid of a tiny bit of phosphorus.
I assume after providing carbon you are running into a nitrogen limitation and therefore adding a nitrogen source (especially ammonia directly) helps. Even if you still have moderate nitrate readings it might take time to reduce NO₃ back to NH₃/NH₄ to actually get to the needed nitrogen. I believe this reduction process requires nitrate reductase, and nitrite reductase enzymes, which are based on molybdenum, and iron or copper, if I remember correctly. This makes trace elements another potential limiting factor.
Long story short: Assimilation of phosphate based on bacteria growth will always be x times slower than nitrogen assimilation (depending on the C:N:P uptake ratio of the participating bacteria species).