I agree, in fact a healthy coral will typically be colorful and grow faster.
To answer the OP's question, if you have any one limiting factor, you will see neither color or growth. I have never heard of anyone saying "yeah my corals grow extremely fast, but they're pale". Or the converse "my corals are brightly colored, but never seem to grow" Usually you have both when things are doing well.
The rumor of "growth" is a high alkalinity and low nutrients.
The rumor of "color" is high nutrients, maybe natural sea water alkalinity.
I don't actually believe that you have one or the other, it's just not a thing. When you have a healthy coral, it's both colorful and growing fast. If you have a weak coral it's growing but with anemic color.
For corals, develop techniques which will provide adequate nutrients, maintain stable water parameters, good lighting and water movement. There are a million ways to get to the below parameters and maintain them, and that's the challenge, following whatever techniques ring true to you and develop those routines and techniques which will keep your tank healthy and growing.
Best advice I've ever gotten which what has really kicked off my success over the past 3 months was a guy who told me to get a hanna ulr phosphorous checker. Sure enough, once I nailed phosphorous levels everything has been coloring up. So there are these leap moments where things get dramatically better. As you learn your tank, your testing, your equipment, you'll improve greatly over time and it's not an achievable goal to have both color and growth. In fact I think growth is partially dependent on color, weak corals don't grow fast. Colorful corals means higher levels of zooxanthellae which means more energy.
Things which I attribute to colorful and fast growth in my tank, in order of importance:
Lighting - I use T5s
Food - I feed the tank heavily which maintains 8+ nitrates and .04+ ppm Phosphates I also have been doing reef chili which I believe to be helping at night
Alkalinity/calcium 11dkh/450ppm
BRS 2 part
water movement - definetly get as much random movement as you can tolerate
Skimming/nutrient removal - big skimmer and carbon filtering
red sea coral colors - I can't say definitively that they're the cause, but my thought is they do help some.
There's an old saying which I like that says "follow those who are also looking for answers, and stay away from those who claim to have found it". Look for the meaning behind the answers, instead of just the answers.