So there are a bunch of reasons like others have said. With saltwater, we're dealing with fish that generally get bigger, have issues with being territorial and showing aggression, typically are used to a decent amount of swimming space, and we have a bioload to worry about on top of that. When keeping a complex system with very little mechanical filtration, water parameters are very important and a sudden shift can cause havoc and destroy everything we've worked so hard for and been patient with.
The other bit is what we consider the "pet" in the situation. I can't speak for everyone here or the freshwater community, but when I think of a freshwater tank, the fish are the pets and the tank is essentially the cage/housing. With reef keeping, the reef as a whole is the "pet" that we keep. For me personally, the fish are important, but it's really about finding that balance of fish, coral, inverts, and everything else (including microfauna, rock, substrate) and having them all work together to produce something beautiful and somewhat self-sustaining.
The other bit is what we consider the "pet" in the situation. I can't speak for everyone here or the freshwater community, but when I think of a freshwater tank, the fish are the pets and the tank is essentially the cage/housing. With reef keeping, the reef as a whole is the "pet" that we keep. For me personally, the fish are important, but it's really about finding that balance of fish, coral, inverts, and everything else (including microfauna, rock, substrate) and having them all work together to produce something beautiful and somewhat self-sustaining.