In terms of the hanging, it should be fine. I have a Reefmat 250, which is smaller, and it 100% would hook over the bracing of a 40 breeder. I’d anticipate a 500 either has the same hanging bracket or larger.
Functionally though, in a sump-less system I’m not sure how effective this would be. It’s meant to be plumbed directly into your overflow so all water from the display flows through the roll, I’m sure plumbing a pump to get some water flow through it + another tube to return to the aquarium would remove some waste, but I anticipate it would be significantly less effective.
I’m in the process of setting up a 40 breeder as a holding tank and have no intention of using the Reef Mat in the temp system.