Your concept is largely what I'm trying as well. Darn baffles just get in the way.. lol.
Can't vouch for the approach until it's worked for a few years, but I can say you can give it a go without sudden and irretrievable failure. Watch out for a lot of build up in the sump/fuge. If you let it go too long and remove it all at once, can cause a good swing in the tank.