Yes, and get progressively louder until I do something about it. It can affect the sump level to the point that my ATO will kick on. I've tried leaving it for as long as 12hrs and the only thing that happened was my nerves getting more and more frayed as the vortex noises got louder.
Let's assume your system is running for a few days without you touching anything and it's running quietly. Then all of a sudden you get a vortex in primary drain. This indicates your return pump flow decreased.
Same scenario but now you are getting noise in the secondary. This indicates your return pump flow increased.
These are the only two possibilities given the circumstances you've describe. (assuming you done have something clogging either drain, which you don't IMO)
*The above assume no filter sock that could alter the height of water in relation to the drain pipe.