I honestly think that will be fun especially with the yoga mat the weight will compress it down as well as the plywood will get compressed as well you could always get some wood putt to build it up right there but what is a millimeter 1/32” or a 1/16 you’re fine. Nothing can be perfect