Yes. You want to offer a consistent user experience to your visitors. If they see a different page / post every time they access your site they would quickly realize they are being your “guinea pigs” and this could influence their behavior. Moreover, from a statistical point of view, the validity of the test depends on keeping the two “samples” (i.e. the two or more groups of users under test, each one seeing a different version of the post/page) independent.