You can set a fixed size
But be aware that fixed sizes can be problematic. I recently got a 4K monitor. I found that the default font size was too small to be easily readable, so I changed it to 125% magnification. As a consequence, everywhere that I had specified fixed sizes for text in my layouts was now too small for the text. The pixel sizes of the layouts hadn't changed, but effectively the font size did.