Try simply changing the directory name of your build tree and see if the installation breaks. If not, it should be safe to remove the files completely.