If you have a small Web-site (under ten web pages), then I wouldn’t get worried far too much about possibly of such data files. Even so, for those who’re employing WordPress, You should utilize a plugin like Yoast Search engine optimisation to produce both of those a robots.txt file https://www.youtube.com/watch?v=EVeTP0aV8gY