Who knew the robots file could be so political. Nevertheless, the new administration Today ushered in a new robots.txt file!
As Jason said, the robots.txt file tells search engines which part of the website they are allowed or not allowed to index. Before, the whitehouse website was more restricted and dictatorial in its search engine diplomacy, and now it is more open and more democratic!