Add nginx bot allowing

[ansell]

The current nginx robots.txt configuration has a generic allow.

In the case of the doi service, it would be useful to allow the datacite verification service named pidcheck access to doi's to verify that they are still accessible:

https://github.com/datacite/pidcheck/blob/master/crawler/pidcheck/settings.py#L25

It would be useful to extend the current robots.txt construction mechanism to allow for some bots to get access while the default rule remains at no access.