BAU: Check for broken external links

Sometimes pages disappear from the internet or move, and we want to keep our outbound links in good condition. We sleep between requests to GitHub to avoid being rate-limited, and ignore some specific repositories which will fail the link checker.

config.rb

@@ -9,15 +9,24 @@
 
 after_build do |builder|
   begin
-    HTMLProofer.check_directory(config[:build_dir],
+    proofer = HTMLProofer.check_directory(config[:build_dir],
       { :assume_extension => true,
-        :disable_external => true,
         :allow_hash_href => true,
         :empty_alt_ignore => true,
         :file_ignore => [
             /search/ # Provided by tech-docs gem but has a "broken" link from html-proofer's point of view
+        ],
+        :url_ignore => [
+            "https://github.com/alphagov/centralised-security-logging-service"
         ]
-      }).run
+      })
+
+      proofer.before_request do |request|
+        # We get rate-limited by GitHub so pause between checking GitHub links
+        sleep 2 if request.base_url == "https://github.com/"
+      end
+
+      proofer.run
   rescue RuntimeError => e
     abort e.to_s
   end