Using RPKI on PEERING

Using Krill on PEERING

Access to PEERING's RPKI is distinct from access to PEERING muxes. If you plan to use RPKI for PEERING prefixes, you will need to engage with PEERING operators to get approval for your experiment and receive the necessary clearance. As for scheduling, please consider that the current Krill infrastructure allows for only one experiment to have access to RPKI at a time (this is due to access-control limitations on Krill's command-line client).

Users connect to PEERING's Krill instance by establishing an OpenVPN tunnel to the VM running Krill. The OpenVPN server uses PEERING's CA, so you can use the same client.crt and client.key files you use on your PEERING client. The OpenVPN server will check whether your experiment is approved to connect, and you will also need a ta.key file (provided by PEERING operators on approval of your experiment) which we use for HMAC-signing of messages between your machine and Krill's OpenVPN server. Finally, you will need Krill's authentication token and openvpn.conf file, which will be provided to you by PEERING operators.

Setting up your PEERING RPKI client

1. Copy OpenVPN secret files to openvpn/certs

Your openvpn/certs file should contain four files: ca.crt, client.crt, client.key, and ta.key. The first three are the same ones used by the PEERING client, and the fourth one is provided to you on approval of your experiment.

2. Start an OpenVPN tunnel to Krill's VP

PEERING production Krill instance runs on peering.ee.columbia.edu. You can use the openvpn.conf file to run openvpn --config openvpn.conf on this directory to establish the tunnel (adjust the paths inside the configuration file if running from other locations).

After successful authentication, OpenVPN will create a tunrpki0 interface, through which you can reach the Krill server.

3. Configure certificate validation for krillc

As krillc performs certificate validation when accessing a Krill server that is not localhost, you need your machine to access the Krill server in a way that will pass certificate validation. This involves two tasks:

1. Setting up an entry in your /etc/hosts so you can access the Krill server through the OpenVPN tunnel by name. Add 100.124.0.2 krill-vpn.peering.ee.columbia.edu to your /etc/hosts file. (The 100.64.0.0/10 prefix is reserved for Carrier-Grade NATs, and is not announced publicly.)

2. The Krill server will provide krillc with a certificate for krill- vpn.peering.ee.columbia.edu. This certificate is signed by PEERING's CA. To configure krillc to accept PEERING's CA, add a path to PEERING's ca.crt (the same one used in the client) file to an environment variable called KRILL_HTTPS_ROOT_CERTS (it's a colon-separated list of file paths).

   This environment variable does not seem to be documented publicly, but it is discussed in this issue and added in this commit.

4. Issue commands to Krill using krillc

Install Krill following its documentation. You can then issue commands to krillc also following the documentation. All RPKI-enabled PEERING prefixes are managed under a single CA called peering-testbed.

Please note that krillc has permission to access all endpoints of Krill's API, including ones that could break the certificate chain between Krill and the registrar. Please read Krill's and krillc's documentation carefully and review your commands before issuing them. You can likely do everything your experiment needs with the following commands:

• List ROAs:

  krillc roas list --ca peering-testbed --token $ADMINTOKEN \
      --server https://krill-vpn.peering.ee.columbia.edu:3000/
  

• Adding a ROA:

  krillc roas update --ca peering-testbed --token $ADMINTOKEN \
      --server https://krill-vpn.peering.ee.columbia.edu:3000/ \
      --add "$PREFIX => $ASN"
  

• Remove a ROA:

  krillc roas update --ca peering-testbed --token $ADMINTOKEN \
      --server https://krill-vpn.peering.ee.columbia.edu:3000/ \
      --remove "$PREFIX => $ASN"
  

• Immediately publish all changed ROAs:

  krillc bulk publish --token $ADMINTOKEN \
      --server https://krill-vpn.peering.ee.columbia.edu:3000/
  

  Please note that this command is executed periodically by Krill every 10 minutes, so your experiment may not need to force bulk updates.

In the commands above, $PREFIX must be one of the NIC.br IPv4 prefixes (in the 138.185.228.0/22 range). Unless explicitly required by your experiment and allowed by PEERING operators, restrict $ASN to one of the PEERING- controlled ASNs (33207, 47065, 61574-61576, or 263842-263844).

5. Checking ROA propagation

You can check your ROAs have propagated with any third-party validator, like Routinator.

Using Krill's Docker container

We can execute Krill from the docker container by passing in the environment variable to set the certificate and mounting the certificate in a volume. This avoids the needs to configure Krill on the machine.

As of 2023-02-28, compiling Krill on aarch64 works, but certificate validation fails (possibly something related to OpenSSL). Using the container on aarch64 works.

Here is an example:

COMMAND=krillc roas list --ca peering-testbed --token $ADMINTOKEN --server https://krill-vpn.peering.ee.columbia.edu:3000/
docker run \
    -e KRILL_HTTPS_ROOT_CERTS=/certs/ca.crt \
    -v /localpath/certs:/certs \
    nlnetlabs/krill \
    $COMMAND

Development Client Configuration

The vm-tests infrastructure contains a client VM that is partially autoconfigured using the rpki-test-client.yml playbook. Here are adjustments to the steps above:

1. OpenVPN secrets should be in vm-tests/rpki/client/certs are copied automatically.

2. The OpenVPN configuration for the test client is copied automatically (it is different from prod because it points to 192.168.252.2 isntead of peering.ee.columbia.edu). You will still need to bring up the OpenVPN tunnel manually after the VM is provisioned:

   # in `server/vm-tests/rpki`
   vagrant ssh client
   cd client
   openvpn --config openvpn.conf
   

3. The /etc/hosts file is patched during provisioning. KRILL_HTTPS_ROOT_CERTS_ENV is set to /home/vagrant/client/certs by a file copied to /etc/profile.d/krillenv.sh.

4. You can check everything is working by running the roas list and roas update commands above.