Replace polipo with an internal proxy to cache HTTPS connection

[mickours]

The Polipo proxy is only available for HTTP connection and the HTTPS connections are not cached.
Use our own cache implementation may be a good idea to add this feature, remove the polipo dependency and allows kameleon to make this cache mechanism evolve more easily.

It seems to be pretty simple to implement this in ruby:
https://gist.github.com/torsten/74107

[SalemHarrache]

The only way to proxy (and cache) HTTPS is the MITM and it's a tlittle hard to setup it in kameleon.

[mickours]

Yes, I agree this is not easy. Maybe this can be replace by good practice like:

• setting exact git commit of all built software
• setting exact package version in classical package dependencies. (pip, gem, deb, rpm,... )

This can be done for home built packages and software but for all dependencies and runtime this is not common nor easy to do.

I think the only way to do it properly is a kind of MITM to catch anything that come from the network.
But you're right this not easy and it might require a complete tool by itself.

[SalemHarrache]

I think the only way to do it properly is a kind of MITM to catch anything that come from the network.
But you're right this not easy and it might require a complete tool by itself.

HTTPS is designed to prevent man-in-the middle attacks, and a transparent proxy is effectively a MITM.

If we want to cache HTTPS, we need to generate and inject the proxy certificate
in every context (local/in/out) so that the clients (wget, etc.) can trust
the proxy (MITM).

You can find an example here with squid and more information about Ssl Bump here.

Another way to achieve that is to make clients (wget --no-check-certificate ...) ignore the SSL certificat checks for all HTTPS connections, but that's not really ideal neither...

Anyway, it won't be with a script of ours that we can make a proxy with such features.
Even polipo does not offer persistent cache for HTTPS (with SSL Bump), it just opens
a tunnel (http://www.pps.univ-paris-diderot.fr/~jch/software/polipo/manual/Tunnelling-connections.html)

Yes, I agree this is not easy. Maybe this can be replace by good practice like:

• setting exact git commit of all built software
• setting exact package version in classical package dependencies. (pip, gem, deb, rpm,... )

A simple workaround is to only use a cachable content when you need to make your experience (recipe) reproductible. At the end avoid HTTPS as much as possible.