Document how to use the wheel cache

[therc]

How does one use or debug wheel caching?

I am using something like this

$ bazel test --test_output=errors -s --build_tests_only \
   --action_env BAZEL_WHEEL_CACHE=https://foo/bar //test:target 

But it still seems to be fetching things upstream, e.g.

  Downloading https://files.pythonhosted.org/packages/74/68/d87d9b36af36f44254a8d512cbfc48369103a3b9e474be9bdfe536abfc45/python_dateutil-2.7.5-py2.py3-none-any.whl (225kB)

[scele]

It doesn't work with --action_env, since the caching happens inside a repository rule. Try this instead:

BAZEL_WHEEL_CACHE=https://foo/bar bazel test --test_output=errors -s --build_tests_only  //test:target 

[therc]

Ah, right. I hadn't tried that because I thought it was too obvious and simple.

[therc]

I still can't get it to work with Dazel (which, I see, you forked, too). I even edited the Dazel sources to pass the hardcoded -e BAZEL_WHEEL_CACHE=foo flag to Docker. I think that at some point I got something to work by going straight inside the VM and running Bazel myself. Is it supposed to set the "urls" field in the wheel dictionary in requirements.bzl?

[scele]

I've been meaning to provide an example how to do this. Actually, I've already stopped using BAZEL_WHEEL_CACHE because it bypasses bazel's repository cache, which makes it re-download wheels all the time. My current setup is such that I have a custom repository rule that generates a file that looks like this:

# generated @pip_cdn//:cdn.bzl
cdn = {
    "000e68abd33c972a5248544925a0cae7d1125f9bf6c58280d37546b946769a08": "https://my-s3-bucket/jsonschema-2.6.0-py2.py3-none-any.whl.000e68abd33c972a5248544925a0cae7d1125f9bf6c58280d37546b946769a08",
    "02aec4bd92ab067f6ff27a38a38a41173bf01bed8f89157768c1573f53e474a6": "https://my-s3-bucket/docutils-0.14-py3-none-any.whl.02aec4bd92ab067f6ff27a38a38a41173bf01bed8f89157768c1573f53e474a6",
    "1309725c0223e25c78231af96fe28802e3804b847e58ba451c5d7379eead0614": "https://my-s3-bucket/boto3-1.9.13-py2.py3-none-any.whl.1309725c0223e25c78231af96fe28802e3804b847e58ba451c5d7379eead0614",
    # ...
}

Then, I have customized pip_install to take the URL from the CDN, if available:

load("@pip_cdn//:cdn.bzl", "cdn")

def pip_install(pip):
    for distribution, attributes in pip.wheels.items():
        kwargs = {}
        sha256 = attributes.get("sha256", None)
        if sha256 and sha256 in cdn:
            kwargs["urls"] = [cdn[sha256]]
        wheel = pip.download_or_build_wheel(
            distribution = distribution,
            **kwargs
        )
        pip.extract_wheel(wheel = wheel, distribution = distribution)

And in WORKSPACE:

load("@pip_deps//:requirements.bzl", py_wheels = "info")
load("//tools/pip:install.bzl", "pip_install")
pip_install(py_wheels)

I haven't had time to clean the "cdn" code to be generic enough to make it part of rules_python. When I get to that, I'd like to remove the BAZEL_WHEEL_CACHE mechanism.

[scele]

Basically, what I explain above is just a fancy / more complex way to achieve this:

wheel_overrides = {
    "pytest": {
        "urls": ["https://my-s3-bucket/pytest-3.0.5-py2.py3-none-any.whl.c97bdefdca852d48c3144b8a534a78527534793bac959c10211ed3e037925020"],
    },
    "tensorflow-gpu": {
        "urls": ["https://my-s3-bucket/tensorflow_gpu-1.12.0-cp27-cp27mu-manylinux1_x86_64.whl.435a9a4a37c1a92f9bc80f577f0328775539c593b9bc9e943712a204ada11db5"],
    },
    "torch": {
        "urls": ["https://my-s3-bucket/torch-0.4.1-cp36-cp36m-manylinux1_x86_64.whl.7e3bac584473688720e26323bf209b97c015fc4e9a12a34962df3ff7ad0ce597"],
    },
    # ...
}

pip_import(
    name = "pip_deps",
    # ...
    requirements_overrides = wheel_overrides,
)