private_artifact_repo¶

Intro¶

protoprimer may need to install dependencies from private artifact repos.

This also requires authn/authz settings for those artifact repos.

There is a choice to rely on python (or pip, or uv, or ...) features to disvover system or user configuration or make it configurable for that repo clone only.

Without taking away the choice, the most autonomous approach (isolated to that repo clone only) is required.

Isolation should allow hooks that configure access to these private artifact repos during bootstrap.

`venv` restrictions¶

There are several things to take care of:

private artifact repo URL
private artifact repo authn/authz

The 1st one (URL) is easy - it can be external (system or user) configuration or internal (via CLI arg specified by the wrapper).

The 2nd one (authn/authz) is harder - it often requires additional packages in venv (e.g. for pip). To avoid relying on system-level or user-level software, those extra packages will have to be installed in the venv before installing packages from private artifacts repos behind authn/authz.

A solution: Google Cloud artifact repo¶

It is testable via these manual steps:

Create (e.g.) local_auth project for editable installation.
The local_auth project should have the necessary dependencies like keyring (they should not require auth*).
Install that project first as a separate install_group (e.g. "install_auth").
Install others (requiring auth*) after that as another install_group (e.g. "install_main").
Add dependency on dummy_private (not used otherwise) to local_repo.
Modify config to use the two groups "install_auth" and "install_main" (see above).
provide --extra-index-url with URL in extra_command_args:
- in case of venv_pip, use "https://asia-southeast1-python.pkg.dev/protoprimer/test-python-repo/simple"
- in case of venv_uv, use "https://oauth2accesstoken@asia-southeast1-python.pkg.dev/protoprimer/test-python-repo/simple"
In case of venv_uv only, provide --keyring-provider and subprocess in extra_command_args.
Accessing to keyring:

To allow uv to find keyring, protoprimer sets PATH env var to include bin dir of the target venv.

In case of venv_pip, it is not required as python process figures it out by itself somehow.

"venv_driver": "venv_pip"¶

Config snippet for venv_pip test:

{
    "project_descriptors": [
        {
            "build_root_dir_rel_path": "src/local_auth",
            "install_extras": [],
            "install_group": "install_auth"
        },
        {
            "build_root_dir_rel_path": "src/local_doc",
            "install_extras": [],
            "install_group": "install_main"
        },
        {
            "build_root_dir_rel_path": "src/local_repo",
            "install_extras": [],
            "install_group": "install_main"
        },
        {
            "build_root_dir_rel_path": "src/local_test",
            "install_extras": [],
            "install_group": "install_main"
        },
        {
            "build_root_dir_rel_path": "src/metaprimer",
            "install_extras": [],
            "install_group": "install_main"
        },
        {
            "build_root_dir_rel_path": "src/protoprimer",
            "install_extras": [
                "test"
            ],
            "install_group": "install_main"
        }
    ],

    "install_specs": [
        {
            "install_auth": {
            }
        },
        {
            "install_main": {
                "extra_command_args": [
                    "--pre",
                    "--extra-index-url",
                    "https://asia-southeast1-python.pkg.dev/protoprimer/test-python-repo/simple"
                ]
            }
        }
    ]
}

"venv_driver": "venv_uv"¶

Config snippet for venv_uv test:

{
    "project_descriptors": [
        {
            "build_root_dir_rel_path": "src/local_auth",
            "install_extras": [],
            "install_group": "install_auth"
        },
        {
            "build_root_dir_rel_path": "src/local_doc",
            "install_extras": [],
            "install_group": "install_main"
        },
        {
            "build_root_dir_rel_path": "src/local_repo",
            "install_extras": [],
            "install_group": "install_main"
        },
        {
            "build_root_dir_rel_path": "src/local_test",
            "install_extras": [],
            "install_group": "install_main"
        },
        {
            "build_root_dir_rel_path": "src/metaprimer",
            "install_extras": [],
            "install_group": "install_main"
        },
        {
            "build_root_dir_rel_path": "src/protoprimer",
            "install_extras": [
                "test"
            ],
            "install_group": "install_main"
        }
    ],

    "install_specs": [
        {
            "install_auth": {
            }
        },
        {
            "install_main": {
                "extra_command_args": [
                    "--keyring-provider",
                    "subprocess",
                    "--pre",
                    "--extra-index-url",
                    "https://oauth2accesstoken@asia-southeast1-python.pkg.dev/protoprimer/test-python-repo/simple"
                ]
            }
        }
    ]
}

Test private repo scenario¶

This is a sample test scenario to use private repo.

It publishes dummy_private into a private artifact repo.

Define the region and the project for Google Cloud:

export CLOUDSDK_COMPUTE_ZONE=asia-southeast1
export CLOUDSDK_CORE_PROJECT=protoprimer
export ARTIFACT_REPO_NAME=test-python-repo

Create artifact repo:

./cmd/gcloud_artifact_repo create "${ARTIFACT_REPO_NAME}" --project="${CLOUDSDK_CORE_PROJECT}" --location="${CLOUDSDK_COMPUTE_ZONE}"

Publish package:

./cmd/publish_package dummy_private --no_tag --repository_url "https://${CLOUDSDK_COMPUTE_ZONE}-python.pkg.dev/protoprimer/${ARTIFACT_REPO_NAME}/"

See the content of the artifact repo (click on the echo-printed URL):

echo "https://console.cloud.google.com/artifacts/python/protoprimer/${CLOUDSDK_COMPUTE_ZONE}/${ARTIFACT_REPO_NAME}?project=protoprimer"

Set up a new test venv:

python -m venv tmp/test-venv
tmp/test-venv/bin/pip install keyring keyrings.google-artifactregistry-auth

NOTE: Several points:

simple suffix in the URL (otherwise, pip does not recognize the artifact repo structure).
--pre arg to allow pre-released packages (what was supposed to be published to the artifact repo).
--extra-index-url (instead of --index-url) to allow dependencies from other artifact repos.

Install package:

tmp/test-venv/bin/pip install dummy_private --pre --extra-index-url "https://${CLOUDSDK_COMPUTE_ZONE}-python.pkg.dev/protoprimer/${ARTIFACT_REPO_NAME}/simple"

Check package:

tmp/test-venv/bin/pip freeze

Delete artifact repo:

./cmd/gcloud_artifact_repo delete "${ARTIFACT_REPO_NAME}" --project="${CLOUDSDK_CORE_PROJECT}" --location="${CLOUDSDK_COMPUTE_ZONE}"

Delete the test venv:

mv tmp/test-venv tmp/test-venv."$( date -u +"%Y%m%dT%H%M%SZ" )"