After enabling client certificate authentication: Error: {"detail":"Authentication credentials were not provided."}

[vkukk]

Summary

[cli]
base_url = "https://pulp3.hostname.tld"
username = "admin"
password = "password"
api_root = "/pulp/"
domain = "default"
format = "json"
cert = "secret/client.cert.pem"
key = "secret/client.key.pem"
verify_ssl = true

First I got CERTIFICATE_VERIFY_FAILED because pulp cli config file has no option for providing CA certificate bundle.
After enabling environment variable export PULP_CA_BUNDLE="secret/ca.crt" a new error appears:

$ pulp --config secret/cli.toml --verbose rpm repository list
repositories_rpm_rpm_list : get https://pulp3.hostname.tld/pulp/api/v3/repositories/rpm/rpm/?offset=0&limit=25
Response: 401
Error: {"detail":"Authentication credentials were not provided."}

Admin credentials didn't change, i just added client certificate authentication requirement to relevant ingress.

Steps to reproduce

As provided above

Expected behavior

pulp cli would return repository list

Pulp and pulp-cli version info

pulp-cli version:
$ pulp --config secret/cli.toml --version
Pulp3 Command Line Interface, Version 0.29.1
Plugin Versions:
common: 0.29.1

Pulp version:
latest default images from pulp-operator

[mdellweg]

pulp-cli is supposed to authenticate either via client certs OR via basic auth.
Does your installation require both? Should the client certificates actually be proxy certs instead?

[vkukk]

I'm migrating from old Pulp 2 server installation to Pulp 3 pulp-operator on Kubernetes.

To match existing working Pulp 2 configuration in Pulp 3 installation, I've created two custom ingresses:

• public using Letsencrypt
• Custom CA protected private content and API access protection, limited by few client certificate fingerprints.

Client certificate protection is for blocking botnets from trying to exploit Pulp weaknesses.
Usernam/password are for Pulp RBAC.

Makes perfect sense to me to support both in pulp cli - TLS for secure tunnel and username/password for API authentication. They are not the same to me, protection is layered, groups of users (mostly robots) are using same client certificate to gain access to API endpoint and protected content under private path.

They authenticate themselves to API with unique usernames and passwords. Creating and managing per robot certificates, integreate all this into k8s ingress and Pulp RBAC system seems too much work to manage manually and not easy to automate.

Also, I'm not aware of pulp-operator supporting anything else but password authentication for the admin user. https://pulpproject.org/pulp-operator/docs/admin/reference/custom_resources/repo_manager/

[vkukk]

I'm unable to find Pulp documentation about enabling client certificate authentication for API users in Pulp.

Must say that Pulp 3 documentation way worse than Pulp 2 was. Both content, its structure and UI.

https://pulpproject.org/pulpcore/restapi/#tag/Users/operation/users_update
does not list any certificate options.

[mdellweg]

The thing you are searching for is probably:
https://pulpproject.org/pulpcore/docs/admin/guides/auth/external/
The problem being that most of it happens outside of Pulp.
For testing purposes we have a POC like solution around here:
https://github.com/pulp/pulp-cli/blob/main/.ci/nginx.conf.j2#L88

I'm not saying that your initial request (somehow allowing multiple auth mechanisms to be used simultaneously) is invalid. It just never occurred to me that this would be needed, and we need to also think about servers where either one of the auth mechanisms is sufficient.

[vkukk]

The only thing needed to change is to comment out one line in code.

pulp-cli/pulpcore/cli/common/generic.py

Line 144 in 2feb2da

if not api_kwargs.get("cert"):

144,145c144,145
<         if not api_kwargs.get("cert"):
<             api_kwargs["auth_provider"] = PulpCLIAuthProvider(pulp_ctx=self)
---
>         #if not api_kwargs.get("cert"):
>         api_kwargs["auth_provider"] = PulpCLIAuthProvider(pulp_ctx=self)

And pulp cli works as I need it to work.

I don't get the external auth thing. There is no need for it, username and password are Pulp internal credentials. Just send them with mTLS cert and key and problem solved.

Since there would only be a handful of client certificates, following is working well in Ingress config:

  annotations:
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "300"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "300"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
    nginx.ingress.kubernetes.io/proxy-body-size: "20m"
    nginx.ingress.kubernetes.io/auth-tls-verify-client: "on"
    nginx.ingress.kubernetes.io/auth-tls-verify-depth: "2"
    nginx.ingress.kubernetes.io/auth-tls-secret: "pulp/pulp-ca-ca-cert-tls-gf2dhht27f"
    nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "false"
    nginx.ingress.kubernetes.io/server-snippet: |
      set $client_allowed "0";

      if ($ssl_client_fingerprint = "57e4c20e641fc1834f80700e3824a0c35c89c10d") {
        set $client_allowed "1";
      }

      if ($ssl_client_fingerprint = "aab6c38da92bc43f6aa181fb1769a606bd006f4d") {
        set $client_allowed "1";
      }

      # You can comment out or add more fingerprints as needed
      # Proper formatting of SHA1 fingerprint:
      # openssl x509 -in cert.pem -noout -fingerprint -sha1 | sed 's/://g; s/.*=//' | tr 'A-Z' 'a-z'
      # if ($ssl_client_fingerprint = "") {
      #   set $client_allowed "1";
      # }

      if ($client_allowed = "0") {
        return 403;
      }

[vkukk]

Actually, two changes are needed: https://github.com/vkukk/pulp-cli/pull/1/commits/a5342e08419067946086f44e9e1a08c10bdef079