implement the last subresources

[clux]

Subresources have a lot of special case logic, that is not easily derivable from k8s_openapi. So far we have implemented (see subresources.rs):

• pods/eviction (impl in add eviction subresource - for #127 #393)
• pods/log (already implemented, used as example herein)
• pods/attach (hard - impl in Add WebSocket connection support and implement /attach and /exec #360)
• pods/portforward (hard How should we implement WebSocket connection? #229, sketch in Add port forward #446)
• pods/exec (hard How should we implement WebSocket connection? #229 - impl in Add WebSocket connection support and implement /attach and /exec #360)
• {service,node,pod}/proxy - see Proxy Subresource #1161
• pods/binding (see https://docs.rs/k8s-openapi/latest/src/k8s_openapi/v1_26/api/core/v1/binding.rs.html#79-100 )
• ephemeral containers - Implement ephemeral containers subresource #1153
• certificatesigningrequest/approval (easy - follow from Request: support arbitrary subresources #487 and see docs) - Add Api fns for arbitrary subresources and approval subresource for CertificateSigningRequest #773
• TokenRequest - for client util (same as csr approach, easy) - done in Add create_subresource method to Api and create_token_request method to Api<ServiceAccount> #989
• arbitrary subresources - probably port Request work from Request: support arbitrary subresources #487 (comment) to API - Add Api fns for arbitrary subresources and approval subresource for CertificateSigningRequest #773

The general outline for the http based subresources:

We need a definition of what they are in subresources. The way to upgrade would be:

1. Implement subresource queries straight on Resource without restrictions (for now):

impl Resource {
    /// Get a pod logs
    pub fn log(&self, name: &str, lp: &LogParams) -> Result<http::Request<Vec<u8>>> {
        ...
        // Check https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/ for urls
    }
}

2. Make a marker trait for the subresource:

pub trait Loggable {}

impl<K> Api<K>
where
    K: Clone + DeserializeOwned + Meta + Loggable,
{
    pub async fn log(&self, name: &str, lp: &LogParams) -> Result<String> {
        let req = self.api.log(name, lp)?;
        Ok(self.client.request_text(req).await?)
    }

    pub async fn log_stream(&self, name: &str, lp: &LogParams) -> Result<impl Stream<Item = Result<Bytes>>> {
        let req = self.api.log(name, lp)?;
        Ok(self.client.request_text_stream(req).await?)
    }
}

3. Designate what resources implement this subresource:

impl Loggable for k8s_openapi::api::core::v1::Pod {}

Write one test case for one resource in examples (e.g. examples/log_openapi.rs).

EDIT (0.46.0):

Now this is not as difficult for the specialized requests requiring websockets (ws feature).
This was discussed at length in #229, and finally implemented in #360.
Implementors of the last subresources ought to look at that PR for help.

[koiuo]

Hi, I'd like to give this a try (I'm interested in support for exec mostly and would like to contribute what I can (which is not much at this point) to bring its support to kube-rs sooner).

I do understand, that this task is just about implementing subresources in rust code, not about handling them correctly. However one item is particularly confusing to me

1. Implement subresource queries straight on Resource without restrictions (for now)

I started with the logs example and came up with this for the exec

    pub fn exec(&self, name: &str, ep: &ExecParams) -> Result<http:Request<Vec<u8>>> {
        let base_url = self.make_url() + "/" + name + "/" + "exec?";
        let mut qp = url::form_urlencoded::Serializer::new(base_url);

        if let Some(container) = &lp.container {
            qp.append_pair("container", &container);
        }

        ...

        let req = http::Request::post(urlstr);
        req.body(vec![]).map_err(Error::HttpError)
    }

The issue is that exec can't be performed over http. It used to require SPDY (which is now deprecated) and now relies on websockets. Hence, an attempt to send a http request to this endpoint will yield 400 as response.

I wanted to clarify, whether you deliberately have chosen Result<http::Request<...>> as a return type to serve as a placeholder or you had something else in mind here.

If it's the former, should we maybe resort to Any to prevent folks using kube-rs from relying on these parts of API while full support is not there yet (you can't do much with Any I imagine)?

[clux]

The issue is that exec can't be performed over http. It used to require SPDY (which is now deprecated) and now relies on websockets. Hence, an attempt to send a http request to this endpoint will yield 400 as response.

I wanted to clarify, whether you deliberately have chosen Result<http::Request<...>> as a return type to serve as a placeholder or you had something else in mind here.

If it's the former, should we maybe resort to Any to prevent folks using kube-rs from relying on these parts of API while full support is not there yet (you can't do much with Any I imagine)?

Honestly, I wasn't aware of how much harder subresources like exec or port-forward` were going to be. We have a bit of an open discussion about it in #229, but we haven't gotten to a satisfying conclusion there yet.

Would assume that we need another return type for exec. Any doesn't sound very useful from an api reuse pov, nor from a user pov. It almost certainly needs a new transport dependency and a new signature specifically for the subresource.

That said, I haven't thought too much about this yet. If you want to help sketch out something or discuss on #229 that would be much appreciated.

[clux]

0.46.0 introduces the ws feature using async-tungstenite to pave the way for the last subresources listed above. Have updated the issue slightly. Implementations of the last ws-based subresources can look at #360 for an excellent PR that covers (among the websocket handling itself) examples for implementing pods/attach and pods/exec.

[kazk]

I'll do pods/portforward. kube should probably just provide AsyncRead + AsyncWrite for each port and not bind to local ports. Users should be able to do that if they want to.

I have a rough sketch that can do something similar to Python client's example:

// Portforward nginx container
let mut pf = pods.portforward("example", &[80]).await?; // Portforwarder
let mut ports = pf.ports().unwrap(); // Vec<Port>
let mut port = ports[0].stream().unwrap(); // impl AsyncRead + AsyncWrite + Unpin
port.write_all(b"GET / HTTP/1.1\r\nHost: 127.0.0.1\r\nConnection: close\r\nAccept: */*\r\n\r\n")
    .await?;
let mut rstream = tokio_util::io::ReaderStream::new(port);
if let Some(res) = rstream.next().await {
    match res {
        Ok(bytes) => {
            let response = std::str::from_utf8(&bytes[..]).unwrap();
            println!("{}", response);
            assert!(response.contains("Welcome to nginx!"));
        }
        Err(err) => eprintln!("{:?}", err),
    }
}

I'll open a draft PR later (it's too messy at the moment to show...) to discuss the design.

That should make kube a gold client except for "Proto encoding".

Gold Requirements

• Support exec, attach, port-forward calls (these are not normally supported out of the box from swagger-codegen)
• Proto encoding

I'm not sure what that means. Do we need to support Protobufs?

[clux]

Yeah, that sounds sensible. Listening sounds like an application concern. In an example we can maybe use TcpListener or something and direct it to the writable stream to show how to do the kubectl port-forward equivalent.

protobufs

Ugh. Ah, yeah, I am not sure that's really possible atm. The go api has protobuf codegen hints (see api/types.go) like:

// +optional
// +patchMergeKey=name
// +patchStrategy=merge
EphemeralContainers []EphemeralContainer `json:"ephemeralContainers,omitempty" patchStrategy:"merge" patchMergeKey:"name" protobuf:"bytes,34,rep,name=ephemeralContainers"`

whereas the (huge) swagger codegen has the equivalent json for that part of the PodSpec:

        "ephemeralContainers": {
          "description": "...",
          "items": {
            "$ref": "#/definitions/io.k8s.api.core.v1.EphemeralContainer"
          },
          "type": "array",
          "x-kubernetes-patch-merge-key": "name",
          "x-kubernetes-patch-strategy": "merge"
        },

think the 34 there would be required, but haven't looked at this much. Arnavion might be able to say for sure whether it's possible, I can ping him on k8s-openapi.