Selfservice-node fails to redirect to user flows

[Bdegraaf1234]

Hi Everyone,

I am trying to set up self-hosted kratos on our cluster.

I managed to get the kratos and selfservice pod up and running, but the self service pod fails to redirect to the registration flow and gives the error below.

{
  "id": "backend-error",
  "error": {
    "config": {
      "url": "kratos-public/self-service/registration/flows?id=bb97cb62-4ec8-4567-847c-f8745e212f83",
      "method": "get",
      "headers": {
        "Accept": "application/json, text/plain, */*",
        "Cookie": "csrf_token_2d29bbe1dcdc5866b6f304cb2f47b5e5a82679d90c1e86d3a61761bd125478ba=FcAqAkMBSLWqST+5aJ9GphrNFPX4PF8AKEq1n5KCJCE=",
        "User-Agent": "axios/0.21.4"
      },
      "transformRequest": [
        null
      ],
      "transformResponse": [
        null
      ],
      "timeout": 0,
      "xsrfCookieName": "XSRF-TOKEN",
      "xsrfHeaderName": "X-XSRF-TOKEN",
      "maxContentLength": -1,
      "maxBodyLength": -1,
      "transitional": {
        "silentJSONParsing": true,
        "forcedJSONParsing": true,
        "clarifyTimeoutError": false
      }
    },
    "code": "ECONNREFUSED"
  }
}

If I perform the request using curl, I get the same error unless I add the header 'Accept: application/json'.

So curl -v -H "Accept: application/json" https://public.kratos.dev-server.local.net/self-service/registration/browser? gives the desired response (screenshot below).

I don't know what to do anymore, does anyone have a clue? I checked the CORS pitfalls, and tried to debug it for hours now to no avail.

Helm files

kratos.yaml

# Default values for kratos.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

# -- Number of replicas in deployment
replicaCount: 1
# -- Deployment update strategy
strategy:
    type: RollingUpdate
    rollingUpdate:
        maxSurge: "25%"
        maxUnavailable: "25%"

image:
    # -- ORY KRATOS image
    repository: oryd/kratos
    # -- ORY KRATOS VERSION
    # Alternative format: image: oryd/kratos:v0.6.3-alpha.1
    tag: v1.0.0
    pullPolicy: IfNotPresent
    # imagePullPolicy: Always

imagePullSecrets: [ ]
nameOverride: ""
fullnameOverride: ""

service:
    admin:
        enabled: true
        type: ClusterIP
        # -- Load balancer IP
        loadBalancerIP: ""
        port: 80
        # -- The service port name. Useful to set a custom service port name if it must follow a scheme (e.g. Istio)
        name: http
        # -- Provide custom labels. Use the same syntax as for annotations.
        labels: { }
        # -- If you do want to specify annotations, uncomment the following
        # lines, adjust them as necessary, and remove the curly braces after 'annotations:'.
        annotations: { }
        # kubernetes.io/ingress.class: nginx
        # kubernetes.io/tls-acme: "true"
        # -- Path to the metrics endpoint
        metricsPath: /admin/metrics/prometheus
    public:
        enabled: true
        type: ClusterIP
        # -- Load balancer IP
        loadBalancerIP: ""
        port: 80
        # -- The service port name. Useful to set a custom service port name if it must follow a scheme (e.g. Istio)
        name: http
        # -- Provide custom labels. Use the same syntax as for annotations.
        labels: { }
        # -- If you do want to specify annotations, uncomment the following
        # lines, adjust them as necessary, and remove the curly braces after 'annotations:'.
        annotations: { }
        # kubernetes.io/ingress.class: nginx
        # kubernetes.io/tls-acme: "true"

secret:
    # -- switch to false to prevent creating the secret
    enabled: true
    # -- Provide custom name of existing secret, or custom name of secret to be created
    nameOverride: ""
    # nameOverride: "myCustomSecret"
    # -- Annotations to be added to secret. Annotations are added only when secret is being created. Existing secret will not be modified.
    secretAnnotations:
        # Create the secret before installation, and only then. This saves the secret from regenerating during an upgrade
        # pre-upgrade is needed to upgrade from 0.7.0 to newer. Can be deleted afterwards.
        helm.sh/hook-weight: "0"
        helm.sh/hook: "pre-install, pre-upgrade"
        helm.sh/hook-delete-policy: "before-hook-creation"
        helm.sh/resource-policy: "keep"
    # -- switch to false to prevent checksum annotations being maintained and propogated to the pods
    hashSumEnabled: true

ingress:
    admin:
        enabled: true
        className: "nginx"
        annotations:
            #      kubernetes.io/ingress.class: nginx
            cert-manager.io/cluster-issuer: letsencrypt-prod
            # kubernetes.io/ingress.class: nginx
        # kubernetes.io/tls-acme: "true"
        hosts:
            -   host: dev-server.local.net
                paths:
                    -   path: "/kratos-admin/?(.*)"
                        pathType: ImplementationSpecific
        tls:
            -   secretName: kratos-admin-tls
                hosts:
                    - dev-server.local.net
    public:
        enabled: true
        className: "nginx"
        annotations:
            nginx.ingress.kubernetes.io/rewrite-target: /$1
            #      kubernetes.io/ingress.class: nginx
            cert-manager.io/cluster-issuer: letsencrypt-prod
            # kubernetes.io/ingress.class: nginx
        # kubernetes.io/tls-acme: "true"
        hosts:
            -   host: dev-server.local.net
                paths:
                    -   path: "/kratos/?(.*)"
                        pathType: ImplementationSpecific
        tls:
            -   secretName: kratos-public-tls
                hosts:
                    - dev-server.local.net

kratos:
    development: true
    # -- Enable the initialization job. Required to work with a DB

    # -- Enables database migration
    automigration:
        enabled: true
        # -- Configure the way to execute database migration. Possible values: job, initContainer
        # When set to job, the migration will be executed as a job on release or upgrade.
        # When set to initContainer, the migration will be executed when Kratos pod is created
        # Defaults to job
        type: job
        # -- Ability to override the entrypoint of the automigration container
        # (e.g. to source dynamic secrets or export environment dynamic variables)
        customCommand: [ ]
        # -- Ability to override arguments of the entrypoint. Can be used in-depended of customCommand
        # eg:
        # - sleep 5;
        #   - kratos
        customArgs: [ ]
        # -- resource requests and limits for the automigration initcontainer
        resources: { }

    # -- You can add multiple identity schemas here. You can pass JSON schema using `--set-file` Helm CLI argument.
    identitySchemas:
        "identity.user.schema.json": |
            {
              "$schema": "http://json-schema.org/draft-07/schema#",
              "$id": "user",
              "title": "user",
              "type": "object",
              "properties": {
                "traits": {
                  "type": "object",
                  "properties": {
                    "email": {
                      "type": "string",
                      "format": "email",
                      "title": "E-Mail",
                      "minLength": 5,
                      "ory.sh/kratos": {
                        "credentials": {
                          "password": {
                            "identifier": true
                          },
                          "totp": {
                            "account_name": true
                          }
                        },
                        "verification": {
                          "via": "email"
                        },
                        "recovery": {
                          "via": "email"
                        }
                      }
                    }
                  },
                  "required": [ "email" ]
                }
              },
              "additionalProperties": false
            }

    #  "identity.email.schema.json": |
    #    {
    #      // ...
    #    }
    #  "identity.phone.schema.json": |
    #    {{ .Values.phone_schema }}

    # -- You can customize the emails Kratos is sending (also uncomment config.courier.template_override_path below)
    emailTemplates: { }
    # emailTemplates:
    #   recovery:
    #     valid:
    #       subject: Recover access to your account
    #       body: |-
    #         Hi, please recover access to your account by clicking the following link:
    #         <a href="{{ .RecoveryURL }}">{{ .RecoveryURL }}</a>
    #       plainBody: |-
    #         Hi, please recover access to your account by clicking the following link: {{ .RecoveryURL }}
    #     invalid:
    #       subject: Account access attempted
    #       body: |-
    #         Hi, you (or someone else) entered this email address when trying to recover access to an account.
    #         However, this email address is not on our database of registered users and therefore the attempt has failed. If this was you, check if you signed up using a different address. If this was not you, please ignore this email.
    #       plainBody: |-
    #         Hi, you (or someone else) entered this email address when trying to recover access to an account.
    #   verification:
    #     valid:
    #       subject: Please verify your email address
    #       body: |-
    #         Hi, please verify your account by clicking the following link:
    #         <a href="{{ .VerificationURL }}">{{ .VerificationURL }}</a>
    #       plainBody: |-
    #         Hi, please verify your account by clicking the following link: {{ .VerificationURL }}
    #     invalid:
    #       subject:
    #       body:
    #       plainBody:

    config:

        session:
            cookie:
                domain: local.net
                persistent: true
                name: "radar"
                same_site: Strict

        dsn: "postgres://postgres:57UppGzDG0@kratos-psql-postgresql.default.svc.cluster.local/kratos?sslmode=disable&max_conn_lifetime=10s"

        courier:
            smtp:
                connection_uri: smtps://test:test@mailslurper:1025/?skip_ssl_verify=true&disable_starttls=true
                from_address: noreply@radar-base.org

        serve:
            public:
                base_url: https://dev-server.local.net/kratos/
                cors:
                    enabled: true
                    allowed_origins:
                        - https://dev-server.local.net/kratos-ui/
                        - http://dev-server.local.net/kratos-ui/
                    allowed_methods:
                        - POST
                        - GET
                        - PUT
                        - PATCH
                        - DELETE
                    allowed_headers:
                        - Authorization
                        - Cookie
                        - Content-Type
                        - Accept
                    exposed_headers:
                        - Content-Type
                        - Set-Cookie
                        - Accept
                    allow_credentials: true

        secrets:
            default:
                - PLEASE-CHANGE-ME-I-AM-VERY-INSECURE
                - "OG5XbmxXa3dYeGplQXpQanYxeEFuRUFa"
                - "foo bar 123 456 lorem"
                - "foo bar 123 456 lorem 1"
                - "foo bar 123 456 lorem 2"
                - "foo bar 123 456 lorem 3"

        selfservice:
            default_browser_return_url: https://dev-server.local.net/managementportal
            allowed_return_urls:
                - "https://dev-server.local.net/"

            methods:
                password:
                    enabled: true
                #    oidc:
                #      config:
                #        providers:
                #          # social sign-in for google. This needs to be tied to a google account. values below were added by bastiaan
                #          - id: google_d292689d # this is `<provider-id>` in the Authorization callback URL. DO NOT CHANGE IT ONCE SET! current google callback: http://127.0.0.1:4433/self-service/methods/oidc/callback/google_d292689d
                #            provider: google
                #            client_id: 922854293804-r3fhl9tom6uutcq5c8fm4592l1t6s3mh.apps.googleusercontent.com # Replace this with the Client ID
                #            client_secret: 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 # Replace this with the Client secret
                #            issuer_url: https://accounts.google.com # Replace this with the providers issuer URL
                #            mapper_url: "base64://bG9jYWwgY2xhaW1zID0gewogIGVtYWlsX3ZlcmlmaWVkOiBmYWxzZSwKfSArIHN0ZC5leHRWYXIoJ2NsYWltcycpOwoKewogIGlkZW50aXR5OiB7CiAgICB0cmFpdHM6IHsKICAgICAgW2lmICdlbWFpbCcgaW4gY2xhaW1zICYmIGNsYWltcy5lbWFpbF92ZXJpZmllZCB0aGVuICdlbWFpbCcgZWxzZSBudWxsXTogY2xhaW1zLmVtYWlsLAogICAgfSwKICB9LAp9"
                #            # currently: GitHub example from: https://www.ory.sh/docs/kratos/social-signin/data-mapping
                #            # Alternatively, use an URL:
                #            # mapper_url: https://storage.googleapis.com/abc-cde-prd/9cac9717f007808bf17
                #            scope:
                #              - email
                #            # supported scopes can be found in your providers dev docs
                #      enabled: true
                totp:
                    config:
                        issuer: Kratos
                    enabled: true
                link:
                    enabled: true

            flows:
                error:
                    ui_url: https://dev-server.local.net/kratos-ui/error

                settings:
                    ui_url: https://dev-server.local.net/kratos-ui/settings

                recovery:
                    enabled: true
                    ui_url: https://dev-server.local.net/kratos-ui/recovery
                    use: link

                verification:
                    # our current flow necessitates that users reset their password after they activate an account in managementportal,
                    # this works as verification
                    ui_url: https://dev-server.local.net/kratos-ui/verification
                    enabled: true
                    use: link
                    after:
                        default_browser_return_url: https://dev-server.local.net/kratos-ui

                logout:
                    after:
                        default_browser_return_url: https://dev-server.local.net/kratos-ui/login

                login:
                    ui_url: https://dev-server.local.net/kratos-ui/login

                registration:
                    ui_url: https://dev-server.local.net/kratos-ui/registration
                    after:
                        password:
                            hooks:
                                -   hook: session
                        oidc:
                            hooks:
                                -   hook: session

        identity:
            default_schema_id: user
            schemas:
                # identitySchemas:
                -   id: user
                    url: file:///etc/config/identity.user.schema.json

# -- Configuration options for the k8s deployment
deployment:
    lifecycle: { }
    # -- Configure the livenessProbe parameters
    livenessProbe:
        initialDelaySeconds: 5
        periodSeconds: 10
        failureThreshold: 5
    # -- Configure the readinessProbe parameters
    readinessProbe:
        initialDelaySeconds: 5
        periodSeconds: 10
        failureThreshold: 5
    # -- Configure the startupProbe parameters
    startupProbe:
        failureThreshold: 60
        successThreshold: 1
        periodSeconds: 1
        timeoutSeconds: 1

    # -- Configure a custom livenessProbe. This overwrites the default object
    customLivenessProbe: { }
    # -- Configure a custom readinessProbe. This overwrites the default object
    customReadinessProbe: { }
    # -- Configure a custom startupProbe. This overwrites the default object
    customStartupProbe: { }

    # -- Array of extra arguments to be passed down to the deployment. Kubernetes args format is expected
    # - --foo
    # - --sqa-opt-out
    extraArgs: [ ]

    # -- Array of extra envs to be passed to the deployment. Kubernetes format is expected
    # - name: FOO
    #   value: BAR
    extraEnv: [ ]
    # -- If you want to mount external volume
    # For example, mount a secret containing Certificate root CA to verify database
    # TLS connection.
    extraVolumes: [ ]
    # - name: my-volume
    #   secret:
    #     secretName: my-secret
    extraVolumeMounts: [ ]
    # - name: my-volume
    #   mountPath: /etc/secrets/my-secret
    #   readOnly: true

    # extraVolumes:
    #   - name: postgresql-tls
    #     secret:
    #       secretName: postgresql-root-ca
    # extraVolumeMounts:
    #   - name: postgresql-tls
    #     mountPath: "/etc/postgresql-tls"
    #     readOnly: true

    # -- If you want to add extra init containers. These are processed before the migration init container.
    extraInitContainers: ""
    # extraInitContainers: |
    #  - name: ...
    #    image: ...

    # -- If you want to add extra sidecar containers.
    extraContainers: ""
    # extraContainers: |
    #  - name: ...
    #    image: ...

    # -- Set desired resource parameters
    #  We usually recommend not to specify default resources and to leave this as a conscious
    #  choice for the user. This also increases chances charts run on environments with little
    #  resources, such as Minikube. If you do want to specify resources, uncomment the following
    #  lines, adjust them as necessary, and remove the curly braces after 'resources:'.
    resources: { }
    #  limits:
    #    cpu: 100m
    #    memory: 128Mi
    #  requests:
    #    cpu: 100m
    #  memory: 128Mi

    # -- Pod priority
    # https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/
    priorityClassName: ""

    # -- Node labels for pod assignment.
    nodeSelector: { }
    # If you do want to specify node labels, uncomment the following
    # lines, adjust them as necessary, and remove the curly braces after 'nodeSelector:'.
    #   foo: bar

    # -- Configure node tolerations.
    tolerations: [ ]

    # -- Configure node affinity
    affinity: { }

    # -- Configure pod topologySpreadConstraints.
    topologySpreadConstraints: [ ]
    # - maxSkew: 1
    #   topologyKey: topology.kubernetes.io/zone
    #   whenUnsatisfiable: DoNotSchedule
    #   labelSelector:
    #     matchLabels:
    #       app.kubernetes.io/name: kratos
    #       app.kubernetes.io/instance: kratos

    # -- Configure pod dnsConfig.
    dnsConfig: { }
    #   options:
    #     - name: "ndots"
    #       value: "1"

    labels: { }
    #      If you do want to specify additional labels, uncomment the following
    #      lines, adjust them as necessary, and remove the curly braces after 'labels:'.
    #      e.g.  type: app

    annotations: { }
    #      If you do want to specify annotations, uncomment the following
    #      lines, adjust them as necessary, and remove the curly braces after 'annotations:'.
    #      e.g.  sidecar.istio.io/rewriteAppHTTPProbers: "true"

    # -- The secret specified here will be used to load environment variables with envFrom.
    # This allows arbitrary environment variables to be provided to the application which is useful for
    # sensitive values which should not be in a configMap.
    # This secret is not created by the helm chart and must already exist in the namespace.
    # https://kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/#configure-all-key-value-pairs-in-a-secret-as-container-environment-variables
    # environmentSecretsName:

    # -- Specify the serviceAccountName value.
    # In some situations it is needed to provide specific permissions to Kratos deployments.
    # Like for example installing Kratos on a cluster with a PosSecurityPolicy and Istio.
    # Uncomment if it is needed to provide a ServiceAccount for the Kratos deployment.
    serviceAccount:
        # -- Specifies whether a service account should be created
        create: true
        # -- Annotations to add to the service account
        annotations: { }
        # -- The name of the service account to use. If not set and create is true, a name is generated using the fullname template
        name: ""

    # https://github.com/kubernetes/kubernetes/issues/57601
    automountServiceAccountToken: true

    # -- Specify pod metadata, this metadata is added directly to the pod, and not higher objects
    podMetadata:
        # -- Extra pod level labels
        labels: { }
        # -- Extra pod level annotations
        annotations: { }

    # -- Parameters for the automigration initContainer
    automigration:
        # -- Array of extra envs to be passed to the initContainer. Kubernetes format is expected
        # - name: FOO
        #   value: BAR
        extraEnv: [ ]
    # -- Number of revisions kept in history
    revisionHistoryLimit: 5

## -- Configuration options for the k8s statefulSet
statefulSet:
    resources: { }
    #  We usually recommend not to specify default resources and to leave this as a conscious
    #  choice for the user. This also increases chances charts run on environments with little
    #  resources, such as Minikube. If you do want to specify resources, uncomment the following
    #  lines, adjust them as necessary, and remove the curly braces after 'resources:'.
    #  limits:
    #    cpu: 100m
    #    memory: 128Mi
    #  requests:
    #    cpu: 100m
    #  memory: 128Mi

    # -- Array of extra arguments to be passed down to the StatefulSet. Kubernetes args format is expected
    extraArgs: [ ]
    # - --foo
    # - --sqa-opt-out

    extraEnv: [ ]
    # -- If you want to mount external volume
    # For example, mount a secret containing Certificate root CA to verify database
    # TLS connection.
    extraVolumes: [ ]
    # - name: my-volume
    #   secret:
    #     secretName: my-secret
    extraVolumeMounts: [ ]
    # - name: my-volume
    #   mountPath: /etc/secrets/my-secret
    #   readOnly: true

    # -- If you want to add extra init containers. These are processed before the migration init container.
    extraInitContainers: ""
    # extraInitContainers: |
    #  - name: ...
    #    image: ...

    # -- If you want to add extra sidecar containers.
    extraContainers: ""
    # extraContainers: |
    #  - name: ...
    #    image: ...

    annotations: { }
    #      If you do want to specify annotations, uncomment the following
    #      lines, adjust them as necessary, and remove the curly braces after 'annotations:'.
    #      e.g.  sidecar.istio.io/rewriteAppHTTPProbers: "true"

    # -- The secret specified here will be used to load environment variables with envFrom.
    # This allows arbitrary environment variables to be provided to the application which is useful for
    # sensitive values which should not be in a configMap.
    # This secret is not created by the helm chart and must already exist in the namespace.
    # https://kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/#configure-all-key-value-pairs-in-a-secret-as-container-environment-variables
    # environmentSecretsName:

    labels: { }
    #      If you do want to specify additional labels, uncomment the following
    #      lines, adjust them as necessary, and remove the curly braces after 'labels:'.
    #      e.g.  type: app

    # -- Pod priority
    # https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/
    priorityClassName: ""

    # -- Node labels for pod assignment.
    nodeSelector: { }
    # If you do want to specify node labels, uncomment the following
    # lines, adjust them as necessary, and remove the curly braces after 'nodeSelector:'.
    #   foo: bar

    # -- Configure node tolerations.
    tolerations: [ ]

    # -- Configure node affinity
    affinity: { }

    # -- Configure pod topologySpreadConstraints.
    topologySpreadConstraints: [ ]
    # - maxSkew: 1
    #   topologyKey: topology.kubernetes.io/zone
    #   whenUnsatisfiable: DoNotSchedule
    #   labelSelector:
    #     matchLabels:
    #       app.kubernetes.io/name: kratos
    #       app.kubernetes.io/instance: kratos

    # -- Configure pod dnsConfig.
    dnsConfig: { }
    #   options:
    #     - name: "ndots"
    #       value: "1"

    log:
        format: json
        level: trace

        # -- Specify pod metadata, this metadata is added directly to the pod, and not higher objects
    podMetadata:
        # -- Extra pod level labels
        labels: { }
        # -- Extra pod level annotations
        annotations: { }
    # -- Number of revisions kept in history
    revisionHistoryLimit: 5

# -- Container level security context
securityContext:
    capabilities:
        drop:
            - ALL
    readOnlyRootFilesystem: true
    runAsNonRoot: true
    runAsUser: 65534
    runAsGroup: 65534
    allowPrivilegeEscalation: false
    privileged: false
    seccompProfile:
        type: RuntimeDefault
    seLinuxOptions:
        level: "s0:c123,c456"

# -- Pod level security context
podSecurityContext:
    fsGroupChangePolicy: "OnRootMismatch"
    runAsNonRoot: true
    runAsUser: 65534
    fsGroup: 65534
    runAsGroup: 65534
    seccompProfile:
        type: RuntimeDefault
    sysctls: [ ]
    supplementalGroups: [ ]

# -- Horizontal pod autoscaling configuration
autoscaling:
    enabled: false
    minReplicas: 1
    maxReplicas: 3
    targetCPU: { }
    #   type: Utilization
    #   averageUtilization: 80
    targetMemory: { }
    #   type: Utilization
    #   averageUtilization: 80

# -- Values for initialization job
job:
    # -- If you do want to specify annotations, uncomment the following
    # lines, adjust them as necessary, and remove the curly braces after 'annotations:'.
    annotations:
        helm.sh/hook-weight: "1"
        helm.sh/hook: "pre-install, pre-upgrade"
        helm.sh/hook-delete-policy: "before-hook-creation,hook-succeeded"
    # kubernetes.io/ingress.class: nginx
    # kubernetes.io/tls-acme: "true"

    # -- If you want to add extra sidecar containers.
    extraContainers: ""
    # extraContainers: |
    #  - name: ...
    #    image: ...

    # -- If you want to add extra init containers.
    extraInitContainers: ""
    # extraInitContainers: |
    #  - name: ...
    #    image: ...

    # -- Array of extra envs to be passed to the job. This takes precedence over deployment variables. Kubernetes format is expected
    # - name: FOO
    #   value: BAR
    extraEnv: [ ]

    # -- Node labels for pod assignment.
    nodeSelector: { }
    # If you do want to specify node labels, uncomment the following
    # lines, adjust them as necessary, and remove the curly braces after 'nodeSelector:'.
    #   foo: bar

    # -- resource requests and limits for the job
    resources: { }

    # -- Configure node tolerations.
    tolerations: [ ]

    # -- If you want to add lifecycle hooks.
    lifecycle: ""
    # lifecycle: |
    #   preStop:
    #     exec:
    #       command: [...]

    # -- Set automounting of the SA token
    automountServiceAccountToken: true

    # -- Set sharing process namespace
    shareProcessNamespace: false

    # -- Specify the serviceAccountName value.
    # In some situations it is needed to provide specific permissions to Kratos deployments
    # Like for example installing Kratos on a cluster with a PosSecurityPolicy and Istio.
    # Uncomment if it is needed to provide a ServiceAccount for the Kratos deployment.
    serviceAccount:
        # -- Specifies whether a service account should be created
        create: true
        # -- Annotations to add to the service account
        annotations:
            helm.sh/hook-weight: "0"
            helm.sh/hook: "pre-install, pre-upgrade"
            helm.sh/hook-delete-policy: "before-hook-creation"
        # -- The name of the service account to use. If not set and create is true, a name is generated using the fullname template
        name: ""

    # -- Specify pod metadata, this metadata is added directly to the pod, and not higher objects
    podMetadata:
        # -- Extra pod level labels
        labels: { }
        # -- Extra pod level annotations
        annotations: { }

    spec:
        # -- Set job back off limit
        backoffLimit: 10

# -- Configuration of the courier
courier:
    enabled: true

# -- Configuration of the watcher sidecar
watcher:
    enabled: false
    image: oryd/k8s-toolbox:0.0.5
    # -- Path to mounted file, which wil be monitored for changes. eg: /etc/secrets/my-secret/foo
    mountFile: ""
    # -- Specify pod metadata, this metadata is added directly to the pod, and not higher objects
    podMetadata:
        # -- Extra pod level labels
        labels: { }
        # -- Extra pod level annotations
        annotations: { }
    # -- Label key used for managing applications
    watchLabelKey: "ory.sh/watcher"
    # -- Number of revisions kept in history
    revisionHistoryLimit: 5

# -- SQL cleanup cron job configuration
cleanup:
    # -- Enable cleanup of stale database rows by periodically running the cleanup sql command
    enabled: false

    # -- Configure how many records are cleaned per run
    batchSize: 100

    # -- Configure how long to wait between each table cleanup
    sleepTables: 1m0s

    # -- Configure the youngest records to keep
    keepLast: 6h

# -- CronJob configuration
cronjob:
    cleanup:
        # -- Configure how often the cron job is ran
        schedule: "0 */1 * * *"

        # -- Configure the arguments of the entrypoint, overriding the default value
        customArgs: [ ]

        # -- Array of extra envs to be passed to the cronjob. This takes precedence over deployment variables. Kubernetes format is expected
        # - name: FOO
        #   value: BAR
        extraEnv: [ ]

        # -- Set custom cron job level labels
        labels: { }

        # -- Set custom cron job level annotations
        annotations: { }

        # -- Specify pod metadata, this metadata is added directly to the pod, and not higher objects
        podMetadata:
            # -- Extra pod level labels
            labels: { }

            # -- Extra pod level annotations
            annotations: { }

        # -- Configure node labels for pod assignment
        nodeSelector: { }

        # -- Configure node tolerations
        tolerations: [ ]

        # -- Configure node affinity
        affinity: { }

        # -- We usually recommend not to specify default resources and to leave this as a conscious choice for the user.
        #  This also increases chances charts run on environments with little
        #  resources, such as Minikube. If you do want to specify resources, uncomment the following
        #  lines, adjust them as necessary, and remove the curly braces after 'resources:'.
        #  limits:
        #    cpu: 100m
        #    memory: 128Mi
        #  requests:
        #    cpu: 100m
        #  memory: 128Mi
        resources:
            limits: { }
            requests: { }

# -- PodDistributionBudget configuration
pdb:
    enabled: false
    spec:
        minAvailable: ""
        maxUnavailable: ""

# -- Parameters for the Prometheus ServiceMonitor objects.
# Reference: https://docs.openshift.com/container-platform/4.6/rest_api/monitoring_apis/servicemonitor-monitoring-coreos-com-v1.html
serviceMonitor:
    # -- switch to true to enable creating the ServiceMonitor
    enabled: false
    # -- HTTP scheme to use for scraping.
    scheme: http
    # -- Interval at which metrics should be scraped
    scrapeInterval: 60s
    # -- Timeout after which the scrape is ended
    scrapeTimeout: 30s
    # -- Provide additional labels to the ServiceMonitor ressource metadata
    labels: { }
    # -- TLS configuration to use when scraping the endpoint
    tlsConfig: { }

configmap:
    # -- switch to false to prevent checksum annotations being maintained and propogated to the pods
    hashSumEnabled: true
    # -- If you do want to specify annotations for configmap, uncomment the following
    # lines, adjust them as necessary, and remove the curly braces after 'annotations:'.
    annotations: { }

test:
    # -- use a busybox image from another repository
    busybox:
        repository: busybox
        tag: 1

kratos-selfservice-ui-node.yaml

# Default values for kratos-selfservice-ui-node.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

# -- Number of replicas in deployment
replicaCount: 1
# -- Number of revisions kept in history
revisionHistoryLimit: 5
# -- Deployment image settings
image:
    # SELFSERVICE image
    repository: oryd/kratos-selfservice-ui-node
    # -- ORY KRATOS VERSION
    tag: "v0.13.0-20"
    pullPolicy: IfNotPresent

imagePullSecrets: [ ]
nameOverride: ""
fullnameOverride: ""

# -- Application config
config:
    csrfCookieName: "radar"
    secrets: {
        csrf_cookie: unsafe_csrf_cookie_secret
    }

# -- Service configuration
service:
    type: ClusterIP
    # -- The load balancer IP
    loadBalancerIP: ""
    port: 80
    # -- The service port name. Useful to set a custom service port name if it must follow a scheme (e.g. Istio)
    name: http

secret:
    # -- switch to false to prevent creating the secret
    enabled: true
    # -- Provide custom name of existing secret, or custom name of secret to be created
    nameOverride: ""
    # nameOverride: "myCustomSecret"
    # -- Annotations to be added to secret. Annotations are added only when secret is being created. Existing secret will not be modified.
    secretAnnotations:
        # Create the secret before installation, and only then. This saves the secret from regenerating during an upgrade
        # pre-upgrade is needed to upgrade from 0.7.0 to newer. Can be deleted afterwards.
        helm.sh/hook-weight: "0"
        helm.sh/hook: "pre-install, pre-upgrade"
        helm.sh/hook-delete-policy: "before-hook-creation"
        helm.sh/resource-policy: "keep"
    # -- switch to false to prevent checksum annotations being maintained and propogated to the pods
    hashSumEnabled: true

# -- Ingress configration
ingress:
    enabled: true
    className: "nginx"
    annotations:
        nginx.ingress.kubernetes.io/rewrite-target: /$1
        cert-manager.io/cluster-issuer: letsencrypt-prod
        # kubernetes.io/ingress.class: nginx
        # kubernetes.io/tls-acme: "true"
    hosts:
        -   host: dev-server.local.net
            paths:
                -   path: "/kratos-ui/?(.*)"
                    pathType: ImplementationSpecific
    tls:
        -   secretName: radar-kratos-tls
            hosts:
                - dev-server.local.net

# -- Container level security context
securityContext:
    capabilities:
        drop:
            - ALL
    readOnlyRootFilesystem: false
    runAsNonRoot: true
    runAsUser: 10000
    runAsGroup: 10000
    allowPrivilegeEscalation: false
    privileged: false
    seccompProfile:
        type: RuntimeDefault
    seLinuxOptions:
        level: "s0:c123,c456"

# -- Pod level security context
podSecurityContext:
    fsGroupChangePolicy: "OnRootMismatch"
    runAsNonRoot: true
    runAsUser: 10000
    fsGroup: 10000
    runAsGroup: 10000
    seccompProfile:
        type: RuntimeDefault
    sysctls: [ ]
    supplementalGroups: [ ]

# -- Deployment configuration
deployment:
    resources: { }
    #  We usually recommend not to specify default resources and to leave this as a conscious
    #  choice for the user. This also increases chances charts run on environments with little
    #  resources, such as Minikube. If you do want to specify resources, uncomment the following
    #  lines, adjust them as necessary, and remove the curly braces after 'resources:'.
    #  limits:
    #    cpu: 100m
    #    memory: 128Mi
    #  requests:
    #    cpu: 100m
    #  memory: 128Mi

    # -- Array of extra envs to be passed to the deployment. Kubernetes format is expected
    # - name: FOO
    #   value: BAR
    extraEnv: [ ]
    # -- If you want to mount external volume
    # For example, mount a secret containing Certificate root CA to verify database
    # TLS connection.
    extraVolumes: [ ]
    # - name: my-volume
    #   secret:
    #     secretName: my-secret
    extraVolumeMounts: [ ]
    # - name: my-volume
    #   mountPath: /etc/secrets/my-secret
    #   readOnly: true

    # -- Node labels for pod assignment.
    nodeSelector: { }
    # If you do want to specify node labels, uncomment the following
    # lines, adjust them as necessary, and remove the curly braces after 'annotations:'.
    #   foo: bar

    # -- Configure node tolerations.
    tolerations: [ ]

    # -- Configure pod topologySpreadConstraints.
    topologySpreadConstraints: [ ]
    # - maxSkew: 1
    #   topologyKey: topology.kubernetes.io/zone
    #   whenUnsatisfiable: DoNotSchedule
    #   labelSelector:
    #     matchLabels:
    #       app.kubernetes.io/name: kratos-selfservice-ui-node
    #       app.kubernetes.io/instance: kratos-selfservice-ui-node

    # -- Configure pod dnsConfig.
    dnsConfig: { }
    #   options:
    #     - name: "ndots"
    #       value: "1"

    labels: { }
    #      If you do want to specify additional labels, uncomment the following
    #      lines, adjust them as necessary, and remove the curly braces after 'labels:'.
    #      e.g.  type: app

    annotations: { }
    #      If you do want to specify annotations, uncomment the following
    #      lines, adjust them as necessary, and remove the curly braces after 'annotations:'.
    #      e.g.  sidecar.istio.io/rewriteAppHTTPProbers: "true"

    # https://github.com/kubernetes/kubernetes/issues/57601
    automountServiceAccountToken: false

affinity: { }

# -- Set this to ORY Kratos's Admin URL
kratosAdminUrl: "kratos-admin"

# -- Set this to ORY Kratos's public URL
kratosPublicUrl: "kratos-public"

# -- Set this to ORY Kratos's public URL accessible from the outside world.
kratosBrowserUrl: "https://dev-server.local.net/kratos"

# -- The basePath
basePath: ""

# -- The jwksUrl
jwksUrl: ""

projectName: "SecureApp"

test:
    # -- use a busybox image from another repository
    busybox:
        repository: busybox
        tag: 1

logs:

After clicking "sign up"

Selfservice node

{"level":"info","message":"HTTP GET /health/ready","meta":{"req":{"headers":{"accept":"*/*","connection":"close","host":"10.42.0.126:3000","user-agent":"kube-probe/1.24"},"httpVersion":"1.1","method":"GET","originalUrl":"/health/ready","query":{},"url":"/health/ready"},"res":{"statusCode":200},"responseTime":4}}
{"level":"info","message":"HTTP GET /registration","meta":{"req":{"headers":{"accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,nl;q=0.8","cookie":"csrf_token_2d29bbe1dcdc5866b6f304cb2f47b5e5a82679d90c1e86d3a61761bd125478ba=FcAqAkMBSLWqST+5aJ9GphrNFPX4PF8AKEq1n5KCJCE=","host":"dev-server.local.net","referer":"https://dev-server.local.net/kratos-ui/welcome","sec-ch-ua":"\"Not_A Brand\";v=\"8\", \"Chromium\";v=\"120\", \"Microsoft Edge\";v=\"120\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"Windows\"","sec-fetch-dest":"document","sec-fetch-mode":"navigate","sec-fetch-site":"same-origin","sec-fetch-user":"?1","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 Edg/120.0.0.0","x-forwarded-for":"10.42.0.1","x-forwarded-host":"dev-server.local.net","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-scheme":"https","x-real-ip":"10.42.0.1","x-request-id":"19ac877f9cef3255b48a6b3974db11d2","x-scheme":"https"},"httpVersion":"1.1","method":"GET","originalUrl":"/registration","query":{},"url":"/registration"},"res":{"statusCode":303},"responseTime":2}}
Error: connect ECONNREFUSED ::1:80
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1300:16) {
  errno: -111,
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '::1',
  port: 80,
  config: {
    url: 'kratos-public/self-service/registration/flows?id=e49c662f-71e6-41c6-b780-d030efa3c133',
    method: 'get',
    headers: {
      Accept: 'application/json, text/plain, */*',
      Cookie: 'csrf_token_2d29bbe1dcdc5866b6f304cb2f47b5e5a82679d90c1e86d3a61761bd125478ba=FcAqAkMBSLWqST+5aJ9GphrNFPX4PF8AKEq1n5KCJCE=',
      'User-Agent': 'axios/0.21.4'
    },
    transformRequest: [ [Function: transformRequest] ],
    transformResponse: [ [Function: transformResponse] ],
    timeout: 0,
    adapter: [Function: httpAdapter],
    xsrfCookieName: 'XSRF-TOKEN',
    xsrfHeaderName: 'X-XSRF-TOKEN',
    maxContentLength: -1,
    maxBodyLength: -1,
    validateStatus: [Function: validateStatus],
    transitional: {
      silentJSONParsing: true,
      forcedJSONParsing: true,
      clarifyTimeoutError: false
    },
    data: undefined
  },
  request: <ref *1> Writable {
    _writableState: WritableState {
      objectMode: false,
      highWaterMark: 16384,
      finalCalled: false,
      needDrain: false,
      ending: false,
      ended: false,
      finished: false,
      destroyed: false,
      decodeStrings: true,
      defaultEncoding: 'utf8',
      length: 0,
      writing: false,
      corked: 0,
      sync: true,
      bufferProcessing: false,
      onwrite: [Function: bound onwrite],
      writecb: null,
      writelen: 0,
      afterWriteTickInfo: null,
      buffered: [],
      bufferedIndex: 0,
      allBuffers: true,
      allNoop: true,
      pendingcb: 0,
      constructed: true,
      prefinished: false,
      errorEmitted: false,
      emitClose: true,
      autoDestroy: true,
      errored: null,
      closed: false,
      closeEmitted: false,
      [Symbol(kOnFinished)]: []
    },
    _events: [Object: null prototype] {
      response: [Function: handleResponse],
      error: [Function: handleRequestError]
    },
    _eventsCount: 2,
    _maxListeners: undefined,
    _options: {
      maxRedirects: 21,
      maxBodyLength: 10485760,
      protocol: 'http:',
      path: 'kratos-public/self-service/registration/flows?id=e49c662f-71e6-41c6-b780-d030efa3c133',
      method: 'GET',
      headers: [Object],
      agent: undefined,
      agents: [Object],
      auth: undefined,
      hostname: '::1',
      port: null,
      nativeProtocols: [Object],
      pathname: 'kratos-public/self-service/registration/flows',
      search: '?id=e49c662f-71e6-41c6-b780-d030efa3c133'
    },
    _ended: true,
    _ending: true,
    _redirectCount: 0,
    _redirects: [],
    _requestBodyLength: 0,
    _requestBodyBuffers: [],
    _onNativeResponse: [Function (anonymous)],
    _currentRequest: ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 7,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: false,
      _last: true,
      chunkedEncoding: false,
      shouldKeepAlive: false,
      maxRequestsOnConnectionReached: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: false,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      strictContentLength: false,
      _contentLength: 0,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      _closed: false,
      socket: [Socket],
      _header: 'GET kratos-public/self-service/registration/flows?id=e49c662f-71e6-41c6-b780-d030efa3c133 HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'Cookie: csrf_token_2d29bbe1dcdc5866b6f304cb2f47b5e5a82679d90c1e86d3a61761bd125478ba=FcAqAkMBSLWqST+5aJ9GphrNFPX4PF8AKEq1n5KCJCE=\r\n' +
        'User-Agent: axios/0.21.4\r\n' +
        'Host: [::1]\r\n' +
        'Connection: close\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: nop],
      agent: [Agent],
      socketPath: undefined,
      method: 'GET',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      path: 'kratos-public/self-service/registration/flows?id=e49c662f-71e6-41c6-b780-d030efa3c133',
      _ended: false,
      res: null,
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      host: '::1',
      protocol: 'http:',
      _redirectable: [Circular *1],
      [Symbol(kCapture)]: false,
      [Symbol(kBytesWritten)]: 0,
      [Symbol(kEndCalled)]: true,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype],
      [Symbol(kUniqueHeaders)]: null
    },
    _currentUrl: 'kratos-public/self-service/registration/flows?id=e49c662f-71e6-41c6-b780-d030efa3c133',
    [Symbol(kCapture)]: false
  },
  response: undefined,
  isAxiosError: true,
  toJSON: [Function: toJSON]
} Error: connect ECONNREFUSED ::1:80
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1300:16)
[@formatjs/intl] "defaultRichTextElements" was specified but "message" was not pre-compiled.
Please consider using "@formatjs/cli" to pre-compile your messages for performance.
For more details see https://formatjs.io/docs/getting-started/message-distribution
[@formatjs/intl] "defaultRichTextElements" was specified but "message" was not pre-compiled.
Please consider using "@formatjs/cli" to pre-compile your messages for performance.
For more details see https://formatjs.io/docs/getting-started/message-distribution
{"level":"info","message":"HTTP GET /registration?flow=e49c662f-71e6-41c6-b780-d030efa3c133","meta":{"req":{"headers":{"accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,nl;q=0.8","cookie":"csrf_token_2d29bbe1dcdc5866b6f304cb2f47b5e5a82679d90c1e86d3a61761bd125478ba=FcAqAkMBSLWqST+5aJ9GphrNFPX4PF8AKEq1n5KCJCE=","host":"dev-server.local.net","referer":"https://dev-server.local.net/kratos-ui/welcome","sec-ch-ua":"\"Not_A Brand\";v=\"8\", \"Chromium\";v=\"120\", \"Microsoft Edge\";v=\"120\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"Windows\"","sec-fetch-dest":"document","sec-fetch-mode":"navigate","sec-fetch-site":"same-origin","sec-fetch-user":"?1","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 Edg/120.0.0.0","x-forwarded-for":"10.42.0.1","x-forwarded-host":"dev-server.local.net","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-scheme":"https","x-real-ip":"10.42.0.1","x-request-id":"c2c3414c17f22ad78c5ed315fa6011b0","x-scheme":"https"},"httpVersion":"1.1","method":"GET","originalUrl":"/registration?flow=e49c662f-71e6-41c6-b780-d030efa3c133","query":{"flow":"e49c662f-71e6-41c6-b780-d030efa3c133"},"url":"/registration?flow=e49c662f-71e6-41c6-b780-d030efa3c133"},"res":{"statusCode":500},"responseTime":22}}

Kratos:

time=2024-01-16T15:24:16Z level=info msg=completed handling request http_request=map[headers:map[accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7 accept-encoding:gzip, deflate, br accept-language:en-US,en;q=0.9,nl;q=0.8 cookie:Value is sensitive and has been redacted. To see the value set config key "log.leak_sensitive_values = true" or environment variable "LOG_LEAK_SENSITIVE_VALUES=true". referer:https://dev-server.local.net/kratos-ui/welcome sec-ch-ua:"Not_A Brand";v="8", "Chromium";v="120", "Microsoft Edge";v="120" sec-ch-ua-mobile:?0 sec-ch-ua-platform:"Windows" sec-fetch-dest:document sec-fetch-mode:navigate sec-fetch-site:same-origin sec-fetch-user:?1 upgrade-insecure-requests:1 user-agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 Edg/120.0.0.0 x-forwarded-for:10.42.0.1 x-forwarded-host:dev-server.local.net x-forwarded-port:443 x-forwarded-proto:https x-forwarded-scheme:https x-real-ip:10.42.0.1 x-request-id:5181eca90087535057bbad730b2502c0 x-scheme:https] host:dev-server.local.net method:GET path:/self-service/registration/browser query:<nil> remote:10.42.0.16:48572 scheme:http] http_response=map[headers:map[cache-control:private, no-cache, no-store, must-revalidate content-type:text/html; charset=utf-8 location:https://dev-server.local.net/kratos-ui/registration?flow=20a99f65-4005-417e-97ae-1018eb8aa4f0 vary:Origin] size:126 status:303 text_status:See Other took:27.468448ms]
time=2024-01-16T15:24:18Z level=info msg=started handling request http_request=map[headers:map[accept:*/* connection:close user-agent:kube-probe/1.24] host:127.0.0.1 method:GET path:/admin/health/alive query:<nil> remote:10.42.0.1:53428 scheme:http]

[Bdegraaf1234]

I resolved the issue by pointing the kratosPublicUrl and kratosBrowserUrl to the same url: https://dev-server.local.net/kratos.

[nguyenhy]

I just post my issue here with the hope some one might need it.
TLDR:

• it's domain, csrf problem
• this and this login flow might help

I was try to setup microservice out of what ory offer, which is:

• ory/kratos
• ory/kratos-selfservice-ui-node

My setup is:
- db:5432 (postgres)
- public_api: kratos:4433
- admin_api : kratos:4434
- ui_node:3000
- nginx: 8080->80/tcp
  - location: / 	 -> public_api
  - location: /auth  -> ui_node:3000
  - location: /admin -> admin_api

You could find the nginx config at this repo ory/examples

the flow of ui in browser would be:

• User visit http://127.0.0.1:8080/auth/welcome
• User click login button
• User being redirect to http://127.0.0.1:8080/auth/login?flow=<flow-id>
  My problem is that I always got error message at the auth/login page, which is:

{
  "id": "interal_server_error",
  "error": {
    "message": "An internal server error occurred. Please try again later.",
    "debug": {
      "name": "Error",
      "code": "ECONNREFUSED"
    }
  }
}

And you need to take a look at this login flow

The issue happened because the ui_node can't make a axios call to the http://127.0.0.1:8080/self-service/login/flows?id=<flow-id> directly. I think , it because that the nginx port.
so to solve this is update the ui_node env like so:

ORY_SDK_URL='http://auth_kratos:4433'
KRATOS_PUBLIC_URL='http://auth_kratos:4433'
KRATOS_BROWSER_URL='http://127.0.0.1:8080'
KRATOS_ADMIN_URL='http://127.0.0.1:8080/admin'

So now when the ui_node fetching data for login, it use the KRATOS_PUBLIC_URL instead of KRATOS_BROWSER_URL.
Hope this would save some one some time to debug it.

[vinckr]

Thanks for sharing @nguyenhy !

Is there something we can improve in the examples?