Adding artifacts upload through S3/Minio #71

rkube · 2025-11-29T16:59:49Z

This PR adds support for uploading artifacts to an S3 bucket associated with a run.

Example usage:

using MLFlowClient
using Minio

s3_cfg = MinioConfig("http://192.168.1.2:9000"; username=ENV["AWS_ACCESS_KEY_ID"], password=ENV["AWS_SECRET_ACCESS_KEY"])

mlf = MLFlow("http://192.168.1.2:5000/api")
experiment_id = getexperimentbyname(mlf, "julia-tests")
exprun = createrun(mlf, experiment_id)

logartifact(s3_cfg, exprun, "scratch/example.png", "example.png")

The PR also

Changes the behavior of createrun to add the current time if start_time is missing.
Handles missing values of data["output"] when creating a run.

I do not know how to test logartifact except that "it works on my machine". I've been using it a bit to log pngs and seems to do fine.

pebeto

Thanks for taking the initiative to make this library support remote artifact logging in S3.
I can't accept these changes until you implement a valid test suite. You can use BrokenRecord.jl to reproduce HTML requests.

ext/logger.jl

rkube · 2025-11-30T17:31:43Z

What about a test that creates an experiment and logs a text file and a figure.
Unit tests would be

Files exist on the bucket
Files are the same as uploaded
These would require a usable S3 configuration during test time. This would be in line with the unit tests that require a usable mlflow server.

Another question: Should the s3 logging be a package extension? There are multiple ways to use S3, for example through minio (which I have setup and can test), or vanilla AWS S3.

For testing, it may be useful to define a reproducible test environment with mlflow and minio (S3) in a docker container.

pebeto · 2025-11-30T20:14:43Z

The described tests are okay.

Is it possible to avoid adding a new package dependency by calling the API directly? Mostly to avoid pulling more from registries.

It's okay to setup a MiniO instance in the CI pipeline, but I don't consider it practical for local testing. Let's see recording the request response and mocking it using BrokenRecord.jl. If that's not possible, we can think about adding one more development dependency.

rkube · 2025-11-30T20:29:42Z

Cool, I'll set up the tests.

Do you mean not adding using Minio, AWSS3 and FileTypes?

I looked into BrokenRecord.jl, but I think it's a bit cumbersome. Doing it this way requires to manually handle traffic logging and keep track of the files. It's also locked in to my particular setup.

A more comfortable way would be to define the extend the .devcontainer/compose.yaml.
That way it's trivial to test other versions of Minio and mlflow, just change the image to the correct version.

I'm trying out the setup below right now, it seems to be fine for local use. Minio doesn't add too much overhead.

❯ cat docker-compose.test.yaml
version: '3.8'

services:
  minio:
    image: minio/minio
    ports:
      - "9000:9000"    # S3 API
      - "9001:9001"    # Minio Console
    environment:
      - MINIO_ROOT_USER=minioadmin
      - MINIO_ROOT_PASSWORD=minioadmin
    command: server /data --console-address ":9001"

  create-buckets:
    image: minio/mc
    depends_on:
      - minio
    entrypoint: >
      /bin/sh -c "
      /usr/bin/mc alias set myminio http://minio:9000 minioadmin minioadmin;
      /usr/bin/mc mb myminio/mlflow;
      exit 0;
      "

  mlflow:
    image: ghcr.io/mlflow/mlflow:v3.6.0
    depends_on:
      - create-buckets
    ports:
      - "5050:5050"
    environment:
      - AWS_ACCESS_KEY_ID=minioadmin
      - AWS_SECRET_ACCESS_KEY=minioadmin
      - MLFLOW_S3_ENDPOINT_URL=http://minio:9000
    command: >
      mlflow server
      --host 0.0.0.0
      --port 5050
      --default-artifact-root s3://mlflow/artifacts
      ```
     
     These instances can then be used for all tests. Start the stack through
     ```
     docker-compose -f .devcontainer/compose.yaml up
     ```
     and stop it through
     ```
     docker-compose -f .devcontainer/compose.yaml down
     ```.

rkube · 2025-11-30T23:11:52Z

Ok, I got the unit tests passing with the docker file above. But with this setup, using mlflow-3.6.0, other unit tests are failing:

The routes used in services/user.jl - users/create fail. They are also not mentioned in the REST API docs: https://mlflow.org/docs/3.6.0/api_reference/rest-api.html
The tests in the end of "services/registered_model.jl" with call to createuser: "create/get/update/delete registered model permission"

create registered model permission: Error During Test at /Users/ralph/source/repos/MLFlowClient.jl/test/services/registered_model.jl:180
  Got exception outside of a @test
  Unexpected character
  Line: 0
  Around: ...<!doctype html> <html ...
              ^

  Stacktrace:
    [1] error(s::String)
      @ Base ./error.jl:44
    [2] _error(message::String, ps::JSON.Parser.MemoryParserState)
      @ JSON.Parser ~/.julia/packages/JSON/93Ea8/src/Parser.jl:140
    [3] parse_jsconstant(::JSON.Parser.ParserContext{Dict{String, Any}, Int64, true, nothing}, ps::JSON.Parser.MemoryParserState)
      @ JSON.Parser ~/.julia/packages/JSON/93Ea8/src/Parser.jl:193
    [4] parse_value(pc::JSON.Parser.ParserContext{Dict{String, Any}, Int64, true, nothing}, ps::JSON.Parser.MemoryParserState)
      @ JSON.Parser ~/.julia/packages/JSON/93Ea8/src/Parser.jl:170
    [5] parse(str::String; dicttype::Type, inttype::Type{Int64}, allownan::Bool, null::Nothing)
      @ JSON.Parser ~/.julia/packages/JSON/93Ea8/src/Parser.jl:450
    [6] parse
      @ ~/.julia/packages/JSON/93Ea8/src/Parser.jl:443 [inlined]
    [7] |>(x::String, f::typeof(JSON.Parser.parse))
      @ Base ./operators.jl:972
    [8] mlfpost(mlf::MLFlow, endpoint::String; kwargs::@Kwargs{username::String, password::String})
      @ MLFlowClient ~/source/repos/MLFlowClient.jl/src/api.jl:63
    [9] mlfpost
      @ ~/source/repos/MLFlowClient.jl/src/api.jl:54 [inlined]
   [10] createuser(instance::MLFlow, username::String, password::String)
      @ MLFlowClient ~/source/repos/MLFlowClient.jl/src/services/user.jl:13
   [11] top-level scope
      @ ~/source/repos/MLFlowClient.jl/test/services/registered_model.jl:181
   [12] macro expansion
      @ ~/.julia/juliaup/julia-1.12.2+0.aarch64.apple.darwin14/share/julia/stdlib/v1.12/Test/src/Test.jl:1776 [inlined]
   [13] macro expansion
      @ ~/source/repos/MLFlowClient.jl/test/services/registered_model.jl:184 [inlined]
   [14] include(mapexpr::Function, mod::Module, _path::String)
      @ Base ./Base.jl:307
   [15] top-level scope
      @ ~/source/repos/MLFlowClient.jl/test/runtests.jl:39
   [16] include(mod::Module, _path::String)
      @ Base ./Base.jl:306
   [17] exec_options(opts::Base.JLOptions)
      @ Base ./client.jl:317
   [18] _start()
      @ Base ./client.jl:550

  caused by: HTTP.Exceptions.StatusError(404, "POST", "/api/2.0/mlflow/users/create", HTTP.Messages.Response:
  """
  HTTP/1.1 404 Not Found
  date: Sun, 30 Nov 2025 23:09:21 GMT
  server: uvicorn
  content-type: text/html; charset=utf-8
  content-length: 207
  x-content-type-options: nosniff
  x-frame-options: SAMEORIGIN

  """)
  Stacktrace:
    [1] (::HTTP.ConnectionRequest.var"#connections#connectionlayer##0"{HTTP.ConnectionRequest.var"#connections#1#connectionlayer##1"{HTTP.TimeoutRequest.var"#timeouts#timeoutlayer##0"{HTTP.TimeoutRequest.var"#timeouts#1#timeoutlayer##1"{HTTP.ExceptionRequest.var"#exceptions#exceptionlayer##0"{HTTP.ExceptionRequest.var"#exceptions#1#exceptionlayer##1"{typeof(HTTP.StreamRequest.streamlayer)}}}}}})(req::HTTP.Messages.Request; proxy::Nothing, socket_type::Type, socket_type_tls::Nothing, readtimeout::Int64, connect_timeout::Int64, logerrors::Bool, logtag::Nothing, closeimmediately::Bool, kw::@Kwargs{iofunction::Nothing, decompress::Nothing, verbose::Int64})
      @ HTTP.ConnectionRequest ~/.julia/packages/HTTP/ShTJs/src/clientlayers/ConnectionRequest.jl:144
    [2] connections
      @ ~/.julia/packages/HTTP/ShTJs/src/clientlayers/ConnectionRequest.jl:60 [inlined]
    [3] (::Base.var"#46#47"{Base.var"#48#49"{ExponentialBackOff, HTTP.RetryRequest.var"#retrylayer##2#retrylayer##3"{Int64, typeof(HTTP.RetryRequest.FALSE), HTTP.Messages.Request, Base.RefValue{Int64}}, HTTP.ConnectionRequest.var"#connections#connectionlayer##0"{HTTP.ConnectionRequest.var"#connections#1#connectionlayer##1"{HTTP.TimeoutRequest.var"#timeouts#timeoutlayer##0"{HTTP.TimeoutRequest.var"#timeouts#1#timeoutlayer##1"{HTTP.ExceptionRequest.var"#exceptions#exceptionlayer##0"{HTTP.ExceptionRequest.var"#exceptions#1#exceptionlayer##1"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}})(args::HTTP.Messages.Request; kwargs::@Kwargs{iofunction::Nothing, decompress::Nothing, verbose::Int64})
      @ Base ./error.jl:309
    [4] (::HTTP.RetryRequest.var"#manageretries#retrylayer##0"{HTTP.RetryRequest.var"#manageretries#1#retrylayer##1"{HTTP.ConnectionRequest.var"#connections#connectionlayer##0"{HTTP.ConnectionRequest.var"#connections#1#connectionlayer##1"{HTTP.TimeoutRequest.var"#timeouts#timeoutlayer##0"{HTTP.TimeoutRequest.var"#timeouts#1#timeoutlayer##1"{HTTP.ExceptionRequest.var"#exceptions#exceptionlayer##0"{HTTP.ExceptionRequest.var"#exceptions#1#exceptionlayer##1"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}})(req::HTTP.Messages.Request; retry::Bool, retries::Int64, retry_delays::ExponentialBackOff, retry_check::Function, retry_non_idempotent::Bool, kw::@Kwargs{iofunction::Nothing, decompress::Nothing, verbose::Int64})
      @ HTTP.RetryRequest ~/.julia/packages/HTTP/ShTJs/src/clientlayers/RetryRequest.jl:75
    [5] manageretries
      @ ~/.julia/packages/HTTP/ShTJs/src/clientlayers/RetryRequest.jl:30 [inlined]
    [6] (::HTTP.CookieRequest.var"#managecookies#cookielayer##0"{HTTP.CookieRequest.var"#managecookies#1#cookielayer##1"{HTTP.RetryRequest.var"#manageretries#retrylayer##0"{HTTP.RetryRequest.var"#manageretries#1#retrylayer##1"{HTTP.ConnectionRequest.var"#connections#connectionlayer##0"{HTTP.ConnectionRequest.var"#connections#1#connectionlayer##1"{HTTP.TimeoutRequest.var"#timeouts#timeoutlayer##0"{HTTP.TimeoutRequest.var"#timeouts#1#timeoutlayer##1"{HTTP.ExceptionRequest.var"#exceptions#exceptionlayer##0"{HTTP.ExceptionRequest.var"#exceptions#1#exceptionlayer##1"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}})(req::HTTP.Messages.Request; cookies::Bool, cookiejar::HTTP.Cookies.CookieJar, kw::@Kwargs{iofunction::Nothing, decompress::Nothing, verbose::Int64})
      @ HTTP.CookieRequest ~/.julia/packages/HTTP/ShTJs/src/clientlayers/CookieRequest.jl:42
    [7] managecookies
      @ ~/.julia/packages/HTTP/ShTJs/src/clientlayers/CookieRequest.jl:19 [inlined]
    [8] (::HTTP.HeadersRequest.var"#defaultheaders#headerslayer##0"{HTTP.HeadersRequest.var"#defaultheaders#1#headerslayer##1"{HTTP.CookieRequest.var"#managecookies#cookielayer##0"{HTTP.CookieRequest.var"#managecookies#1#cookielayer##1"{HTTP.RetryRequest.var"#manageretries#retrylayer##0"{HTTP.RetryRequest.var"#manageretries#1#retrylayer##1"{HTTP.ConnectionRequest.var"#connections#connectionlayer##0"{HTTP.ConnectionRequest.var"#connections#1#connectionlayer##1"{HTTP.TimeoutRequest.var"#timeouts#timeoutlayer##0"{HTTP.TimeoutRequest.var"#timeouts#1#timeoutlayer##1"{HTTP.ExceptionRequest.var"#exceptions#exceptionlayer##0"{HTTP.ExceptionRequest.var"#exceptions#1#exceptionlayer##1"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}})(req::HTTP.Messages.Request; iofunction::Nothing, decompress::Nothing, basicauth::Bool, detect_content_type::Bool, canonicalize_headers::Bool, kw::@Kwargs{verbose::Int64})
      @ HTTP.HeadersRequest ~/.julia/packages/HTTP/ShTJs/src/clientlayers/HeadersRequest.jl:71
    [9] defaultheaders
      @ ~/.julia/packages/HTTP/ShTJs/src/clientlayers/HeadersRequest.jl:14 [inlined]
   [10] (::HTTP.RedirectRequest.var"#redirects#redirectlayer##0"{HTTP.RedirectRequest.var"#redirects#1#redirectlayer##1"{HTTP.HeadersRequest.var"#defaultheaders#headerslayer##0"{HTTP.HeadersRequest.var"#defaultheaders#1#headerslayer##1"{HTTP.CookieRequest.var"#managecookies#cookielayer##0"{HTTP.CookieRequest.var"#managecookies#1#cookielayer##1"{HTTP.RetryRequest.var"#manageretries#retrylayer##0"{HTTP.RetryRequest.var"#manageretries#1#retrylayer##1"{HTTP.ConnectionRequest.var"#connections#connectionlayer##0"{HTTP.ConnectionRequest.var"#connections#1#connectionlayer##1"{HTTP.TimeoutRequest.var"#timeouts#timeoutlayer##0"{HTTP.TimeoutRequest.var"#timeouts#1#timeoutlayer##1"{HTTP.ExceptionRequest.var"#exceptions#exceptionlayer##0"{HTTP.ExceptionRequest.var"#exceptions#1#exceptionlayer##1"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}})(req::HTTP.Messages.Request; redirect::Bool, redirect_limit::Int64, redirect_method::Nothing, forwardheaders::Bool, response_stream::Nothing, kw::@Kwargs{verbose::Int64})
      @ HTTP.RedirectRequest ~/.julia/packages/HTTP/ShTJs/src/clientlayers/RedirectRequest.jl:25
   [11] redirects
      @ ~/.julia/packages/HTTP/ShTJs/src/clientlayers/RedirectRequest.jl:14 [inlined]
   [12] (::HTTP.MessageRequest.var"#makerequest#messagelayer##0"{HTTP.MessageRequest.var"#makerequest#1#messagelayer##1"{HTTP.RedirectRequest.var"#redirects#redirectlayer##0"{HTTP.RedirectRequest.var"#redirects#1#redirectlayer##1"{HTTP.HeadersRequest.var"#defaultheaders#headerslayer##0"{HTTP.HeadersRequest.var"#defaultheaders#1#headerslayer##1"{HTTP.CookieRequest.var"#managecookies#cookielayer##0"{HTTP.CookieRequest.var"#managecookies#1#cookielayer##1"{HTTP.RetryRequest.var"#manageretries#retrylayer##0"{HTTP.RetryRequest.var"#manageretries#1#retrylayer##1"{HTTP.ConnectionRequest.var"#connections#connectionlayer##0"{HTTP.ConnectionRequest.var"#connections#1#connectionlayer##1"{HTTP.TimeoutRequest.var"#timeouts#timeoutlayer##0"{HTTP.TimeoutRequest.var"#timeouts#1#timeoutlayer##1"{HTTP.ExceptionRequest.var"#exceptions#exceptionlayer##0"{HTTP.ExceptionRequest.var"#exceptions#1#exceptionlayer##1"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}}}})(method::String, url::URI, headers::Dict{String, String}, body::String; copyheaders::Bool, response_stream::Nothing, http_version::HTTP.Strings.HTTPVersion, verbose::Int64, kw::@Kwargs{})
      @ HTTP.MessageRequest ~/.julia/packages/HTTP/ShTJs/src/clientlayers/MessageRequest.jl:35
   [13] makerequest
      @ ~/.julia/packages/HTTP/ShTJs/src/clientlayers/MessageRequest.jl:24 [inlined]
   [14] request(stack::HTTP.MessageRequest.var"#makerequest#messagelayer##0"{HTTP.MessageRequest.var"#makerequest#1#messagelayer##1"{HTTP.RedirectRequest.var"#redirects#redirectlayer##0"{HTTP.RedirectRequest.var"#redirects#1#redirectlayer##1"{HTTP.HeadersRequest.var"#defaultheaders#headerslayer##0"{HTTP.HeadersRequest.var"#defaultheaders#1#headerslayer##1"{HTTP.CookieRequest.var"#managecookies#cookielayer##0"{HTTP.CookieRequest.var"#managecookies#1#cookielayer##1"{HTTP.RetryRequest.var"#manageretries#retrylayer##0"{HTTP.RetryRequest.var"#manageretries#1#retrylayer##1"{HTTP.ConnectionRequest.var"#connections#connectionlayer##0"{HTTP.ConnectionRequest.var"#connections#1#connectionlayer##1"{HTTP.TimeoutRequest.var"#timeouts#timeoutlayer##0"{HTTP.TimeoutRequest.var"#timeouts#1#timeoutlayer##1"{HTTP.ExceptionRequest.var"#exceptions#exceptionlayer##0"{HTTP.ExceptionRequest.var"#exceptions#1#exceptionlayer##1"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}}}}, method::String, url::URI, h::Dict{String, String}, b::String, q::Nothing; headers::Dict{String, String}, body::String, query::Nothing, kw::@Kwargs{})
      @ HTTP ~/.julia/packages/HTTP/ShTJs/src/HTTP.jl:457
   [15] request(stack::Function, method::String, url::URI, h::Dict{String, String}, b::String, q::Nothing)
      @ HTTP ~/.julia/packages/HTTP/ShTJs/src/HTTP.jl:455
   [16] #request#21
      @ ~/.julia/packages/HTTP/ShTJs/src/HTTP.jl:315 [inlined]
   [17] request(method::String, url::URI, h::Dict{String, String}, b::String)
      @ HTTP ~/.julia/packages/HTTP/ShTJs/src/HTTP.jl:313
   [18] post(::URI, ::Vararg{Any}; kw::@Kwargs{})
      @ HTTP ~/.julia/packages/HTTP/ShTJs/src/HTTP.jl:532
   [19] post
      @ ~/.julia/packages/HTTP/ShTJs/src/HTTP.jl:532 [inlined]
   [20] mlfpost(mlf::MLFlow, endpoint::String; kwargs::@Kwargs{username::String, password::String})
      @ MLFlowClient ~/source/repos/MLFlowClient.jl/src/api.jl:60
   [21] mlfpost
      @ ~/source/repos/MLFlowClient.jl/src/api.jl:54 [inlined]
   [22] createuser(instance::MLFlow, username::String, password::String)
      @ MLFlowClient ~/source/repos/MLFlowClient.jl/src/services/user.jl:13
   [23] top-level scope
      @ ~/source/repos/MLFlowClient.jl/test/services/registered_model.jl:181
   [24] macro expansion
      @ ~/.julia/juliaup/julia-1.12.2+0.aarch64.apple.darwin14/share/julia/stdlib/v1.12/Test/src/Test.jl:1776 [inlined]
   [25] macro expansion
      @ ~/source/repos/MLFlowClient.jl/test/services/registered_model.jl:184 [inlined]
   [26] include(mapexpr::Function, mod::Module, _path::String)
      @ Base ./Base.jl:307
   [27] top-level scope
      @ ~/source/repos/MLFlowClient.jl/test/runtests.jl:39
   [28] include(mod::Module, _path::String)
      @ Base ./Base.jl:306
   [29] exec_options(opts::Base.JLOptions)
      @ Base ./client.jl:317
   [30] _start()
      @ Base ./client.jl:550

  caused by: HTTP.Exceptions.StatusError(404, "POST", "/api/2.0/mlflow/users/create", HTTP.Messages.Response:
  """
  HTTP/1.1 404 Not Found
  date: Sun, 30 Nov 2025 23:09:21 GMT
  server: uvicorn
  content-type: text/html; charset=utf-8
  content-length: 207
  x-content-type-options: nosniff
  x-frame-options: SAMEORIGIN

  """)
  Stacktrace:
    [1] (::HTTP.ExceptionRequest.var"#exceptions#exceptionlayer##0"{HTTP.ExceptionRequest.var"#exceptions#1#exceptionlayer##1"{typeof(HTTP.StreamRequest.streamlayer)}})(stream::HTTP.Streams.Stream{HTTP.Messages.Response, HTTP.Connections.Connection{Sockets.TCPSocket}}; status_exception::Bool, timedout::Nothing, logerrors::Bool, logtag::Nothing, kw::@Kwargs{iofunction::Nothing, decompress::Nothing, verbose::Int64})
      @ HTTP.ExceptionRequest ~/.julia/packages/HTTP/ShTJs/src/clientlayers/ExceptionRequest.jl:19
    [2] exceptions
      @ ~/.julia/packages/HTTP/ShTJs/src/clientlayers/ExceptionRequest.jl:13 [inlined]
    [3] (::HTTP.TimeoutRequest.var"#timeouts#timeoutlayer##0"{HTTP.TimeoutRequest.var"#timeouts#1#timeoutlayer##1"{HTTP.ExceptionRequest.var"#exceptions#exceptionlayer##0"{HTTP.ExceptionRequest.var"#exceptions#1#exceptionlayer##1"{typeof(HTTP.StreamRequest.streamlayer)}}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Response, HTTP.Connections.Connection{Sockets.TCPSocket}}; readtimeout::Int64, logerrors::Bool, logtag::Nothing, kw::@Kwargs{iofunction::Nothing, decompress::Nothing, verbose::Int64})
      @ HTTP.TimeoutRequest ~/.julia/packages/HTTP/ShTJs/src/clientlayers/TimeoutRequest.jl:18
    [4] (::HTTP.ConnectionRequest.var"#connections#connectionlayer##0"{HTTP.ConnectionRequest.var"#connections#1#connectionlayer##1"{HTTP.TimeoutRequest.var"#timeouts#timeoutlayer##0"{HTTP.TimeoutRequest.var"#timeouts#1#timeoutlayer##1"{HTTP.ExceptionRequest.var"#exceptions#exceptionlayer##0"{HTTP.ExceptionRequest.var"#exceptions#1#exceptionlayer##1"{typeof(HTTP.StreamRequest.streamlayer)}}}}}})(req::HTTP.Messages.Request; proxy::Nothing, socket_type::Type, socket_type_tls::Nothing, readtimeout::Int64, connect_timeout::Int64, logerrors::Bool, logtag::Nothing, closeimmediately::Bool, kw::@Kwargs{iofunction::Nothing, decompress::Nothing, verbose::Int64})
      @ HTTP.ConnectionRequest ~/.julia/packages/HTTP/ShTJs/src/clientlayers/ConnectionRequest.jl:122
    [5] connections
      @ ~/.julia/packages/HTTP/ShTJs/src/clientlayers/ConnectionRequest.jl:60 [inlined]
    [6] (::Base.var"#46#47"{Base.var"#48#49"{ExponentialBackOff, HTTP.RetryRequest.var"#retrylayer##2#retrylayer##3"{Int64, typeof(HTTP.RetryRequest.FALSE), HTTP.Messages.Request, Base.RefValue{Int64}}, HTTP.ConnectionRequest.var"#connections#connectionlayer##0"{HTTP.ConnectionRequest.var"#connections#1#connectionlayer##1"{HTTP.TimeoutRequest.var"#timeouts#timeoutlayer##0"{HTTP.TimeoutRequest.var"#timeouts#1#timeoutlayer##1"{HTTP.ExceptionRequest.var"#exceptions#exceptionlayer##0"{HTTP.ExceptionRequest.var"#exceptions#1#exceptionlayer##1"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}})(args::HTTP.Messages.Request; kwargs::@Kwargs{iofunction::Nothing, decompress::Nothing, verbose::Int64})
      @ Base ./error.jl:309
    [7] (::HTTP.RetryRequest.var"#manageretries#retrylayer##0"{HTTP.RetryRequest.var"#manageretries#1#retrylayer##1"{HTTP.ConnectionRequest.var"#connections#connectionlayer##0"{HTTP.ConnectionRequest.var"#connections#1#connectionlayer##1"{HTTP.TimeoutRequest.var"#timeouts#timeoutlayer##0"{HTTP.TimeoutRequest.var"#timeouts#1#timeoutlayer##1"{HTTP.ExceptionRequest.var"#exceptions#exceptionlayer##0"{HTTP.ExceptionRequest.var"#exceptions#1#exceptionlayer##1"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}})(req::HTTP.Messages.Request; retry::Bool, retries::Int64, retry_delays::ExponentialBackOff, retry_check::Function, retry_non_idempotent::Bool, kw::@Kwargs{iofunction::Nothing, decompress::Nothing, verbose::Int64})
      @ HTTP.RetryRequest ~/.julia/packages/HTTP/ShTJs/src/clientlayers/RetryRequest.jl:75
    [8] manageretries
      @ ~/.julia/packages/HTTP/ShTJs/src/clientlayers/RetryRequest.jl:30 [inlined]
    [9] (::HTTP.CookieRequest.var"#managecookies#cookielayer##0"{HTTP.CookieRequest.var"#managecookies#1#cookielayer##1"{HTTP.RetryRequest.var"#manageretries#retrylayer##0"{HTTP.RetryRequest.var"#manageretries#1#retrylayer##1"{HTTP.ConnectionRequest.var"#connections#connectionlayer##0"{HTTP.ConnectionRequest.var"#connections#1#connectionlayer##1"{HTTP.TimeoutRequest.var"#timeouts#timeoutlayer##0"{HTTP.TimeoutRequest.var"#timeouts#1#timeoutlayer##1"{HTTP.ExceptionRequest.var"#exceptions#exceptionlayer##0"{HTTP.ExceptionRequest.var"#exceptions#1#exceptionlayer##1"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}})(req::HTTP.Messages.Request; cookies::Bool, cookiejar::HTTP.Cookies.CookieJar, kw::@Kwargs{iofunction::Nothing, decompress::Nothing, verbose::Int64})
      @ HTTP.CookieRequest ~/.julia/packages/HTTP/ShTJs/src/clientlayers/CookieRequest.jl:42
   [10] managecookies
      @ ~/.julia/packages/HTTP/ShTJs/src/clientlayers/CookieRequest.jl:19 [inlined]
   [11] (::HTTP.HeadersRequest.var"#defaultheaders#headerslayer##0"{HTTP.HeadersRequest.var"#defaultheaders#1#headerslayer##1"{HTTP.CookieRequest.var"#managecookies#cookielayer##0"{HTTP.CookieRequest.var"#managecookies#1#cookielayer##1"{HTTP.RetryRequest.var"#manageretries#retrylayer##0"{HTTP.RetryRequest.var"#manageretries#1#retrylayer##1"{HTTP.ConnectionRequest.var"#connections#connectionlayer##0"{HTTP.ConnectionRequest.var"#connections#1#connectionlayer##1"{HTTP.TimeoutRequest.var"#timeouts#timeoutlayer##0"{HTTP.TimeoutRequest.var"#timeouts#1#timeoutlayer##1"{HTTP.ExceptionRequest.var"#exceptions#exceptionlayer##0"{HTTP.ExceptionRequest.var"#exceptions#1#exceptionlayer##1"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}})(req::HTTP.Messages.Request; iofunction::Nothing, decompress::Nothing, basicauth::Bool, detect_content_type::Bool, canonicalize_headers::Bool, kw::@Kwargs{verbose::Int64})
      @ HTTP.HeadersRequest ~/.julia/packages/HTTP/ShTJs/src/clientlayers/HeadersRequest.jl:71
   [12] defaultheaders
      @ ~/.julia/packages/HTTP/ShTJs/src/clientlayers/HeadersRequest.jl:14 [inlined]
   [13] (::HTTP.RedirectRequest.var"#redirects#redirectlayer##0"{HTTP.RedirectRequest.var"#redirects#1#redirectlayer##1"{HTTP.HeadersRequest.var"#defaultheaders#headerslayer##0"{HTTP.HeadersRequest.var"#defaultheaders#1#headerslayer##1"{HTTP.CookieRequest.var"#managecookies#cookielayer##0"{HTTP.CookieRequest.var"#managecookies#1#cookielayer##1"{HTTP.RetryRequest.var"#manageretries#retrylayer##0"{HTTP.RetryRequest.var"#manageretries#1#retrylayer##1"{HTTP.ConnectionRequest.var"#connections#connectionlayer##0"{HTTP.ConnectionRequest.var"#connections#1#connectionlayer##1"{HTTP.TimeoutRequest.var"#timeouts#timeoutlayer##0"{HTTP.TimeoutRequest.var"#timeouts#1#timeoutlayer##1"{HTTP.ExceptionRequest.var"#exceptions#exceptionlayer##0"{HTTP.ExceptionRequest.var"#exceptions#1#exceptionlayer##1"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}})(req::HTTP.Messages.Request; redirect::Bool, redirect_limit::Int64, redirect_method::Nothing, forwardheaders::Bool, response_stream::Nothing, kw::@Kwargs{verbose::Int64})
      @ HTTP.RedirectRequest ~/.julia/packages/HTTP/ShTJs/src/clientlayers/RedirectRequest.jl:25
   [14] redirects
      @ ~/.julia/packages/HTTP/ShTJs/src/clientlayers/RedirectRequest.jl:14 [inlined]
   [15] (::HTTP.MessageRequest.var"#makerequest#messagelayer##0"{HTTP.MessageRequest.var"#makerequest#1#messagelayer##1"{HTTP.RedirectRequest.var"#redirects#redirectlayer##0"{HTTP.RedirectRequest.var"#redirects#1#redirectlayer##1"{HTTP.HeadersRequest.var"#defaultheaders#headerslayer##0"{HTTP.HeadersRequest.var"#defaultheaders#1#headerslayer##1"{HTTP.CookieRequest.var"#managecookies#cookielayer##0"{HTTP.CookieRequest.var"#managecookies#1#cookielayer##1"{HTTP.RetryRequest.var"#manageretries#retrylayer##0"{HTTP.RetryRequest.var"#manageretries#1#retrylayer##1"{HTTP.ConnectionRequest.var"#connections#connectionlayer##0"{HTTP.ConnectionRequest.var"#connections#1#connectionlayer##1"{HTTP.TimeoutRequest.var"#timeouts#timeoutlayer##0"{HTTP.TimeoutRequest.var"#timeouts#1#timeoutlayer##1"{HTTP.ExceptionRequest.var"#exceptions#exceptionlayer##0"{HTTP.ExceptionRequest.var"#exceptions#1#exceptionlayer##1"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}}}})(method::String, url::URI, headers::Dict{String, String}, body::String; copyheaders::Bool, response_stream::Nothing, http_version::HTTP.Strings.HTTPVersion, verbose::Int64, kw::@Kwargs{})
      @ HTTP.MessageRequest ~/.julia/packages/HTTP/ShTJs/src/clientlayers/MessageRequest.jl:35
   [16] makerequest
      @ ~/.julia/packages/HTTP/ShTJs/src/clientlayers/MessageRequest.jl:24 [inlined]
   [17] request(stack::HTTP.MessageRequest.var"#makerequest#messagelayer##0"{HTTP.MessageRequest.var"#makerequest#1#messagelayer##1"{HTTP.RedirectRequest.var"#redirects#redirectlayer##0"{HTTP.RedirectRequest.var"#redirects#1#redirectlayer##1"{HTTP.HeadersRequest.var"#defaultheaders#headerslayer##0"{HTTP.HeadersRequest.var"#defaultheaders#1#headerslayer##1"{HTTP.CookieRequest.var"#managecookies#cookielayer##0"{HTTP.CookieRequest.var"#managecookies#1#cookielayer##1"{HTTP.RetryRequest.var"#manageretries#retrylayer##0"{HTTP.RetryRequest.var"#manageretries#1#retrylayer##1"{HTTP.ConnectionRequest.var"#connections#connectionlayer##0"{HTTP.ConnectionRequest.var"#connections#1#connectionlayer##1"{HTTP.TimeoutRequest.var"#timeouts#timeoutlayer##0"{HTTP.TimeoutRequest.var"#timeouts#1#timeoutlayer##1"{HTTP.ExceptionRequest.var"#exceptions#exceptionlayer##0"{HTTP.ExceptionRequest.var"#exceptions#1#exceptionlayer##1"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}}}}, method::String, url::URI, h::Dict{String, String}, b::String, q::Nothing; headers::Dict{String, String}, body::String, query::Nothing, kw::@Kwargs{})
      @ HTTP ~/.julia/packages/HTTP/ShTJs/src/HTTP.jl:457
   [18] request(stack::Function, method::String, url::URI, h::Dict{String, String}, b::String, q::Nothing)
      @ HTTP ~/.julia/packages/HTTP/ShTJs/src/HTTP.jl:455
   [19] #request#21
      @ ~/.julia/packages/HTTP/ShTJs/src/HTTP.jl:315 [inlined]
   [20] request(method::String, url::URI, h::Dict{String, String}, b::String)
      @ HTTP ~/.julia/packages/HTTP/ShTJs/src/HTTP.jl:313
   [21] post(::URI, ::Vararg{Any}; kw::@Kwargs{})
      @ HTTP ~/.julia/packages/HTTP/ShTJs/src/HTTP.jl:532
   [22] post
      @ ~/.julia/packages/HTTP/ShTJs/src/HTTP.jl:532 [inlined]
   [23] mlfpost(mlf::MLFlow, endpoint::String; kwargs::@Kwargs{username::String, password::String})
      @ MLFlowClient ~/source/repos/MLFlowClient.jl/src/api.jl:60
   [24] mlfpost
      @ ~/source/repos/MLFlowClient.jl/src/api.jl:54 [inlined]
   [25] createuser(instance::MLFlow, username::String, password::String)
      @ MLFlowClient ~/source/repos/MLFlowClient.jl/src/services/user.jl:13
   [26] top-level scope
      @ ~/source/repos/MLFlowClient.jl/test/services/registered_model.jl:181
   [27] macro expansion
      @ ~/.julia/juliaup/julia-1.12.2+0.aarch64.apple.darwin14/share/julia/stdlib/v1.12/Test/src/Test.jl:1776 [inlined]
   [28] macro expansion
      @ ~/source/repos/MLFlowClient.jl/test/services/registered_model.jl:184 [inlined]
   [29] include(mapexpr::Function, mod::Module, _path::String)
      @ Base ./Base.jl:307
   [30] top-level scope
      @ ~/source/repos/MLFlowClient.jl/test/runtests.jl:39
   [31] include(mod::Module, _path::String)
      @ Base ./Base.jl:306
   [32] exec_options(opts::Base.JLOptions)
      @ Base ./client.jl:317
   [33] _start()
      @ Base ./client.jl:550
Test Summary:                      | Error  Total  Time
create registered model permission |     1      1  0.6s
RNG of the outermost testset: Random.Xoshiro(0xe7de5a19f795a603, 0x96177932e0ad11f8, 0x4160ec669e169dff, 0x8db72b612fdf047c, 0xd10b935bff27114a)
ERROR: LoadError: Some tests did not pass: 0 passed, 0 failed, 1 errored, 0 broken.
in expression starting at /Users/ralph/source/repos/MLFlowClient.jl/test/services/registered_model.jl:180
in expression starting at /Users/ralph/source/repos/MLFlowClient.jl/test/runtests.jl:39

rkube · 2025-12-01T04:07:53Z

Oh, the unit tests probably fail because the docker image is not using auth https://mlflow.org/docs/latest/self-hosting/security/basic-http-auth/

pebeto · 2025-12-01T04:14:48Z

Do you mean not adding using Minio, AWSS3 and FileTypes?

Yes, because those operations are just compatible REST API calls, right?

The docker-compose way to setup the development environment is great. Thanks to set it up!

The routes used in services/user.jl - users/create fail. They are also not mentioned in the REST API docs: https://mlflow.org/docs/3.6.0/api_reference/rest-api.html

The Authentication API is documented here, and to make it work you need to configure the instance as the CI pipeline does

pebeto · 2025-12-01T04:22:16Z

.github/workflows/CI.yml

-          python3 /opt/hostedtoolcache/Python/3.12.3/x64/bin/mlflow server --app-name basic-auth --host 0.0.0.0 --port 5000 &
-          sleep 5
+      - name: Start services
+        run: docker-compose -f docker compose.test.yaml up -d


The custom mlflow setup is related to the way lts and release CI pipelines treat the instance. Instead of changing it, you could setup MiniO using minio-action.

pebeto · 2025-12-01T04:23:16Z

.github/workflows/CI.yml

      - uses: julia-actions/julia-buildpkg@v1
      - uses: julia-actions/julia-runtest@v1
        env:
-          JULIA_NUM_THREADS: '2'


Is this change required? Also the URI must not be modified.

pebeto · 2025-12-01T04:26:04Z

test/assets/julia.png

What about generating an image instead of saving one to the repository? You can add Plots.jl to the test suite, create a plot and save it as a figure under execution.

pebeto · 2025-12-01T04:29:08Z

test/runtests.jl

+`docker-compose .devcontainers/compose.yaml up`.
+
+Then set the environment variables
+MLFLOW_TRACKING_URI="http://localhost:5050/api"


In the case we setup the development environment as required, this message won't be necessarily. I mean, it's okay to warn the user to run the tests with the docker setup.

pebeto · 2025-12-01T04:30:40Z

test/services/experiment.jl

#71 (comment)

pebeto · 2025-12-01T04:30:56Z

test/services/registered_model.jl

#71 (comment)

rkube · 2025-12-01T14:45:44Z

Thanks for the review. I had some unit tests commented out because I didn't have mlflow[auth] set up in docker. It's set up now and all unit tests pass.

I'll take a closer look at the minio-action you suggested above. Also the changes like the threads are just me not knowing what is going on yet, I'll revert them.

Regarding the plots.jl test-suite:
This PR implements function logartifact(s3_cfg::MinioConfig, run::run, path::String="", artifact_name::String="").

For figures (either Plots.jl or Makie.jl) a package extension would be cool. That way neither plotting package is a direct dependency for MLFlowClient.jl. They would need to be added to the Test dependencies though.

…lizing ENVs in tests

rkube · 2025-12-06T16:39:01Z

Ok, all commits on this should be included in this PR (used 'git pull origin fetch').
All unit tests (including auth etc.) pass locally when the docker-compose.test.yaml stack is running.
For gitlab CI, I'm suggesting to use rclone serve s3 instead of minio. That is more light-weight than running minio and ok for only a single client. But I have no idea how to test the CI.yaml without pushing. I tried running locally with act, but run into errors.

Am I missing anything?

Adding artifacts upload through S3/Minio

a9b53d5

pebeto self-requested a review November 30, 2025 04:40

pebeto added the enhancement New feature or request label Nov 30, 2025

pebeto requested changes Nov 30, 2025

View reviewed changes

ext/logger.jl Outdated Show resolved Hide resolved

rkube force-pushed the artifacts branch from b5be208 to a9b53d5 Compare November 30, 2025 17:18

rkube added 3 commits November 30, 2025 19:33

Added unit testing with containerized mlflow/minio

5322504

fixing CI.yml

e61470d

fixing CI.yml

c8c1e3e

pebeto requested changes Dec 1, 2025

View reviewed changes

Adding authentication to docker file.

ebfcaee

pebeto and others added 3 commits December 1, 2025 15:31

Fixing CI pipeline, generating image for artifact testing, and genera…

b3dc479

…lizing ENVs in tests

Changing mlflow port in CI pipeline to pair with test suite

e7b047f

Using rclone as s3 server in workflows

b617b1d

Adding artifacts upload through S3/Minio #71

Are you sure you want to change the base?

Adding artifacts upload through S3/Minio #71

Uh oh!

Conversation

rkube commented Nov 29, 2025

Uh oh!

pebeto left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rkube commented Nov 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pebeto commented Nov 30, 2025

Uh oh!

rkube commented Nov 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rkube commented Nov 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rkube commented Dec 1, 2025

Uh oh!

pebeto commented Dec 1, 2025

Uh oh!

pebeto Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

pebeto Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

pebeto Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

pebeto Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

pebeto Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

pebeto Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

rkube commented Dec 1, 2025

Uh oh!

rkube commented Dec 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

rkube commented Nov 30, 2025 •

edited

Loading

rkube commented Nov 30, 2025 •

edited

Loading

rkube commented Nov 30, 2025 •

edited

Loading

rkube commented Dec 6, 2025 •

edited

Loading