Cannot upload eval to langtrace

[gyliu513]

I was doing some test for eval, after inspect eval finished, upload to server failed.

(bedrock) gyliu513@Guangyas-MacBook-Pro langtrace % inspect eval example_eval.py --model openai/gpt-3.5-turbo --log-dir langtracefs://cm4lrz7tq00075jmgkdtlq6w4
Fetching dataset with id: cm4lrz7tq00075jmgkdtlq6w4 from Langtrace
Successfully fetched dataset with id: cm4lrz7tq00075jmgkdtlq6w4 from Langtrace
╭─ example_eval (1 sample): openai/gpt-3.5-turbo ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ total time:                                            0:00:04                                                                                                                        dataset: cm4lrz7tq00075jmgkdtlq6w4 │
│ openai/gpt-3.5-turbo                                   505 tokens [I: 415, O: 90]                                                                                                                                        │
│ openai/gpt-4o                                          97 tokens [I: 90, O: 7]                                                                                                                                           │
│                                                                                                                                                                                                                          │
│ accuracy: 1.0  stderr: 0                                                                                                                                                                                                 │
│                                                                                                                                                                                                                          │
│ Log: langtracefs://cm4lrz7tq00075jmgkdtlq6w4/2024-12-12T16-13-53-05-00_example-eval_VPDWd4DhQD4rhCFnAF7FWv.eval                                                                                                          │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Traceback (most recent call last):
  File "/Users/gyliu513/bedrock/lib/python3.11/site-packages/inspect_ai/_util/file.py", line 82, in file
    f.close()
  File "/Users/gyliu513/bedrock/lib/python3.11/site-packages/langtrace_python_sdk/extensions/langtrace_filesystem.py", line 57, in close
    self.upload_to_server(file_data)
  File "/Users/gyliu513/bedrock/lib/python3.11/site-packages/langtrace_python_sdk/extensions/langtrace_filesystem.py", line 64, in upload_to_server
    log = file_data.decode("utf-8")
          ^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xba in position 10: invalid start byte

(bedrock) gyliu513@Guangyas-MacBook-Pro langtrace % cat example_eval.py
from inspect_ai import Task, task
from inspect_ai.dataset import csv_dataset
from inspect_ai.scorer import model_graded_fact
from inspect_ai.solver import self_critique, generate

@task
def example_eval():
    return Task(
        dataset=csv_dataset("langtracefs://cm4lrz7tq00075jmgkdtlq6w4"),
        plan=[
            generate(),
            self_critique(model="openai/gpt-4o")
        ],
        scorer=model_graded_fact()
    )

@karthikscale3 ^^

[karthikscale3]

@gyliu513 - thanks for reporting, and apologies for the issue. Let me take a look and report back

[karthikscale3]

@gyliu513 - sorry for the delay and thanks for your patience, could you try running by setting INSPECT_LOG_FORMAT=json as env var?

[gyliu513]

@karthikscale3 thanks! It works after setting export INSPECT_LOG_FORMAT=json, does this mean the document needs some update?

[karthikscale3]

INSPECT_LOG_FORMAT

Hi @gyliu513 , great! yes, we updated our docs here - https://docs.langtrace.ai/features/evaluations#how-to-run-evaluations . It's a new feature added by inspect-ai and it does not default to json which is why we need this env var added now.

[vanildo]

Hi, I'm with a @gyliu513 similar issue, I'm receive error when trying to upload evaluation to langtrace:

`JSONDecodeError Traceback (most recent call last)
Cell In[19], line 2
1 # eval_result = example_eval()
----> 2 eval(example_eval(), model="ollama/deepseek-r1:8b", log_dir=dataset_id, log_format="json")
3 # result = eval(example_eval(), model="ollama/deepseek-r1:8b", log_dir='../logs', log_format="json")
4
5 # print(result)

File ~/projetos/cpt-lab/mlflow_crewai/.venv/lib/python3.12/site-packages/inspect_ai/_eval/eval.py:169, in eval(tasks, model, model_base_url, model_args, task_args, sandbox, sandbox_cleanup, solver, tags, trace, display, approval, log_level, log_level_transcript, log_dir, log_format, limit, sample_id, epochs, fail_on_error, debug_errors, message_limit, token_limit, time_limit, working_limit, max_samples, max_tasks, max_subprocesses, max_sandboxes, log_samples, log_images, log_buffer, score, score_display, **kwargs)
164 # resolve eval trace
165 max_tasks, max_samples = init_eval_display(
166 display, trace, max_tasks, max_samples, model
167 )
--> 169 return task_display().run_task_app(
170 main=eval_async(
171 tasks=tasks,
172 model=model,
173 model_base_url=model_base_url,
174 model_args=model_args,
175 task_args=task_args,
176 sandbox=sandbox,
177 sandbox_cleanup=sandbox_cleanup,
178 solver=solver,
179 tags=tags,
180 approval=approval,
181 log_level=log_level,
182 log_level_transcript=log_level_transcript,
183 log_dir=log_dir,
184 log_format=log_format,
185 limit=limit,
186 sample_id=sample_id,
187 epochs=epochs,
188 fail_on_error=fail_on_error,
189 debug_errors=debug_errors,
190 message_limit=message_limit,
191 token_limit=token_limit,
192 time_limit=time_limit,
193 working_limit=working_limit,
194 max_samples=max_samples,
195 max_tasks=max_tasks,
196 max_subprocesses=max_subprocesses,
197 max_sandboxes=max_sandboxes,
198 log_samples=log_samples,
199 log_images=log_images,
200 log_buffer=log_buffer,
201 score=score,
202 score_display=score_display,
203 **kwargs,
204 )
205 )

File ~/projetos/cpt-lab/mlflow_crewai/.venv/lib/python3.12/site-packages/inspect_ai/_display/rich/display.py:78, in RichDisplay.run_task_app(self, main)
76 @OverRide
77 def run_task_app(self, main: Coroutine[Any, Any, TR]) -> TR:
---> 78 return asyncio.run(main)

File ~/projetos/cpt-lab/mlflow_crewai/.venv/lib/python3.12/site-packages/nest_asyncio.py:30, in _patch_asyncio..run(main, debug)
28 task = asyncio.ensure_future(main)
29 try:
---> 30 return loop.run_until_complete(task)
31 finally:
32 if not task.done():

File ~/projetos/cpt-lab/mlflow_crewai/.venv/lib/python3.12/site-packages/nest_asyncio.py:98, in _patch_loop..run_until_complete(self, future)
95 if not f.done():
96 raise RuntimeError(
97 'Event loop stopped before Future completed.')
---> 98 return f.result()

File ~/miniforge3/lib/python3.12/asyncio/futures.py:203, in Future.result(self)
201 self.__log_traceback = False
202 if self._exception is not None:
--> 203 raise self._exception.with_traceback(self._exception_tb)
204 return self._result

File ~/miniforge3/lib/python3.12/asyncio/tasks.py:314, in Task.__step_run_and_handle_result(failed resolving arguments)
310 try:
311 if exc is None:
312 # We use the send method directly, because coroutines
313 # don't have __iter__ and __next__ methods.
--> 314 result = coro.send(None)
315 else:
316 result = coro.throw(exc)

File ~/projetos/cpt-lab/mlflow_crewai/.venv/lib/python3.12/site-packages/inspect_ai/_eval/eval.py:375, in eval_async(tasks, model, model_base_url, model_args, task_args, sandbox, sandbox_cleanup, solver, tags, approval, log_level, log_level_transcript, log_dir, log_format, limit, sample_id, epochs, fail_on_error, debug_errors, message_limit, token_limit, time_limit, working_limit, max_samples, max_tasks, max_subprocesses, max_sandboxes, log_samples, log_images, log_buffer, score, score_display, **kwargs)
373 log_dir = absolute_file_path(log_dir)
374 recorder = create_recorder_for_format(log_format or DEFAULT_LOG_FORMAT, log_dir)
--> 375 if not recorder.is_writeable():
376 raise PrerequisiteError(
377 f"ERROR: You do not have write permission for the log_dir '{log_dir}'"
378 )
380 # resolve solver

File ~/projetos/cpt-lab/mlflow_crewai/.venv/lib/python3.12/site-packages/inspect_ai/log/_recorders/file.py:33, in FileRecorder.is_writeable(self)
31 @OverRide
32 def is_writeable(self) -> bool:
---> 33 return self.fs.is_writeable(self.log_dir)

File ~/projetos/cpt-lab/mlflow_crewai/.venv/lib/python3.12/site-packages/inspect_ai/_util/file.py:229, in FileSystem.is_writeable(self, path)
227 path = path.rstrip("/\")
228 touch_file = f"{path}{self.fs.sep}{uuid()}"
--> 229 self.touch(touch_file)
230 self.rm(touch_file)
231 return True

File ~/projetos/cpt-lab/mlflow_crewai/.venv/lib/python3.12/site-packages/inspect_ai/_util/file.py:174, in FileSystem.touch(self, path)
173 def touch(self, path: str) -> None:
--> 174 self.fs.touch(path)

File ~/projetos/cpt-lab/mlflow_crewai/.venv/lib/python3.12/site-packages/fsspec/spec.py:1342, in AbstractFileSystem.touch(self, path, truncate, **kwargs)
1331 """Create empty file, or update timestamp
1332
1333 Parameters
(...) 1339 leave file unchanged, if backend allows this
1340 """
1341 if truncate or not self.exists(path):
-> 1342 with self.open(path, "wb", **kwargs):
1343 pass
1344 else:

File ~/projetos/cpt-lab/mlflow_crewai/.venv/lib/python3.12/site-packages/langtrace_python_sdk/extensions/langtrace_filesystem.py:57, in LangTraceFile.close(self)
54 self.fs.files[self.path] = file_data
56 # Upload the file to the remote server
---> 57 self.upload_to_server(file_data)
59 super().close()

File ~/projetos/cpt-lab/mlflow_crewai/.venv/lib/python3.12/site-packages/langtrace_python_sdk/extensions/langtrace_filesystem.py:65, in LangTraceFile.upload_to_server(self, file_data)
62 try:
63 # Parse the log file and upload it to the server
64 log = file_data.decode("utf-8")
---> 65 eval_log = json.loads(log)
66 data = {
67 "runId": eval_log["eval"]["run_id"],
68 "taskId": eval_log["eval"]["task_id"],
69 "log": log,
70 }
71 if self.path is not None:

File ~/miniforge3/lib/python3.12/json/init.py:346, in loads(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
341 s = s.decode(detect_encoding(s), 'surrogatepass')
343 if (cls is None and object_hook is None and
344 parse_int is None and parse_float is None and
345 parse_constant is None and object_pairs_hook is None and not kw):
--> 346 return _default_decoder.decode(s)
347 if cls is None:
348 cls = JSONDecoder

File ~/miniforge3/lib/python3.12/json/decoder.py:337, in JSONDecoder.decode(self, s, _w)
332 def decode(self, s, _w=WHITESPACE.match):
333 """Return the Python representation of s (a str instance
334 containing a JSON document).
335
336 """
--> 337 obj, end = self.raw_decode(s, idx=_w(s, 0).end())
338 end = _w(s, end).end()
339 if end != len(s):

File ~/miniforge3/lib/python3.12/json/decoder.py:355, in JSONDecoder.raw_decode(self, s, idx)
353 obj, end = self.scan_once(s, idx)
354 except StopIteration as err:
--> 355 raise JSONDecodeError("Expecting value", s, err.value) from None
356 return obj, end

JSONDecodeError: Expecting value: line 1 column 1 (char 0)`

Can you give some guidance?

thank you in advance.

[karthikscale3]

Hi @vanildo , we will take a look and get back to you soon. Thanks for your patience