add trust_remote_code to tokenizer loading

loubnabnl · web-flow · commit b28574b97641 · 2023-05-03T09:16:15.000+02:00
some tokenizers require running local code hence the need for trust_remote_code argument
diff --git a/main.py b/main.py
@@ -183,6 +183,7 @@ def main():
         tokenizer = AutoTokenizer.from_pretrained(
             args.model,
             revision=args.revision,
+            trust_remote_code=args.trust_remote_code,
             use_auth_token=args.use_auth_token,
             truncation_side="left",
         )

Original file line number	Diff line number	Diff line change
`@@ -183,6 +183,7 @@ def main():`
`183`	`183`	`tokenizer = AutoTokenizer.from_pretrained(`
`184`	`184`	`args.model,`
`185`	`185`	`revision=args.revision,`
	`186`	`+ trust_remote_code=args.trust_remote_code,`
`186`	`187`	`use_auth_token=args.use_auth_token,`
`187`	`188`	`truncation_side="left",`
`188`	`189`	`)`