Skip to content

Commit

Permalink
Fixing the qwen tokenizer location.
Browse files Browse the repository at this point in the history
Using the chatglm one causes a bug where the "<|endoftext|>" is not
found.
  • Loading branch information
Narsil committed Feb 11, 2024
1 parent 8ab384e commit accbd98
Showing 1 changed file with 1 addition and 3 deletions.
4 changes: 1 addition & 3 deletions candle-examples/examples/qwen/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -235,9 +235,7 @@ fn main() -> Result<()> {
));
let tokenizer_filename = match args.tokenizer_file {
Some(file) => std::path::PathBuf::from(file),
None => api
.model("lmz/candle-chatglm".to_string())
.get("chatglm-tokenizer.json")?,
None => repo.get("tokenizer.json")?,
};
let filenames = match args.weight_files {
Some(files) => files
Expand Down

0 comments on commit accbd98

Please sign in to comment.