Help with grammars, beginner

[MichaelWkTran]

I wanted to use grammars with Llama. I have slightly modified the "simple" example and modified the following segment from line 125:

// initialize the sampler

auto sparams = llama_sampler_chain_default_params();
llama_sampler* smpl = llama_sampler_chain_init(sparams);
llama_sampler_chain_add(smpl, llama_sampler_init_min_p(0.05f, 1));
llama_sampler_chain_add(smpl, llama_sampler_init_temp(0.8));
llama_sampler_chain_add(smpl, llama_sampler_init_dist(LLAMA_DEFAULT_SEED));
llama_sampler_chain_add(smpl, llama_sampler_init_grammar(vocab, "root ::= emotion\nemotion ::= \"happy\" | \"sad\" | \"angry\"", "root"));

My prompt is "Charlie yelled with the first of a hurricane. Charlie is ".

I thought given the gbnf I gave that the next word outputted is "Angry". However I get the error D:\Projects\llama.cpp\src\llama-grammar.cpp:1138: GGML_ASSERT(!grammar.stacks.empty()) failed. Did I misunderstand on how grammars work or is there something wrong with my code?

[ggerganov]

Try to put the grammar at the start of the chain.

Also, in general you need to accept all tokens from the prompt by calling the llama_sampler_accept() function for each token. This will initialize the state of the grammar sampler with the prompt. Probably not needed for this particular case, but still better to do it.

[MichaelWkTran]

Thankyou for your reponse, moving the grammar to the start of the chain resolved the issue. I did get an error can calling llama_sampler_accept (__debugbreak() statement or a similar call) was executed in TestLlama.exe. I found that the function already gets called when using llama_sampler_sample() when looking at its definition.

[ggerganov]

Yes, it is called. But I meant that you should also call it for the input prompt tokens.

[MichaelWkTran]

The error of statement or a similar call was executed was caused because of this code directly after adding the sampler chains.

// print the prompt token-by-token

for (auto id : prompt_tokens) {
    char buf[128];
    llama_sampler_accept(smpl, id);
    int n = llama_token_to_piece(vocab, id, buf, sizeof(buf), 0, true);
    if (n < 0) {
        fprintf(stderr, "%s: error: failed to convert token to piece\n", __func__);
        return 1;
    }
    std::string s(buf, n);
    printf("%s", s.c_str());
}

Is this not what you meant? What is the right way to do this?