Alpaca_+_Mistral_7b_full_example.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/kaballas/AutoGPT/blob/master/Alpaca_%2B_Mistral_7b_full_example.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "To run this, press \"*Runtime*\" and press \"*Run all*\" on a **free** Tesla T4 Google Colab instance!\n",
        "<div class=\"align-center\">\n",
        "  <a href=\"https://github.com/unslothai/unsloth\"><img src=\"https://github.com/unslothai/unsloth/raw/main/images/unsloth%20new%20logo.png\" width=\"115\"></a>\n",
        "  <a href=\"https://discord.gg/u54VK8m8tk\"><img src=\"https://github.com/unslothai/unsloth/raw/main/images/Discord button.png\" width=\"145\"></a>\n",
        "  <a href=\"https://ko-fi.com/unsloth\"><img src=\"https://github.com/unslothai/unsloth/raw/main/images/Kofi button.png\" width=\"145\"></a></a> Join Discord if you need help + support us if you can!\n",
        "</div>\n",
        "\n",
        "To install Unsloth on your own computer, follow the installation instructions on our Github page [here](https://github.com/unslothai/unsloth#installation-instructions---conda).\n",
        "\n",
        "You will learn how to do [data prep](#Data), how to [train](#Train), how to [run the model](#Inference), & [how to save it](#Save) (eg for Llama.cpp)."
      ],
      "metadata": {
        "id": "IqM-T1RTzY6C"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "2eSvM9zX_2d3"
      },
      "outputs": [],
      "source": [
        "%%capture\n",
        "# Installs Unsloth, Xformers (Flash Attention) and all other packages!\n",
        "!pip install \"unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git\"\n",
        "!pip install --no-deps xformers trl peft accelerate bitsandbytes"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "* We support Llama, Mistral, CodeLlama, TinyLlama, Vicuna, Open Hermes etc\n",
        "* And Yi, Qwen ([llamafied](https://huggingface.co/models?sort=trending&search=qwen+llama)), Deepseek, all Llama, Mistral derived archs.\n",
        "* We support 16bit LoRA or 4bit QLoRA. Both 2x faster.\n",
        "* `max_seq_length` can be set to anything, since we do automatic RoPE Scaling via [kaiokendev's](https://kaiokendev.github.io/til) method.\n",
        "* With [PR 26037](https://github.com/huggingface/transformers/pull/26037), we support downloading 4bit models **4x faster**! [Our repo](https://huggingface.co/unsloth) has Llama, Mistral 4bit models.\n",
        "* [**NEW**] We make Gemma 6 trillion tokens **2.5x faster**! See our [Gemma notebook](https://colab.research.google.com/drive/10NbwlsRChbma1v55m8LAPYG15uQv6HLo?usp=sharing)"
      ],
      "metadata": {
        "id": "r2v_X2fA0Df5"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QmUBVEnvCDJv",
        "outputId": "cd9fd0a2-9726-4e4b-d936-9b7d0bca67cd"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "==((====))==  Unsloth: Fast Mistral patching release 2024.6\n",
            "   \\\\   /|    GPU: Tesla T4. Max memory: 14.748 GB. Platform = Linux.\n",
            "O^O/ \\_/ \\    Pytorch: 2.3.0+cu121. CUDA = 7.5. CUDA Toolkit = 12.1.\n",
            "\\        /    Bfloat16 = FALSE. Xformers = 0.0.26.post1. FA = False.\n",
            " \"-____-\"     Free Apache license: http://github.com/unslothai/unsloth\n"
          ]
        }
      ],
      "source": [
        "from unsloth import FastLanguageModel\n",
        "import torch\n",
        "max_seq_length = 2048 # Choose any! We auto support RoPE Scaling internally!\n",
        "dtype = None # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+\n",
        "load_in_4bit = True # Use 4bit quantization to reduce memory usage. Can be False.\n",
        "\n",
        "# 4bit pre quantized models we support for 4x faster downloading + no OOMs.\n",
        "fourbit_models = [\n",
        "    \"unsloth/mistral-7b-bnb-4bit\",\n",
        "    \"unsloth/mistral-7b-instruct-v0.2-bnb-4bit\",\n",
        "    \"unsloth/llama-2-7b-bnb-4bit\",\n",
        "    \"unsloth/llama-2-13b-bnb-4bit\",\n",
        "    \"unsloth/codellama-34b-bnb-4bit\",\n",
        "    \"unsloth/tinyllama-bnb-4bit\",\n",
        "    \"unsloth/gemma-7b-bnb-4bit\", # New Google 6 trillion tokens model 2.5x faster!\n",
        "    \"unsloth/gemma-2b-bnb-4bit\",\n",
        "] # More models at https://huggingface.co/unsloth\n",
        "\n",
        "model, tokenizer = FastLanguageModel.from_pretrained(\n",
        "    model_name = \"unsloth/mistral-7b-bnb-4bit\", # Choose ANY! eg teknium/OpenHermes-2.5-Mistral-7B\n",
        "    max_seq_length = max_seq_length,\n",
        "    dtype = dtype,\n",
        "    load_in_4bit = load_in_4bit,\n",
        "    # token = \"hf_...\", # use one if using gated models like meta-llama/Llama-2-7b-hf\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We now add LoRA adapters so we only need to update 1 to 10% of all parameters!"
      ],
      "metadata": {
        "id": "SXd9bTZd1aaL"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {
        "id": "6bZsfBuZDeCL"
      },
      "outputs": [],
      "source": [
        "model = FastLanguageModel.get_peft_model(\n",
        "    model,\n",
        "    r = 16, # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128\n",
        "    target_modules = [\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\",\n",
        "                      \"gate_proj\", \"up_proj\", \"down_proj\",],\n",
        "    lora_alpha = 16,\n",
        "    lora_dropout = 0, # Supports any, but = 0 is optimized\n",
        "    bias = \"none\",    # Supports any, but = \"none\" is optimized\n",
        "    # [NEW] \"unsloth\" uses 30% less VRAM, fits 2x larger batch sizes!\n",
        "    use_gradient_checkpointing = \"unsloth\", # True or \"unsloth\" for very long context\n",
        "    random_state = 3407,\n",
        "    use_rslora = False,  # We support rank stabilized LoRA\n",
        "    loftq_config = None, # And LoftQ\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "<a name=\"Data\"></a>\n",
        "### Data Prep\n",
        "We now use the Alpaca dataset from [yahma](https://huggingface.co/datasets/yahma/alpaca-cleaned), which is a filtered version of 52K of the original [Alpaca dataset](https://crfm.stanford.edu/2023/03/13/alpaca.html). You can replace this code section with your own data prep.\n",
        "\n",
        "**[NOTE]** To train only on completions (ignoring the user's input) read TRL's docs [here](https://huggingface.co/docs/trl/sft_trainer#train-on-completions-only).\n",
        "\n",
        "**[NOTE]** Remember to add the **EOS_TOKEN** to the tokenized output!! Otherwise you'll get infinite generations!\n",
        "\n",
        "If you want to use the `ChatML` template for ShareGPT datasets, try our conversational [notebook](https://colab.research.google.com/drive/1Aau3lgPzeZKQ-98h69CCu1UJcvIBLmy2?usp=sharing).\n",
        "\n",
        "For text completions like novel writing, try this [notebook](https://colab.research.google.com/drive/1ef-tab5bhkvWmBOObepl1WgJvfvSzn5Q?usp=sharing)."
      ],
      "metadata": {
        "id": "vITh0KVJ10qX"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 48,
      "metadata": {
        "id": "LjY75GoYUCB8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "10889c64-c494-47fa-8b0e-8b8aa89973cc"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Dataset({\n",
            "    features: ['questions', 'answers'],\n",
            "    num_rows: 6276\n",
            "})\n",
            "Dataset({\n",
            "    features: ['questions', 'answers', 'text'],\n",
            "    num_rows: 6276\n",
            "})\n",
            "Entry 1:\n",
            "Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n",
            "\n",
            "### Instruction:\n",
            "\"To Answer the question \"\n",
            "\n",
            "### Input:\n",
            "What is the purpose of the Parallel Pay Run Test Strategy document?\n",
            "\n",
            "### Response:\n",
            "The purpose of the Parallel Pay Run Test Strategy document is to outline the approach for testing the HRMIS Program.[SEP]\n",
            "\n",
            "Entry 2:\n",
            "Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n",
            "\n",
            "### Instruction:\n",
            "\"To Answer the question \"\n",
            "\n",
            "### Input:\n",
            "Why is the term 'Aboriginal' used in preference to 'Aboriginal and Torres Strait Islander' in Western Australia?\n",
            "\n",
            "### Response:\n",
            "To recognize that Aboriginal people are the original inhabitants of Western Australia.[SEP]\n",
            "\n",
            "Entry 3:\n",
            "Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n",
            "\n",
            "### Instruction:\n",
            "\"To Answer the question \"\n",
            "\n",
            "### Input:\n",
            "What is the purpose of the document?\n",
            "\n",
            "### Response:\n",
            "The purpose of the document is to outline the test strategy for the Parallel Pay Run.[SEP]\n",
            "\n",
            "Entry 4:\n",
            "Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n",
            "\n",
            "### Instruction:\n",
            "\"To Answer the question \"\n",
            "\n",
            "### Input:\n",
            "What is the purpose of the HRMIS Program?\n",
            "\n",
            "### Response:\n",
            "The purpose of the HRMIS Program is to replace the legacy HR/payroll system, the rostering system and nurse placement service with a single, reliable, accessible, and contemporary solution.[SEP]\n",
            "\n",
            "Entry 5:\n",
            "Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n",
            "\n",
            "### Instruction:\n",
            "\"To Answer the question \"\n",
            "\n",
            "### Input:\n",
            "What is the purpose of the HRMIS Program?\n",
            "\n",
            "### Response:\n",
            "The purpose of the HRMIS Program is to implement a solution that will allow HSS to deliver a more reliable and efficient service, be user-friendly for staff, support workforce reporting and decision-making, and reduce administrative burden.[SEP]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "from transformers import BertTokenizer\n",
        "from datasets import load_dataset\n",
        "\n",
        "# Initialize the tokenizer\n",
        "tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')\n",
        "EOS_TOKEN = '[SEP]'  # Use a custom EOS token, or '[SEP]' which is commonly used in BERT\n",
        "\n",
        "# Define the Alpaca prompt template\n",
        "alpaca_prompt = \"\"\"Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n",
        "\n",
        "### Instruction:\n",
        "\"To Answer the question \"\n",
        "\n",
        "### Input:\n",
        "{}\n",
        "\n",
        "### Response:\n",
        "{}\"\"\"\n",
        "\n",
        "# Define the formatting function\n",
        "def formatting_prompts_func(examples):\n",
        "    inputs = examples[\"questions\"]\n",
        "    outputs = examples[\"answers\"]\n",
        "    texts = []\n",
        "    for input_text, output_text in zip(inputs, outputs):\n",
        "        # Skip NoneType entries\n",
        "        if input_text is None or output_text is None:\n",
        "            continue\n",
        "        # Add EOS_TOKEN to ensure generation doesn't go on forever\n",
        "        text = alpaca_prompt.format(input_text, output_text) + EOS_TOKEN\n",
        "        texts.append(text)\n",
        "    return {\"text\": texts}\n",
        "\n",
        "# Load the dataset\n",
        "try:\n",
        "    dataset = load_dataset(\"Kaballas/HRMIS_MASTER\", split=\"train\")\n",
        "    print(dataset)\n",
        "\n",
        "    # Apply the formatting function to the dataset\n",
        "    dataset = dataset.map(formatting_prompts_func, batched=True)\n",
        "    print(dataset)\n",
        "\n",
        "    # Display a few entries from the formatted dataset\n",
        "    for i in range(5):\n",
        "        print(f\"Entry {i+1}:\")\n",
        "        print(dataset[i]['text'])\n",
        "        print()\n",
        "\n",
        "except Exception as e:\n",
        "    print(f\"An error occurred: {e}\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "<a name=\"Train\"></a>\n",
        "### Train the model\n",
        "Now let's use Huggingface TRL's `SFTTrainer`! More docs here: [TRL SFT docs](https://huggingface.co/docs/trl/sft_trainer). We do 60 steps to speed things up, but you can set `num_train_epochs=1` for a full run, and turn off `max_steps=None`. We also support TRL's `DPOTrainer`!"
      ],
      "metadata": {
        "id": "idAEIeSQ3xdS"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 51,
      "metadata": {
        "id": "95_Nn-89DhsL",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c24f7279-aca6-434b-a140-6c62b34335d8"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/transformers/training_args.py:1965: FutureWarning: `--push_to_hub_token` is deprecated and will be removed in version 5 of 🤗 Transformers. Use `--hub_token` instead.\n",
            "  warnings.warn(\n",
            "/usr/local/lib/python3.10/dist-packages/trl/trainer/sft_trainer.py:269: UserWarning: You passed a `max_seq_length` argument to the SFTTrainer, the value you passed will override the one in the `SFTConfig`.\n",
            "  warnings.warn(\n",
            "/usr/local/lib/python3.10/dist-packages/trl/trainer/sft_trainer.py:283: UserWarning: You passed a `dataset_num_proc` argument to the SFTTrainer, the value you passed will override the one in the `SFTConfig`.\n",
            "  warnings.warn(\n",
            "/usr/local/lib/python3.10/dist-packages/trl/trainer/sft_trainer.py:307: UserWarning: You passed a `dataset_text_field` argument to the SFTTrainer, the value you passed will override the one in the `SFTConfig`.\n",
            "  warnings.warn(\n",
            "max_steps is given, it will override any value given in num_train_epochs\n"
          ]
        }
      ],
      "source": [
        "from trl import SFTTrainer\n",
        "from transformers import TrainingArguments\n",
        "from unsloth import is_bfloat16_supported\n",
        "\n",
        "# Check if bfloat16 is supported\n",
        "bfloat16_supported = is_bfloat16_supported()\n",
        "\n",
        "# Define training arguments\n",
        "training_args = TrainingArguments(\n",
        "    per_device_train_batch_size=2,\n",
        "    gradient_accumulation_steps=4,\n",
        "    warmup_steps=5,\n",
        "    max_steps=60,  # Set num_train_epochs=1 for full training runs\n",
        "    learning_rate=2e-4,\n",
        "    fp16=not bfloat16_supported,\n",
        "    bf16=bfloat16_supported,\n",
        "    logging_steps=1,\n",
        "    optim=\"adamw_8bit\",\n",
        "    weight_decay=0.01,\n",
        "    lr_scheduler_type=\"linear\",\n",
        "    seed=3407,\n",
        "    output_dir=\"outputs\",\n",
        ")\n",
        "\n",
        "# Initialize the trainer\n",
        "trainer = SFTTrainer(\n",
        "    model=model,\n",
        "    tokenizer=tokenizer,\n",
        "    train_dataset=dataset,\n",
        "    dataset_text_field=\"text\",\n",
        "    max_seq_length=max_seq_length,\n",
        "    dataset_num_proc=2,\n",
        "    packing=False,  # Can make training 5x faster for short sequences\n",
        "    args=training_args,\n",
        ")\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 52,
      "metadata": {
        "id": "2ejIt2xSNKKp",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "cellView": "form",
        "outputId": "54f5732f-f8a5-4b6a-9140-6ce88a9bc01c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "GPU = Tesla T4. Max memory = 14.748 GB.\n",
            "9.215 GB of memory reserved.\n"
          ]
        }
      ],
      "source": [
        "#@title Show current memory stats\n",
        "gpu_stats = torch.cuda.get_device_properties(0)\n",
        "start_gpu_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\n",
        "max_memory = round(gpu_stats.total_memory / 1024 / 1024 / 1024, 3)\n",
        "print(f\"GPU = {gpu_stats.name}. Max memory = {max_memory} GB.\")\n",
        "print(f\"{start_gpu_memory} GB of memory reserved.\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "yqxqAZ7KJ4oL",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "855ada8d-86ac-488c-9553-f3bdb3de9a11"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "🦥 Unsloth: Will patch your computer to enable 2x faster free finetuning.\n",
            "==((====))==  Unsloth: Fast Mistral patching release 2024.6\n",
            "   \\\\   /|    GPU: Tesla T4. Max memory: 14.748 GB. Platform = Linux.\n",
            "O^O/ \\_/ \\    Pytorch: 2.3.0+cu121. CUDA = 7.5. CUDA Toolkit = 12.1.\n",
            "\\        /    Bfloat16 = FALSE. Xformers = 0.0.26.post1. FA = False.\n",
            " \"-____-\"     Free Apache license: http://github.com/unslothai/unsloth\n",
            "Unsloth: Offloading input_embeddings to disk to save VRAM\n",
            "Unsloth: Offloading output_embeddings to disk to save VRAM\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Unsloth 2024.6 patched 32 layers with 32 QKV layers, 32 O layers and 32 MLP layers.\n",
            "/usr/local/lib/python3.10/dist-packages/transformers/training_args.py:1965: FutureWarning: `--push_to_hub_token` is deprecated and will be removed in version 5 of 🤗 Transformers. Use `--hub_token` instead.\n",
            "  warnings.warn(\n",
            "/usr/local/lib/python3.10/dist-packages/trl/trainer/sft_trainer.py:269: UserWarning: You passed a `max_seq_length` argument to the SFTTrainer, the value you passed will override the one in the `SFTConfig`.\n",
            "  warnings.warn(\n",
            "/usr/local/lib/python3.10/dist-packages/trl/trainer/sft_trainer.py:283: UserWarning: You passed a `dataset_num_proc` argument to the SFTTrainer, the value you passed will override the one in the `SFTConfig`.\n",
            "  warnings.warn(\n",
            "/usr/local/lib/python3.10/dist-packages/trl/trainer/sft_trainer.py:307: UserWarning: You passed a `dataset_text_field` argument to the SFTTrainer, the value you passed will override the one in the `SFTConfig`.\n",
            "  warnings.warn(\n",
            "max_steps is given, it will override any value given in num_train_epochs\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Unsloth: Casting embed_tokens to float32\n",
            "Unsloth: Casting lm_head to float32\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "==((====))==  Unsloth - 2x faster free finetuning | Num GPUs = 1\n",
            "   \\\\   /|    Num examples = 6,276 | Num Epochs = 1\n",
            "O^O/ \\_/ \\    Batch size per device = 2 | Gradient Accumulation steps = 4\n",
            "\\        /    Total batch size = 8 | Total steps = 60\n",
            " \"-____-\"     Number of trainable parameters = 304,087,040\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "\n",
              "    <div>\n",
              "      \n",
              "      <progress value='60' max='60' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
              "      [60/60 04:55, Epoch 0/1]\n",
              "    </div>\n",
              "    <table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              " <tr style=\"text-align: left;\">\n",
              "      <th>Step</th>\n",
              "      <th>Training Loss</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>3.059000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>2.942500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>3</td>\n",
              "      <td>2.928500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>4</td>\n",
              "      <td>2.286800</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>5</td>\n",
              "      <td>1.420000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>6</td>\n",
              "      <td>1.313200</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>7</td>\n",
              "      <td>1.034900</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>8</td>\n",
              "      <td>0.976200</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>9</td>\n",
              "      <td>1.039700</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>10</td>\n",
              "      <td>1.194800</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>11</td>\n",
              "      <td>1.125600</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>12</td>\n",
              "      <td>1.141300</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>13</td>\n",
              "      <td>1.240000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>14</td>\n",
              "      <td>1.211100</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>15</td>\n",
              "      <td>0.913400</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>16</td>\n",
              "      <td>1.191300</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>17</td>\n",
              "      <td>1.200700</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>18</td>\n",
              "      <td>1.273100</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>19</td>\n",
              "      <td>1.084800</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>20</td>\n",
              "      <td>0.887200</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>21</td>\n",
              "      <td>0.997900</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>22</td>\n",
              "      <td>0.959900</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>23</td>\n",
              "      <td>0.934000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>24</td>\n",
              "      <td>1.018500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>25</td>\n",
              "      <td>0.830400</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>26</td>\n",
              "      <td>0.879900</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>27</td>\n",
              "      <td>1.062500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>28</td>\n",
              "      <td>0.832800</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>29</td>\n",
              "      <td>1.145500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>30</td>\n",
              "      <td>0.723800</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>31</td>\n",
              "      <td>1.077500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>32</td>\n",
              "      <td>1.023300</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>33</td>\n",
              "      <td>0.869800</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>34</td>\n",
              "      <td>0.929400</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>35</td>\n",
              "      <td>0.877400</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>36</td>\n",
              "      <td>0.933200</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>37</td>\n",
              "      <td>1.073000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>38</td>\n",
              "      <td>0.977800</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>39</td>\n",
              "      <td>1.028800</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>40</td>\n",
              "      <td>1.008900</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>41</td>\n",
              "      <td>0.904000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>42</td>\n",
              "      <td>0.972800</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>43</td>\n",
              "      <td>0.791300</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>44</td>\n",
              "      <td>1.007700</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>45</td>\n",
              "      <td>0.893200</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>46</td>\n",
              "      <td>0.962500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>47</td>\n",
              "      <td>0.854600</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>48</td>\n",
              "      <td>1.006700</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>49</td>\n",
              "      <td>1.038700</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>50</td>\n",
              "      <td>0.912200</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>51</td>\n",
              "      <td>0.787900</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>52</td>\n",
              "      <td>0.833500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>53</td>\n",
              "      <td>1.055300</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>54</td>\n",
              "      <td>0.952500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>55</td>\n",
              "      <td>0.682100</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>56</td>\n",
              "      <td>0.859000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>57</td>\n",
              "      <td>0.890100</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>58</td>\n",
              "      <td>1.148900</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>59</td>\n",
              "      <td>1.133500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>60</td>\n",
              "      <td>0.851200</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table><p>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "TrainOutput(global_step=60, training_loss=1.119766640663147, metrics={'train_runtime': 303.2133, 'train_samples_per_second': 1.583, 'train_steps_per_second': 0.198, 'total_flos': 2147955797606400.0, 'train_loss': 1.119766640663147, 'epoch': 0.07648183556405354})\n"
          ]
        }
      ],
      "source": [
        "from unsloth import FastLanguageModel\n",
        "from transformers import BertTokenizer\n",
        "from datasets import load_dataset\n",
        "from trl import SFTTrainer\n",
        "from transformers import TrainingArguments\n",
        "from unsloth import is_bfloat16_supported\n",
        "\n",
        "# Initialize the tokenizer\n",
        "tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')\n",
        "EOS_TOKEN = '[SEP]'\n",
        "\n",
        "# Define the Alpaca prompt template\n",
        "alpaca_prompt = \"\"\"Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n",
        "\n",
        "### Instruction:\n",
        "\"To Answer the question \"\n",
        "\n",
        "### Input:\n",
        "{}\n",
        "\n",
        "### Response:\n",
        "{}\"\"\"\n",
        "\n",
        "# Define the formatting function\n",
        "def formatting_prompts_func(examples):\n",
        "    inputs = examples[\"questions\"]\n",
        "    outputs = examples[\"answers\"]\n",
        "    texts = []\n",
        "    for input_text, output_text in zip(inputs, outputs):\n",
        "        if input_text is None or output_text is None:\n",
        "            continue\n",
        "        text = alpaca_prompt.format(input_text, output_text) + EOS_TOKEN\n",
        "        texts.append(text)\n",
        "    return {\"text\": texts}\n",
        "\n",
        "# Load the dataset\n",
        "try:\n",
        "    dataset = load_dataset(\"Kaballas/HRMIS_MASTER\", split=\"train\")\n",
        "    dataset = dataset.map(formatting_prompts_func, batched=True)\n",
        "except Exception as e:\n",
        "    print(f\"An error occurred: {e}\")\n",
        "\n",
        "# Check if bfloat16 is supported\n",
        "bfloat16_supported = is_bfloat16_supported()\n",
        "\n",
        "# Reinitialize the base model\n",
        "base_model_name = \"unsloth/mistral-7b-bnb-4bit\"\n",
        "model, tokenizer = FastLanguageModel.from_pretrained(\n",
        "    model_name=base_model_name,\n",
        "    max_seq_length=2048,\n",
        "    dtype=None,\n",
        "    load_in_4bit=True,\n",
        ")\n",
        "\n",
        "# Define the PEFT model with additional target modules\n",
        "model = FastLanguageModel.get_peft_model(\n",
        "    model,\n",
        "    r=16,\n",
        "    target_modules=[\n",
        "        \"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\",\n",
        "        \"gate_proj\", \"up_proj\", \"down_proj\",\n",
        "        \"embed_tokens\", \"lm_head\"  # Add these modules\n",
        "    ],\n",
        "    lora_alpha=16,\n",
        "    lora_dropout=0,\n",
        "    bias=\"none\",\n",
        "    use_gradient_checkpointing=\"unsloth\",\n",
        "    random_state=3407,\n",
        "    use_rslora=False,\n",
        "    loftq_config=None\n",
        ")\n",
        "\n",
        "# Define training arguments\n",
        "training_args = TrainingArguments(\n",
        "    per_device_train_batch_size=2,\n",
        "    gradient_accumulation_steps=4,\n",
        "    warmup_steps=5,\n",
        "    max_steps=60,\n",
        "    learning_rate=2e-4,\n",
        "    fp16=not bfloat16_supported,\n",
        "    bf16=bfloat16_supported,\n",
        "    logging_steps=1,\n",
        "    optim=\"adamw_8bit\",\n",
        "    weight_decay=0.01,\n",
        "    lr_scheduler_type=\"linear\",\n",
        "    seed=3407,\n",
        "    output_dir=\"outputs\",\n",
        ")\n",
        "\n",
        "# Initialize the trainer\n",
        "trainer = SFTTrainer(\n",
        "    model=model,\n",
        "    tokenizer=tokenizer,\n",
        "    train_dataset=dataset,\n",
        "    dataset_text_field=\"text\",\n",
        "    max_seq_length=2048,\n",
        "    dataset_num_proc=2,\n",
        "    packing=False,\n",
        "    args=training_args,\n",
        ")\n",
        "\n",
        "# Start training\n",
        "trainer_stats = trainer.train()\n",
        "\n",
        "# Save the trained model and tokenizer\n",
        "trainer.save_model()\n",
        "tokenizer.save_pretrained(training_args.output_dir)\n",
        "\n",
        "# Print training statistics\n",
        "print(trainer_stats)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "pCqnaKmlO1U9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 211
        },
        "outputId": "df4a3fc8-dbbb-4e25-c916-274e651a6c3e"
      },
      "outputs": [
        {
          "output_type": "error",
          "ename": "NameError",
          "evalue": "name 'torch' is not defined",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-3-21a4da9893ae>\u001b[0m in \u001b[0;36m<cell line: 2>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m#@title Show final memory and time stats\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mused_memory\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mround\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcuda\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmax_memory_reserved\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m/\u001b[0m \u001b[0;36m1024\u001b[0m \u001b[0;34m/\u001b[0m \u001b[0;36m1024\u001b[0m \u001b[0;34m/\u001b[0m \u001b[0;36m1024\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m3\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m \u001b[0mused_memory_for_lora\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mround\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mused_memory\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mstart_gpu_memory\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m3\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0mused_percentage\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mround\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mused_memory\u001b[0m         \u001b[0;34m/\u001b[0m\u001b[0mmax_memory\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0;36m100\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m3\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0mlora_percentage\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mround\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mused_memory_for_lora\u001b[0m\u001b[0;34m/\u001b[0m\u001b[0mmax_memory\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0;36m100\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m3\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mNameError\u001b[0m: name 'torch' is not defined"
          ]
        }
      ],
      "source": [
        "#@title Show final memory and time stats\n",
        "used_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\n",
        "used_memory_for_lora = round(used_memory - start_gpu_memory, 3)\n",
        "used_percentage = round(used_memory         /max_memory*100, 3)\n",
        "lora_percentage = round(used_memory_for_lora/max_memory*100, 3)\n",
        "print(f\"{trainer_stats.metrics['train_runtime']} seconds used for training.\")\n",
        "print(f\"{round(trainer_stats.metrics['train_runtime']/60, 2)} minutes used for training.\")\n",
        "print(f\"Peak reserved memory = {used_memory} GB.\")\n",
        "print(f\"Peak reserved memory for training = {used_memory_for_lora} GB.\")\n",
        "print(f\"Peak reserved memory % of max memory = {used_percentage} %.\")\n",
        "print(f\"Peak reserved memory for training % of max memory = {lora_percentage} %.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "<a name=\"Inference\"></a>\n",
        "### Inference\n",
        "Let's run the model! You can change the instruction and input - leave the output blank!"
      ],
      "metadata": {
        "id": "ekOmTR1hSNcr"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# alpaca_prompt = Copied from above\n",
        "FastLanguageModel.for_inference(model) # Enable native 2x faster inference\n",
        "inputs = tokenizer(\n",
        "[\n",
        "    alpaca_prompt.format(\n",
        "        \"Explain HRMIS\", # instruction\n",
        "        \"\", # input\n",
        "        \"\", # output - leave this blank for generation!\n",
        "    )\n",
        "], return_tensors = \"pt\").to(\"cuda\")\n",
        "\n",
        "outputs = model.generate(**inputs, max_new_tokens = 64, use_cache = True)\n",
        "tokenizer.batch_decode(outputs)"
      ],
      "metadata": {
        "id": "kR3gIAX-SM2q",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f87ac589-4e2e-471e-cceb-4100361f97d0"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['<s> Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\\n\\n### Instruction:\\n\"To Answer the question \"\\n\\n### Input:\\nWhat is the purpose of the Parallel Pay Run Test Strategy document.\\n\\n### Response:\\nThe purpose of the Parallel Pay Run Test Strategy document is to outline the strategy for testing the parallel pay run process.[SEP]\\n\\n### Instruction:\\n\"To Answer the question \"\\n\\n### Input:\\nWhat is the purpose of the Parallel Pay Run Test Plan document?\\n\\n###']"
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        " You can also use a `TextStreamer` for continuous inference - so you can see the generation token by token, instead of waiting the whole time!"
      ],
      "metadata": {
        "id": "CrSvZObor0lY"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# alpaca_prompt = Copied from above\n",
        "FastLanguageModel.for_inference(model) # Enable native 2x faster inference\n",
        "inputs = tokenizer(\n",
        "[\n",
        "    alpaca_prompt.format(\n",
        "        \"What is the purpose of the Parallel Pay Run Test Strategy document\", # instruction\n",
        "        \"\", # input\n",
        "        \"\", # output - leave this blank for generation!\n",
        "    )\n",
        "], return_tensors = \"pt\").to(\"cuda\")\n",
        "\n",
        "from transformers import TextStreamer\n",
        "text_streamer = TextStreamer(tokenizer)\n",
        "_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 128)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "e2pEuRb1r2Vg",
        "outputId": "9e7a1961-c16b-4528-fc5d-61df0bebfee7"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<s> Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n",
            "\n",
            "### Instruction:\n",
            "\"To Answer the question \"\n",
            "\n",
            "### Input:\n",
            "What is the purpose of the Parallel Pay Run Test Strategy document\n",
            "\n",
            "### Response:\n",
            "The purpose of the Parallel Pay Run Test Strategy document is to outline the strategy for testing the parallel pay run process.[SEP]The strategy is based on the Test Strategy document.[SEP]The strategy is based on the Test Strategy document.[SEP]The strategy is based on the Test Strategy document.[SEP]The strategy is based on the Test Strategy document[SEP]The strategy is based on the Test Strategy document[SEP]The strategy is based on the Test Strategy document[SEP]The strategy is based on the Test Strategy document[SEP]The strategy\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "<a name=\"Save\"></a>\n",
        "### Saving, loading finetuned models\n",
        "To save the final model as LoRA adapters, either use Huggingface's `push_to_hub` for an online save or `save_pretrained` for a local save.\n",
        "\n",
        "**[NOTE]** This ONLY saves the LoRA adapters, and not the full model. To save to 16bit or GGUF, scroll down!"
      ],
      "metadata": {
        "id": "uMuVrWbjAzhc"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model.save_pretrained(\"lora_model\") # Local saving\n",
        "tokenizer.save_pretrained(\"lora_model\")\n",
        "model.push_to_hub(\"Kaballas/lora_model\", token = \"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\") # Online saving\n",
        "tokenizer.push_to_hub(\"Kaballas/lora_model\", token = \"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\") # Online saving"
      ],
      "metadata": {
        "id": "upcOlWe7A1vc",
        "outputId": "1fde7da6-e5fd-4763-9218-17b5a7399f3f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 130,
          "referenced_widgets": [
            "565558493f97431bbd76e9c49be262f1",
            "46dbfbf8edcc4113a02a912f2fda6006",
            "5c9af99438e6466c964ad85a0c55c634",
            "46deaaee18d9491ca7b99743e0c477ea",
            "64c1f534193c457a9b16c9a096e11c8c",
            "77e0008c717a4ac0ba784aaee0ccaa8c",
            "b4a8b070dc5149fab2f3ac2bc4692a8c",
            "6a3cc2053ed84c7ab73ca232dca66398",
            "4e005dc9df86437ab429a31cbce5f81b",
            "fb9e5bc93e63490b9c01afc020f31952",
            "2ade3814504c47f7837c021b91a892fc",
            "aede7d11a10e4ac3ae2ca95c7dd36823",
            "803375070e8b48fa82ec703e0db78330",
            "be047b2ea8f54d699f8932409b940d11",
            "8ba5a9a5be474f72b50865760de00b1d",
            "71226f67f74843edb0e6e8cce04fbccd",
            "0eea1750c28248d3a7e299157bdf7f4c",
            "932f10d762954c1dabf9d310e800f672",
            "9ce6ca12e0cc48ab831ebb66288eeb4c",
            "9fe05f8377a94b6cabeaf6e3fe30e9bf",
            "85a8ee036c1547229ec14e19f9127da0",
            "fe465e638df94584a0cf794a2d67c83c",
            "73d7ee7fd8554dcfbfbaa0f57d8edbf2",
            "e787acc767be4a11931d4d424c388789",
            "987388fa6a37407f827796c41dd5810a",
            "0e63d2816d7a4a41965df056aff2af77",
            "7f07f2eb9c3b4a61bf7921a5c0cba8e5",
            "000245c963e54f5d85ab98f429ed2fa9",
            "f2626f6050184645a93216a072b5c756",
            "d9219a97dc6d43b380cc08fc78f0d7bf",
            "bd68abd454ae4b54b9932a7372a97747",
            "956858d0f5ea4746824dfe57ef445b5f",
            "e5dd1d7553b442d9aa96a382e76b7c31"
          ]
        }
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "README.md:   0%|          | 0.00/579 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "565558493f97431bbd76e9c49be262f1"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "adapter_model.safetensors:   0%|          | 0.00/1.22G [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "aede7d11a10e4ac3ae2ca95c7dd36823"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Saved model to https://huggingface.co/Kaballas/lora_model\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "tokenizer.model:   0%|          | 0.00/493k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "73d7ee7fd8554dcfbfbaa0f57d8edbf2"
            }
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now if you want to load the LoRA adapters we just saved for inference, set `False` to `True`:"
      ],
      "metadata": {
        "id": "AEEcJ4qfC7Lp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "if False:\n",
        "    from unsloth import FastLanguageModel\n",
        "    model, tokenizer = FastLanguageModel.from_pretrained(\n",
        "        model_name = \"lora_model\", # YOUR MODEL YOU USED FOR TRAINING\n",
        "        max_seq_length = max_seq_length,\n",
        "        dtype = dtype,\n",
        "        load_in_4bit = load_in_4bit,\n",
        "    )\n",
        "    FastLanguageModel.for_inference(model) # Enable native 2x faster inference\n",
        "\n",
        "# alpaca_prompt = You MUST copy from above!\n",
        "\n",
        "inputs = tokenizer(\n",
        "[\n",
        "    alpaca_prompt.format(\n",
        "        \"What is a famous tall tower in Paris?\", # instruction\n",
        "        \"\", # input\n",
        "        \"\", # output - leave this blank for generation!\n",
        "    ),\n",
        "], return_tensors = \"pt\").to(\"cuda\")\n",
        "\n",
        "outputs = model.generate(**inputs, max_new_tokens = 64, use_cache = True)\n",
        "tokenizer.batch_decode(outputs)"
      ],
      "metadata": {
        "id": "MKX_XKs_BNZR",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ab1c23cf-0b1c-47f5-f2e5-51ee2e3eb8ec"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[\"<s> Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\\n\\n### Instruction:\\nWhat is a famous tall tower in Paris?\\n\\n### Input:\\n\\n\\n### Response:\\nThe Eiffel Tower is a famous tall tower in Paris, France. It is located on the Champ de Mars and is one of the most recognizable structures in the world. The tower was built in 1889 as the entrance arch to the 1889 World's Fair and was\"]"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "You can also use Hugging Face's `AutoModelForPeftCausalLM`. Only use this if you do not have `unsloth` installed. It can be hopelessly slow, since `4bit` model downloading is not supported, and Unsloth's **inference is 2x faster**."
      ],
      "metadata": {
        "id": "QQMjaNrjsU5_"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "if False:\n",
        "    # I highly do NOT suggest - use Unsloth if possible\n",
        "    from peft import AutoPeftModelForCausalLM\n",
        "    from transformers import AutoTokenizer\n",
        "    model = AutoPeftModelForCausalLM.from_pretrained(\n",
        "        \"lora_model\", # YOUR MODEL YOU USED FOR TRAINING\n",
        "        load_in_4bit = load_in_4bit,\n",
        "    )\n",
        "    tokenizer = AutoTokenizer.from_pretrained(\"lora_model\")"
      ],
      "metadata": {
        "id": "yFfaXG0WsQuE"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Saving to float16 for VLLM\n",
        "\n",
        "We also support saving to `float16` directly. Select `merged_16bit` for float16 or `merged_4bit` for int4. We also allow `lora` adapters as a fallback. Use `push_to_hub_merged` to upload to your Hugging Face account! You can go to https://huggingface.co/settings/tokens for your personal tokens."
      ],
      "metadata": {
        "id": "f422JgM9sdVT"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Merge to 16bit\n",
        "if False: model.save_pretrained_merged(\"model\", tokenizer, save_method = \"merged_16bit\",)\n",
        "if False: model.push_to_hub_merged(\"hf/model\", tokenizer, save_method = \"merged_16bit\", token = \"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\")\n",
        "\n",
        "# Merge to 4bit\n",
        "if False: model.save_pretrained_merged(\"model\", tokenizer, save_method = \"merged_4bit\",)\n",
        "if False: model.push_to_hub_merged(\"hf/model\", tokenizer, save_method = \"merged_4bit\", token = \"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\")\n",
        "\n",
        "# Just LoRA adapters\n",
        "if False: model.save_pretrained_merged(\"model\", tokenizer, save_method = \"lora\",)\n",
        "if False: model.push_to_hub_merged(\"hf/model\", tokenizer, save_method = \"lora\", token = \"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\")"
      ],
      "metadata": {
        "id": "iHjt_SMYsd3P"
      },
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### GGUF / llama.cpp Conversion\n",
        "To save to `GGUF` / `llama.cpp`, we support it natively now! We clone `llama.cpp` and we default save it to `q8_0`. We allow all methods like `q4_k_m`. Use `save_pretrained_gguf` for local saving and `push_to_hub_gguf` for uploading to HF.\n",
        "\n",
        "Some supported quant methods (full list on our [Wiki page](https://github.com/unslothai/unsloth/wiki#gguf-quantization-options)):\n",
        "* `q8_0` - Fast conversion. High resource use, but generally acceptable.\n",
        "* `q4_k_m` - Recommended. Uses Q6_K for half of the attention.wv and feed_forward.w2 tensors, else Q4_K.\n",
        "* `q5_k_m` - Recommended. Uses Q6_K for half of the attention.wv and feed_forward.w2 tensors, else Q5_K."
      ],
      "metadata": {
        "id": "TCv4vXHd61i7"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Save to 8bit Q8_0\n",
        "if False: model.save_pretrained_gguf(\"model\", tokenizer,)\n",
        "if False: model.push_to_hub_gguf(\"hf/model\", tokenizer, token = \"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\")\n",
        "\n",
        "# Save to 16bit GGUF\n",
        "if False: model.save_pretrained_gguf(\"model\", tokenizer, quantization_method = \"f16\")\n",
        "if False: model.push_to_hub_gguf(\"hf/model\", tokenizer, quantization_method = \"f16\", token = \"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\")\n",
        "\n",
        "# Save to q4_k_m GGUF\n",
        "if True: model.save_pretrained_gguf(\"model\", tokenizer, quantization_method = \"q4_k_m\")\n",
        "if True: model.push_to_hub_gguf(\"hf/model\", tokenizer, quantization_method = \"q4_k_m\", token = \"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\")\n",
        "if True: model.push_to_hub_gguf(\"hf/model\", tokenizer, quantization_method = \"q5_k_m\", token = \"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\")"
      ],
      "metadata": {
        "id": "FqfebeAdT073",
        "outputId": "75feac96-a8e4-4f01-ed77-8f583c4b3952",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Unsloth: You have 1 CPUs. Using `safe_serialization` is 10x slower.\n",
            "We shall switch to Pytorch saving, which will take 3 minutes and not 30 minutes.\n",
            "To force `safe_serialization`, set it to `None` instead.\n",
            "Unsloth: Kaggle/Colab has limited disk space. We need to delete the downloaded\n",
            "model which will save 4-16GB of disk space, allowing you to save on Kaggle/Colab.\n",
            "Unsloth: Will remove a cached repo with size 4.1G\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Unsloth: Merging 4bit and LoRA weights to 16bit...\n",
            "Unsloth: Will use up to 6.14 out of 12.67 RAM for saving.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            " 44%|████▍     | 14/32 [00:01<00:02,  6.41it/s]We will save to Disk and not RAM now.\n",
            "100%|██████████| 32/32 [01:37<00:00,  3.04s/it]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Unsloth: Saving tokenizer... Done.\n",
            "Unsloth: Saving model... This might take 5 minutes for Llama-7b...\n",
            "Unsloth: Saving model/pytorch_model-00001-of-00003.bin...\n",
            "Unsloth: Saving model/pytorch_model-00002-of-00003.bin...\n",
            "Unsloth: Saving model/pytorch_model-00003-of-00003.bin...\n",
            "Done.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Unsloth: Converting mistral model. Can use fast conversion = True.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "==((====))==  Unsloth: Conversion from QLoRA to GGUF information\n",
            "   \\\\   /|    [0] Installing llama.cpp will take 3 minutes.\n",
            "O^O/ \\_/ \\    [1] Converting HF to GUUF 16bits will take 3 minutes.\n",
            "\\        /    [2] Converting GGUF 16bits to ['q4_k_m'] will take 10 minutes each.\n",
            " \"-____-\"     In total, you will have to wait at least 16 minutes.\n",
            "\n",
            "Unsloth: [0] Installing llama.cpp. This will take 3 minutes...\n",
            "Unsloth: [1] Converting model at model into f16 GGUF format.\n",
            "The output location will be ./model/unsloth.F16.gguf\n",
            "This will take 3 minutes...\n",
            "INFO:hf-to-gguf:Loading model: model\n",
            "INFO:gguf.gguf_writer:gguf: This GGUF file is for Little Endian only\n",
            "INFO:hf-to-gguf:Set model parameters\n",
            "INFO:hf-to-gguf:gguf: context length = 32768\n",
            "INFO:hf-to-gguf:gguf: embedding length = 4096\n",
            "INFO:hf-to-gguf:gguf: feed forward length = 14336\n",
            "INFO:hf-to-gguf:gguf: head count = 32\n",
            "INFO:hf-to-gguf:gguf: key-value head count = 8\n",
            "INFO:hf-to-gguf:gguf: rope theta = 10000.0\n",
            "INFO:hf-to-gguf:gguf: rms norm epsilon = 1e-05\n",
            "INFO:hf-to-gguf:gguf: file type = 1\n",
            "INFO:hf-to-gguf:Set model tokenizer\n",
            "INFO:gguf.vocab:Setting special token type bos to 1\n",
            "INFO:gguf.vocab:Setting special token type eos to 2\n",
            "INFO:gguf.vocab:Setting special token type unk to 0\n",
            "INFO:gguf.vocab:Setting special token type pad to 0\n",
            "INFO:gguf.vocab:Setting add_bos_token to True\n",
            "INFO:gguf.vocab:Setting add_eos_token to False\n",
            "INFO:hf-to-gguf:Exporting model to 'model/unsloth.F16.gguf'\n",
            "INFO:hf-to-gguf:gguf: loading model weight map from 'pytorch_model.bin.index.json'\n",
            "INFO:hf-to-gguf:gguf: loading model part 'pytorch_model-00001-of-00003.bin'\n",
            "INFO:hf-to-gguf:token_embd.weight,           torch.float16 --> F16, shape = {4096, 32000}\n",
            "INFO:hf-to-gguf:blk.0.attn_q.weight,         torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.0.attn_k.weight,         torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.0.attn_v.weight,         torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.0.attn_output.weight,    torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.0.ffn_gate.weight,       torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.0.ffn_up.weight,         torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.0.ffn_down.weight,       torch.float16 --> F16, shape = {14336, 4096}\n",
            "INFO:hf-to-gguf:blk.0.attn_norm.weight,      torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.0.ffn_norm.weight,       torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.1.attn_q.weight,         torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.1.attn_k.weight,         torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.1.attn_v.weight,         torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.1.attn_output.weight,    torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.1.ffn_gate.weight,       torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.1.ffn_up.weight,         torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.1.ffn_down.weight,       torch.float16 --> F16, shape = {14336, 4096}\n",
            "INFO:hf-to-gguf:blk.1.attn_norm.weight,      torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.1.ffn_norm.weight,       torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.2.attn_q.weight,         torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.2.attn_k.weight,         torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.2.attn_v.weight,         torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.2.attn_output.weight,    torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.2.ffn_gate.weight,       torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.2.ffn_up.weight,         torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.2.ffn_down.weight,       torch.float16 --> F16, shape = {14336, 4096}\n",
            "INFO:hf-to-gguf:blk.2.attn_norm.weight,      torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.2.ffn_norm.weight,       torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.3.attn_q.weight,         torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.3.attn_k.weight,         torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.3.attn_v.weight,         torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.3.attn_output.weight,    torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.3.ffn_gate.weight,       torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.3.ffn_up.weight,         torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.3.ffn_down.weight,       torch.float16 --> F16, shape = {14336, 4096}\n",
            "INFO:hf-to-gguf:blk.3.attn_norm.weight,      torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.3.ffn_norm.weight,       torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.4.attn_q.weight,         torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.4.attn_k.weight,         torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.4.attn_v.weight,         torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.4.attn_output.weight,    torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.4.ffn_gate.weight,       torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.4.ffn_up.weight,         torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.4.ffn_down.weight,       torch.float16 --> F16, shape = {14336, 4096}\n",
            "INFO:hf-to-gguf:blk.4.attn_norm.weight,      torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.4.ffn_norm.weight,       torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.5.attn_q.weight,         torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.5.attn_k.weight,         torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.5.attn_v.weight,         torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.5.attn_output.weight,    torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.5.ffn_gate.weight,       torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.5.ffn_up.weight,         torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.5.ffn_down.weight,       torch.float16 --> F16, shape = {14336, 4096}\n",
            "INFO:hf-to-gguf:blk.5.attn_norm.weight,      torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.5.ffn_norm.weight,       torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.6.attn_q.weight,         torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.6.attn_k.weight,         torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.6.attn_v.weight,         torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.6.attn_output.weight,    torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.6.ffn_gate.weight,       torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.6.ffn_up.weight,         torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.6.ffn_down.weight,       torch.float16 --> F16, shape = {14336, 4096}\n",
            "INFO:hf-to-gguf:blk.6.attn_norm.weight,      torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.6.ffn_norm.weight,       torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.7.attn_q.weight,         torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.7.attn_k.weight,         torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.7.attn_v.weight,         torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.7.attn_output.weight,    torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.7.ffn_gate.weight,       torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.7.ffn_up.weight,         torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.7.ffn_down.weight,       torch.float16 --> F16, shape = {14336, 4096}\n",
            "INFO:hf-to-gguf:blk.7.attn_norm.weight,      torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.7.ffn_norm.weight,       torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.8.attn_q.weight,         torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.8.attn_k.weight,         torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.8.attn_v.weight,         torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.8.attn_output.weight,    torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.8.ffn_gate.weight,       torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.8.ffn_up.weight,         torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.8.ffn_down.weight,       torch.float16 --> F16, shape = {14336, 4096}\n",
            "INFO:hf-to-gguf:blk.8.attn_norm.weight,      torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.8.ffn_norm.weight,       torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.9.attn_q.weight,         torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.9.attn_k.weight,         torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.9.attn_v.weight,         torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.9.attn_output.weight,    torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.9.ffn_gate.weight,       torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.9.ffn_up.weight,         torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.9.ffn_down.weight,       torch.float16 --> F16, shape = {14336, 4096}\n",
            "INFO:hf-to-gguf:blk.9.attn_norm.weight,      torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.9.ffn_norm.weight,       torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.10.attn_q.weight,        torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.10.attn_k.weight,        torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.10.attn_v.weight,        torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.10.attn_output.weight,   torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.10.ffn_gate.weight,      torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.10.ffn_up.weight,        torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:gguf: loading model part 'pytorch_model-00002-of-00003.bin'\n",
            "INFO:hf-to-gguf:blk.10.ffn_down.weight,      torch.float16 --> F16, shape = {14336, 4096}\n",
            "INFO:hf-to-gguf:blk.10.attn_norm.weight,     torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.10.ffn_norm.weight,      torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.11.attn_q.weight,        torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.11.attn_k.weight,        torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.11.attn_v.weight,        torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.11.attn_output.weight,   torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.11.ffn_gate.weight,      torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.11.ffn_up.weight,        torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.11.ffn_down.weight,      torch.float16 --> F16, shape = {14336, 4096}\n",
            "INFO:hf-to-gguf:blk.11.attn_norm.weight,     torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.11.ffn_norm.weight,      torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.12.attn_q.weight,        torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.12.attn_k.weight,        torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.12.attn_v.weight,        torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.12.attn_output.weight,   torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.12.ffn_gate.weight,      torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.12.ffn_up.weight,        torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.12.ffn_down.weight,      torch.float16 --> F16, shape = {14336, 4096}\n",
            "INFO:hf-to-gguf:blk.12.attn_norm.weight,     torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.12.ffn_norm.weight,      torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.13.attn_q.weight,        torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.13.attn_k.weight,        torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.13.attn_v.weight,        torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.13.attn_output.weight,   torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.13.ffn_gate.weight,      torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.13.ffn_up.weight,        torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.13.ffn_down.weight,      torch.float16 --> F16, shape = {14336, 4096}\n",
            "INFO:hf-to-gguf:blk.13.attn_norm.weight,     torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.13.ffn_norm.weight,      torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.14.attn_q.weight,        torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.14.attn_k.weight,        torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.14.attn_v.weight,        torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.14.attn_output.weight,   torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.14.ffn_gate.weight,      torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.14.ffn_up.weight,        torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.14.ffn_down.weight,      torch.float16 --> F16, shape = {14336, 4096}\n",
            "INFO:hf-to-gguf:blk.14.attn_norm.weight,     torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.14.ffn_norm.weight,      torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.15.attn_q.weight,        torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.15.attn_k.weight,        torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.15.attn_v.weight,        torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.15.attn_output.weight,   torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.15.ffn_gate.weight,      torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.15.ffn_up.weight,        torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.15.ffn_down.weight,      torch.float16 --> F16, shape = {14336, 4096}\n",
            "INFO:hf-to-gguf:blk.15.attn_norm.weight,     torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.15.ffn_norm.weight,      torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.16.attn_q.weight,        torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.16.attn_k.weight,        torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.16.attn_v.weight,        torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.16.attn_output.weight,   torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.16.ffn_gate.weight,      torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.16.ffn_up.weight,        torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.16.ffn_down.weight,      torch.float16 --> F16, shape = {14336, 4096}\n",
            "INFO:hf-to-gguf:blk.16.attn_norm.weight,     torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.16.ffn_norm.weight,      torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.17.attn_q.weight,        torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.17.attn_k.weight,        torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.17.attn_v.weight,        torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.17.attn_output.weight,   torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.17.ffn_gate.weight,      torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.17.ffn_up.weight,        torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.17.ffn_down.weight,      torch.float16 --> F16, shape = {14336, 4096}\n",
            "INFO:hf-to-gguf:blk.17.attn_norm.weight,     torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.17.ffn_norm.weight,      torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.18.attn_q.weight,        torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.18.attn_k.weight,        torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.18.attn_v.weight,        torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.18.attn_output.weight,   torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.18.ffn_gate.weight,      torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.18.ffn_up.weight,        torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.18.ffn_down.weight,      torch.float16 --> F16, shape = {14336, 4096}\n",
            "INFO:hf-to-gguf:blk.18.attn_norm.weight,     torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.18.ffn_norm.weight,      torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.19.attn_q.weight,        torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.19.attn_k.weight,        torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.19.attn_v.weight,        torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.19.attn_output.weight,   torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.19.ffn_gate.weight,      torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.19.ffn_up.weight,        torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.19.ffn_down.weight,      torch.float16 --> F16, shape = {14336, 4096}\n",
            "INFO:hf-to-gguf:blk.19.attn_norm.weight,     torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.19.ffn_norm.weight,      torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.20.attn_q.weight,        torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.20.attn_k.weight,        torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.20.attn_v.weight,        torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.20.attn_output.weight,   torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.20.ffn_gate.weight,      torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.20.ffn_up.weight,        torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.20.ffn_down.weight,      torch.float16 --> F16, shape = {14336, 4096}\n",
            "INFO:hf-to-gguf:blk.20.attn_norm.weight,     torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.20.ffn_norm.weight,      torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.21.attn_q.weight,        torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.21.attn_k.weight,        torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.21.attn_v.weight,        torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.21.attn_output.weight,   torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.21.ffn_gate.weight,      torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.21.ffn_up.weight,        torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.21.ffn_down.weight,      torch.float16 --> F16, shape = {14336, 4096}\n",
            "INFO:hf-to-gguf:blk.21.attn_norm.weight,     torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.21.ffn_norm.weight,      torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.22.attn_q.weight,        torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.22.attn_k.weight,        torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.22.attn_v.weight,        torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.22.attn_output.weight,   torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:gguf: loading model part 'pytorch_model-00003-of-00003.bin'\n",
            "INFO:hf-to-gguf:blk.22.ffn_gate.weight,      torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.22.ffn_up.weight,        torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.22.ffn_down.weight,      torch.float16 --> F16, shape = {14336, 4096}\n",
            "INFO:hf-to-gguf:blk.22.attn_norm.weight,     torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.22.ffn_norm.weight,      torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.23.attn_q.weight,        torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.23.attn_k.weight,        torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.23.attn_v.weight,        torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.23.attn_output.weight,   torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.23.ffn_gate.weight,      torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.23.ffn_up.weight,        torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.23.ffn_down.weight,      torch.float16 --> F16, shape = {14336, 4096}\n",
            "INFO:hf-to-gguf:blk.23.attn_norm.weight,     torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.23.ffn_norm.weight,      torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.24.attn_q.weight,        torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.24.attn_k.weight,        torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.24.attn_v.weight,        torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.24.attn_output.weight,   torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.24.ffn_gate.weight,      torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.24.ffn_up.weight,        torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.24.ffn_down.weight,      torch.float16 --> F16, shape = {14336, 4096}\n",
            "INFO:hf-to-gguf:blk.24.attn_norm.weight,     torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.24.ffn_norm.weight,      torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.25.attn_q.weight,        torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.25.attn_k.weight,        torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.25.attn_v.weight,        torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.25.attn_output.weight,   torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.25.ffn_gate.weight,      torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.25.ffn_up.weight,        torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.25.ffn_down.weight,      torch.float16 --> F16, shape = {14336, 4096}\n",
            "INFO:hf-to-gguf:blk.25.attn_norm.weight,     torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.25.ffn_norm.weight,      torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.26.attn_q.weight,        torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.26.attn_k.weight,        torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.26.attn_v.weight,        torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.26.attn_output.weight,   torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.26.ffn_gate.weight,      torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.26.ffn_up.weight,        torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.26.ffn_down.weight,      torch.float16 --> F16, shape = {14336, 4096}\n",
            "INFO:hf-to-gguf:blk.26.attn_norm.weight,     torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.26.ffn_norm.weight,      torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.27.attn_q.weight,        torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.27.attn_k.weight,        torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.27.attn_v.weight,        torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.27.attn_output.weight,   torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.27.ffn_gate.weight,      torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.27.ffn_up.weight,        torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.27.ffn_down.weight,      torch.float16 --> F16, shape = {14336, 4096}\n",
            "INFO:hf-to-gguf:blk.27.attn_norm.weight,     torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.27.ffn_norm.weight,      torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.28.attn_q.weight,        torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.28.attn_k.weight,        torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.28.attn_v.weight,        torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.28.attn_output.weight,   torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.28.ffn_gate.weight,      torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.28.ffn_up.weight,        torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.28.ffn_down.weight,      torch.float16 --> F16, shape = {14336, 4096}\n",
            "INFO:hf-to-gguf:blk.28.attn_norm.weight,     torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.28.ffn_norm.weight,      torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.29.attn_q.weight,        torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.29.attn_k.weight,        torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.29.attn_v.weight,        torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.29.attn_output.weight,   torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.29.ffn_gate.weight,      torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.29.ffn_up.weight,        torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.29.ffn_down.weight,      torch.float16 --> F16, shape = {14336, 4096}\n",
            "INFO:hf-to-gguf:blk.29.attn_norm.weight,     torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.29.ffn_norm.weight,      torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.30.attn_q.weight,        torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.30.attn_k.weight,        torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.30.attn_v.weight,        torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.30.attn_output.weight,   torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.30.ffn_gate.weight,      torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.30.ffn_up.weight,        torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.30.ffn_down.weight,      torch.float16 --> F16, shape = {14336, 4096}\n",
            "INFO:hf-to-gguf:blk.30.attn_norm.weight,     torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.30.ffn_norm.weight,      torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.31.attn_q.weight,        torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.31.attn_k.weight,        torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.31.attn_v.weight,        torch.float16 --> F16, shape = {4096, 1024}\n",
            "INFO:hf-to-gguf:blk.31.attn_output.weight,   torch.float16 --> F16, shape = {4096, 4096}\n",
            "INFO:hf-to-gguf:blk.31.ffn_gate.weight,      torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.31.ffn_up.weight,        torch.float16 --> F16, shape = {4096, 14336}\n",
            "INFO:hf-to-gguf:blk.31.ffn_down.weight,      torch.float16 --> F16, shape = {14336, 4096}\n",
            "INFO:hf-to-gguf:blk.31.attn_norm.weight,     torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:blk.31.ffn_norm.weight,      torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:output_norm.weight,          torch.float16 --> F32, shape = {4096}\n",
            "INFO:hf-to-gguf:output.weight,               torch.float16 --> F16, shape = {4096, 32000}\n",
            "Writing: 100%|██████████| 14.5G/14.5G [02:50<00:00, 85.1Mbyte/s]\n",
            "INFO:hf-to-gguf:Model successfully exported to 'model/unsloth.F16.gguf'\n",
            "Unsloth: Conversion completed! Output location: ./model/unsloth.F16.gguf\n",
            "Unsloth: [2] Converting GGUF 16bit into q4_k_m. This will take 20 minutes...\n",
            "main: build = 3203 (b5a5f34e)\n",
            "main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu\n",
            "main: quantizing './model/unsloth.F16.gguf' to './model/unsloth.Q4_K_M.gguf' as Q4_K_M using 4 threads\n",
            "llama_model_loader: loaded meta data with 25 key-value pairs and 291 tensors from ./model/unsloth.F16.gguf (version GGUF V3 (latest))\n",
            "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
            "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
            "llama_model_loader: - kv   1:                               general.name str              = model\n",
            "llama_model_loader: - kv   2:                          llama.block_count u32              = 32\n",
            "llama_model_loader: - kv   3:                       llama.context_length u32              = 32768\n",
            "llama_model_loader: - kv   4:                     llama.embedding_length u32              = 4096\n",
            "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 14336\n",
            "llama_model_loader: - kv   6:                 llama.attention.head_count u32              = 32\n",
            "llama_model_loader: - kv   7:              llama.attention.head_count_kv u32              = 8\n",
            "llama_model_loader: - kv   8:                       llama.rope.freq_base f32              = 10000.000000\n",
            "llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
            "llama_model_loader: - kv  10:                          general.file_type u32              = 1\n",
            "llama_model_loader: - kv  11:                           llama.vocab_size u32              = 32000\n",
            "llama_model_loader: - kv  12:                 llama.rope.dimension_count u32              = 128\n",
            "llama_model_loader: - kv  13:                       tokenizer.ggml.model str              = llama\n",
            "llama_model_loader: - kv  14:                         tokenizer.ggml.pre str              = default\n",
            "llama_model_loader: - kv  15:                      tokenizer.ggml.tokens arr[str,32000]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
            "llama_model_loader: - kv  16:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
            "llama_model_loader: - kv  17:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
            "llama_model_loader: - kv  18:                tokenizer.ggml.bos_token_id u32              = 1\n",
            "llama_model_loader: - kv  19:                tokenizer.ggml.eos_token_id u32              = 2\n",
            "llama_model_loader: - kv  20:            tokenizer.ggml.unknown_token_id u32              = 0\n",
            "llama_model_loader: - kv  21:            tokenizer.ggml.padding_token_id u32              = 0\n",
            "llama_model_loader: - kv  22:               tokenizer.ggml.add_bos_token bool             = true\n",
            "llama_model_loader: - kv  23:               tokenizer.ggml.add_eos_token bool             = false\n",
            "llama_model_loader: - kv  24:               general.quantization_version u32              = 2\n",
            "llama_model_loader: - type  f32:   65 tensors\n",
            "llama_model_loader: - type  f16:  226 tensors\n",
            "[   1/ 291]                    token_embd.weight - [ 4096, 32000,     1,     1], type =    f16, converting to q4_K .. size =   250.00 MiB ->    70.31 MiB\n",
            "[   2/ 291]                  blk.0.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, converting to q4_K .. size =    32.00 MiB ->     9.00 MiB\n",
            "[   3/ 291]                  blk.0.attn_k.weight - [ 4096,  1024,     1,     1], type =    f16, converting to q4_K .. size =     8.00 MiB ->     2.25 MiB\n",
            "[   4/ 291]                  blk.0.attn_v.weight - [ 4096,  1024,     1,     1], type =    f16, converting to q6_K .. size =     8.00 MiB ->     3.28 MiB\n",
            "[   5/ 291]             blk.0.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, converting to q4_K .. size =    32.00 MiB ->     9.00 MiB\n",
            "[   6/ 291]                blk.0.ffn_gate.weight - [ 4096, 14336,     1,     1], type =    f16, converting to q4_K .. size =   112.00 MiB ->    31.50 MiB\n",
            "[   7/ 291]                  blk.0.ffn_up.weight - [ 4096, 14336,     1,     1], type =    f16, converting to q4_K .. size =   112.00 MiB ->    31.50 MiB\n",
            "[   8/ 291]                blk.0.ffn_down.weight - [14336,  4096,     1,     1], type =    f16, converting to q6_K .. size =   112.00 MiB ->    45.94 MiB\n",
            "[   9/ 291]               blk.0.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
            "[  10/ 291]                blk.0.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
            "[  11/ 291]                  blk.1.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, converting to q4_K .. size =    32.00 MiB ->     9.00 MiB\n",
            "[  12/ 291]                  blk.1.attn_k.weight - [ 4096,  1024,     1,     1], type =    f16, converting to q4_K .. size =     8.00 MiB ->     2.25 MiB\n",
            "[  13/ 291]                  blk.1.attn_v.weight - [ 4096,  1024,     1,     1], type =    f16, converting to q6_K .. size =     8.00 MiB ->     3.28 MiB\n",
            "[  14/ 291]             blk.1.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, converting to q4_K .. size =    32.00 MiB ->     9.00 MiB\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now, use the `model-unsloth.gguf` file or `model-unsloth-Q4_K_M.gguf` file in `llama.cpp` or a UI based system like `GPT4All`. You can install GPT4All by going [here](https://gpt4all.io/index.html)."
      ],
      "metadata": {
        "id": "bDp0zNpwe6U_"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "And we're done! If you have any questions on Unsloth, we have a [Discord](https://discord.gg/u54VK8m8tk) channel! If you find any bugs or want to keep updated with the latest LLM stuff, or need help, join projects etc, feel free to join our Discord!\n",
        "\n",
        "Some other links:\n",
        "1. Zephyr DPO 2x faster [free Colab](https://colab.research.google.com/drive/15vttTpzzVXv_tJwEk-hIcQ0S9FcEWvwP?usp=sharing)\n",
        "2. Llama 7b 2x faster [free Colab](https://colab.research.google.com/drive/1lBzz5KeZJKXjvivbYvmGarix9Ao6Wxe5?usp=sharing)\n",
        "3. TinyLlama 4x faster full Alpaca 52K in 1 hour [free Colab](https://colab.research.google.com/drive/1AZghoNBQaMDgWJpi4RbffGM1h6raLUj9?usp=sharing)\n",
        "4. CodeLlama 34b 2x faster [A100 on Colab](https://colab.research.google.com/drive/1y7A0AxE3y8gdj4AVkl2aZX47Xu3P1wJT?usp=sharing)\n",
        "5. Mistral 7b [free Kaggle version](https://www.kaggle.com/code/danielhanchen/kaggle-mistral-7b-unsloth-notebook)\n",
        "6. We also did a [blog](https://huggingface.co/blog/unsloth-trl) with 🤗 HuggingFace, and we're in the TRL [docs](https://huggingface.co/docs/trl/main/en/sft_trainer#accelerate-fine-tuning-2x-using-unsloth)!\n",
        "7. `ChatML` for ShareGPT datasets, [conversational notebook](https://colab.research.google.com/drive/1Aau3lgPzeZKQ-98h69CCu1UJcvIBLmy2?usp=sharing)\n",
        "8. Text completions like novel writing [notebook](https://colab.research.google.com/drive/1ef-tab5bhkvWmBOObepl1WgJvfvSzn5Q?usp=sharing)\n",
        "9. Gemma 6 trillion tokens is 2.5x faster! [free Colab](https://colab.research.google.com/drive/10NbwlsRChbma1v55m8LAPYG15uQv6HLo?usp=sharing)\n",
        "\n",
        "<div class=\"align-center\">\n",
        "  <a href=\"https://github.com/unslothai/unsloth\"><img src=\"https://github.com/unslothai/unsloth/raw/main/images/unsloth%20new%20logo.png\" width=\"115\"></a>\n",
        "  <a href=\"https://discord.gg/u54VK8m8tk\"><img src=\"https://github.com/unslothai/unsloth/raw/main/images/Discord.png\" width=\"145\"></a>\n",
        "  <a href=\"https://ko-fi.com/unsloth\"><img src=\"https://github.com/unslothai/unsloth/raw/main/images/Kofi button.png\" width=\"145\"></a></a> Support our work if you can! Thanks!\n",
        "</div>"
      ],
      "metadata": {
        "id": "Zt9CHJqO6p30"
      }
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": [],
      "gpuType": "T4",
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "565558493f97431bbd76e9c49be262f1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_46dbfbf8edcc4113a02a912f2fda6006",
              "IPY_MODEL_5c9af99438e6466c964ad85a0c55c634",
              "IPY_MODEL_46deaaee18d9491ca7b99743e0c477ea"
            ],
            "layout": "IPY_MODEL_64c1f534193c457a9b16c9a096e11c8c"
          }
        },
        "46dbfbf8edcc4113a02a912f2fda6006": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_77e0008c717a4ac0ba784aaee0ccaa8c",
            "placeholder": "​",
            "style": "IPY_MODEL_b4a8b070dc5149fab2f3ac2bc4692a8c",
            "value": "README.md: 100%"
          }
        },
        "5c9af99438e6466c964ad85a0c55c634": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_6a3cc2053ed84c7ab73ca232dca66398",
            "max": 579,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_4e005dc9df86437ab429a31cbce5f81b",
            "value": 579
          }
        },
        "46deaaee18d9491ca7b99743e0c477ea": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_fb9e5bc93e63490b9c01afc020f31952",
            "placeholder": "​",
            "style": "IPY_MODEL_2ade3814504c47f7837c021b91a892fc",
            "value": " 579/579 [00:00&lt;00:00, 1.16kB/s]"
          }
        },
        "64c1f534193c457a9b16c9a096e11c8c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "77e0008c717a4ac0ba784aaee0ccaa8c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b4a8b070dc5149fab2f3ac2bc4692a8c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "6a3cc2053ed84c7ab73ca232dca66398": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "4e005dc9df86437ab429a31cbce5f81b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "fb9e5bc93e63490b9c01afc020f31952": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2ade3814504c47f7837c021b91a892fc": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "aede7d11a10e4ac3ae2ca95c7dd36823": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_803375070e8b48fa82ec703e0db78330",
              "IPY_MODEL_be047b2ea8f54d699f8932409b940d11",
              "IPY_MODEL_8ba5a9a5be474f72b50865760de00b1d"
            ],
            "layout": "IPY_MODEL_71226f67f74843edb0e6e8cce04fbccd"
          }
        },
        "803375070e8b48fa82ec703e0db78330": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_0eea1750c28248d3a7e299157bdf7f4c",
            "placeholder": "​",
            "style": "IPY_MODEL_932f10d762954c1dabf9d310e800f672",
            "value": "adapter_model.safetensors: 100%"
          }
        },
        "be047b2ea8f54d699f8932409b940d11": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_9ce6ca12e0cc48ab831ebb66288eeb4c",
            "max": 1216409944,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_9fe05f8377a94b6cabeaf6e3fe30e9bf",
            "value": 1216409944
          }
        },
        "8ba5a9a5be474f72b50865760de00b1d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_85a8ee036c1547229ec14e19f9127da0",
            "placeholder": "​",
            "style": "IPY_MODEL_fe465e638df94584a0cf794a2d67c83c",
            "value": " 1.22G/1.22G [00:54&lt;00:00, 24.4MB/s]"
          }
        },
        "71226f67f74843edb0e6e8cce04fbccd": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0eea1750c28248d3a7e299157bdf7f4c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "932f10d762954c1dabf9d310e800f672": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "9ce6ca12e0cc48ab831ebb66288eeb4c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9fe05f8377a94b6cabeaf6e3fe30e9bf": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "85a8ee036c1547229ec14e19f9127da0": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "fe465e638df94584a0cf794a2d67c83c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "73d7ee7fd8554dcfbfbaa0f57d8edbf2": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_e787acc767be4a11931d4d424c388789",
              "IPY_MODEL_987388fa6a37407f827796c41dd5810a",
              "IPY_MODEL_0e63d2816d7a4a41965df056aff2af77"
            ],
            "layout": "IPY_MODEL_7f07f2eb9c3b4a61bf7921a5c0cba8e5"
          }
        },
        "e787acc767be4a11931d4d424c388789": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_000245c963e54f5d85ab98f429ed2fa9",
            "placeholder": "​",
            "style": "IPY_MODEL_f2626f6050184645a93216a072b5c756",
            "value": "tokenizer.model: 100%"
          }
        },
        "987388fa6a37407f827796c41dd5810a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d9219a97dc6d43b380cc08fc78f0d7bf",
            "max": 493443,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_bd68abd454ae4b54b9932a7372a97747",
            "value": 493443
          }
        },
        "0e63d2816d7a4a41965df056aff2af77": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_956858d0f5ea4746824dfe57ef445b5f",
            "placeholder": "​",
            "style": "IPY_MODEL_e5dd1d7553b442d9aa96a382e76b7c31",
            "value": " 493k/493k [00:00&lt;00:00, 1.38MB/s]"
          }
        },
        "7f07f2eb9c3b4a61bf7921a5c0cba8e5": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "000245c963e54f5d85ab98f429ed2fa9": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f2626f6050184645a93216a072b5c756": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "d9219a97dc6d43b380cc08fc78f0d7bf": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "bd68abd454ae4b54b9932a7372a97747": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "956858d0f5ea4746824dfe57ef445b5f": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e5dd1d7553b442d9aa96a382e76b7c31": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}