minOLMo - An Explainable Open Language Model

minolmo has been forked from the original OLMo model, with the primary goal of removing extra complexity and distributed training capabilities. This streamlined version is now a large language model featuring a simplified codebase that is easy to understand and follow. It is designed to be run and explored by researchers on a single GPU, making it accessible for those who want to delve into the workings of large language models without the need for extensive computational resources.

Fork History

• Apr 5, 2024: David Brandfonbrener forked the original OLMo repository to create the min-olmo repository. David removed the distributed training capabilities.
• Apr 22, 2024: Kempner Institute forked the min-olmo repository from David's fork to create the KempnerInstitute/min-olmo repository. Any code after this date is from Kempner Institute affiliated contributors.

Package Structure

The project includes two main categories of files and directories:

• minOLMo python package: This package contains the source code for the model.
• Scripts, configs, and other helper files to run the model.

In the following, we provide a brief description of the main directories and files in the project:

• configs: This directory contains the configuration files for the model. The configuration files include the model parameters, input data, and other necessary parameters to train the model.
• docs: This directory contains the documentation for the project.
  • Single documentation files can be added using a single markdown file.
  • The PDF technical report is located in docs/technical_report directory.
  • Documentations that my need extra files (e.g., images) can be added in a separate directory.
• minolmo: This directory contains the source code for the model.
• scripts: This directory contains the scripts to run the model.
• notebooks: This directory contains the notebooks to explore the model.
• tests: This directory contains the tests for the model.
• CHANGELOG.md: This file contains the changes made to the project.
• README.md: The current file.
• LICENSE: The license file for the project.
• pyproject.toml: The configuration file for building the python package.

Installation

To install the package:

• Step 1: Clone the repository

  • For developers:

    git clone git@github.com:KempnerInstitute/minOLMo.git

  • For users:

    git clone https://github.com/KempnerInstitute/minOLMo.git

• Step 2: Create a conda environment

  • Please visit: Setting up a conda environment

• Step 3: Install the package

  • To install the package, you need to run the following command:

    pip install -e .

Build the documentation locally

• Step 1: Install the required packages

pip install -e '.[docs]'

• Step 2: Build the documentation

cd docs
make html

• Step 3: Open the documentation in your browser

open _build/html/index.html

Other useful resources

This resources are being updated in a reverse chronological order. The most recent resources are at the top. Some documents (e.g., single runs) might be outdated, to see the exact codes, please checkout to the specific commit of that report.

• Submitting a sweep job
• min-olmo with one GPU, last update [April 22, 2024]
• How to contribute to the project?