RFC: Adding Telemetry to Axolotl

[djsaunde]

This RFC proposes adding opt-out telemetry to Axolotl to better understand user engagement patterns and improve the library's functionality. The telemetry system will be designed with user privacy as the top priority, collecting only non-personally identifiable information (non-PII) about library usage patterns and system configurations.

Motivation

Currently, we lack visibility into how users interact with Axolotl, which models and configurations are most popular, which hardware configurations are most common, and where users encounter issues. Since Axolotl runs exclusively on users' local machines or their configured cloud providers, we have no centralized way to understand usage patterns.

This information gap makes it challenging to:

1. Prioritize feature development for the most commonly used models and configurations
2. Identify common failure patterns
3. Optimize performance for popular hardware configurations

Design

• Privacy first: No collection of personally identifiable information (PII), no tracking of individual users, no tracking of private model or dataset usage, and all data collected is strictly related to library usage and system config.
• Transparency: We plan to document all the collected data points, open source the telemetry implementation as part of the axolotl library, and regularly report stats / insights gained to the community.
• Opt-out: We are planning to implement this on an opt-out basis using env variables / a CLI argument. We will clearly document / log to the command line about how to disable the telemetry. No library functionality will be removed if the telemetry is disabled.

Data Collection Points

• Pre-training Start
  • YAML configuration (sanitized of paths and personal information)
  • Hardware information (GPU count, types, available memory)
  • Operating system, Python version
• Model Configuration (post Hugging Face Hub download)
  • Base model architecture and size
  • Model-specific architecture details (hidden size, layer count, attention heads)
  • Tokenizer configuration and special tokens
  • Hardware optimization support (flash attention, etc.)
  • Quantization configuration
  • Training approach (LoRA, QLoRA, full finetune, etc.)
• Training Events
  • Training start / completion
  • Runtime metrics (epochs, steps, time per epoch)
  • Memory usage
  • Error messages (sanitized of paths and personal information)

Implementation Details

Each training run will be assigned a unique ID (UUID4) that will be used to associate different telemetry events from the same run. This ID will not be tied to any user information but will allow us to understand the flow of events in a single training session.

run_id = uuid.uuid4()  # Generated at the start of each run

Env Variables

Users can opt out of telemetry using environment variables:

DO_NOT_TRACK=1          # Global opt-out signal respected by many applications
AXOLOTL_DO_NOT_TRACK=1  # Axolotl-specific opt-out

The telemetry system will respect both the global DO_NOT_TRACK environment variable (an established convention across many OSS applications) and the Axolotl-specific AXOLOTL_DO_NOT_TRACK variable. If either of these is set to 1, no telemetry will be collected.

Data Structure

We are planning to track events using PostHog. We'll use PostHog's built-in properties for system information where available, and custom properties for Axolotl-specific data. E.g.:

telemetry_event = {
    "run_id": "uuid4-string",
    "timestamp": "iso8601-timestamp",
    "event_type": "training_start|training_complete|error",
    "hardware_info": {
        "gpu_count": int,
        "gpu_types": List[str],
        "total_memory": int,
    },
    "config": {
        "base_model": "...",
        # etc. YAML config
    },
    "metrics": {
        "duration": float,
        "peak_memory": int,
        "steps_completed": int,
    }
}

Privacy and Security

• Data sanitization / PII removal: All file paths will be removed, user and system-specific info will be removed, and error messages will be sanitized.
• Whitelisting: We will maintain a whitelist of known organizations and models (Llama, Mistral, etc.), and only collect information for whitelisted entries.

Alternatives?

• Opt-in telemetry: This is more privacy-focused, but would probably result in low participation rates, making the data less useful for improving the library.
• User surveys: We could collect information through periodic surveys, but this wouldn't provide real-time usage data and would likely have low participation rates.

Implementation Considerations

• Performance
  • Telemetry operations will run in a separate thread to prevent blocking
  • Failed telemetry sends will fail silently without retries
  • Telemetry data will be batched and sent periodically
  • Memory overhead will be limited by using a fixed-size queue for pending telemetry events
• Telemetry storage / retention
  • Stored for a maximum of 12 months
  • After 12 months, data will be aggregated into summary statistics and individual records will be deleted
  • All stored data will be encrypted at rest
  • Access to raw telemetry data will be limited to core maintainers
• Organization / model whitelist
  • Whitelist will be maintained in a separate config file in the repository
  • Updates to the whitelist are subject to the usual PR review process
  • Users / orgs can request addition to or removal from the whitelist through GitHub issues

Next Steps

1. Community feedback period (2-3 weeks)
2. Address privacy concerns and update design
3. Create implementation plan
4. Begin rollout

[icec102]

"opt-out"??? No thanks. Not cool.

[icec102]

"opt-in" + nice request to participate = +1

[djsaunde]

Thanks for the feedback. This is definitely something we're considering, but opt-in can be challenging since we expect to get less data in this setting, and the data that we do get could have a selection bias (see @nyxkrage's comment).

Something else to note (as per @AlpinDale's comment on Discord) is that telemetry is fairly common in open source in order to get feedback from users (e.g., huggingface libraries have some implementation of opt-out telemetry).

[nyxkrage]

My thoughts on telemetry is that it needs to be opt-out or you might as well not bother. The people who do opt-in to something like telemetry are in my experience the same people who will take the time to write a detailed bug report/issue.

Perhaps a compromise for the "no opt-out telemetry at all" group would be to have axolotl prominently display a warning/message and then wait for X seconds to let the user cancel and disable the telemetry if they so wish. Though this should probably be accompanied by a setting to simply skip the wait (perhaps unifying it into a single AXOLOTL_TELEMTRY=yes|no, with it defaulting to yes + the wait)

But overall my stance is that telemetry, even if opt-in, would be a boon for the project. Though I don't believe opt-in would be worth the effort to set up infrastructure and code changes.

[djsaunde]

Thanks for the comment, @nyxkrage. Your point about the warning / message is interesting; we'll definitely chat about it internally.

[fizzAI]

I would have to see an implementation that isn't a listicle that reads like Cursor slop (no offense :p), but as long as nothing too bad is sent upstream (private model names, private dataset samples, etc) and its redacted on the client I dont have much of a problem with opt-out telemetry here
In fact it might even be interesting if you do store certain config details like hyperparams -- for example, it would be interesting to see statistics as to whether there's a consensus on learning rates for specific model families

[djsaunde]

Thanks for the feedback!

Yeah, I wrote all the notes and had Claude format it, and then added edits on top. But I hear you, it's kind of ugly. I'll do better in the future.

No code implementation yet as we're just gathering feedback prior to writing any code. That's our goal with this RFC.

Yup, we're approaching this with privacy top of mind: no PII, no storage of anything sensitive. And that can be verified once our implementation is done (if we end up taking this route).

Great point about statistics. We plan on having a stats rundown posted in the Discord server on a regular basis (focusing on interesting stats, of course).

[fergusq]

I believe all data collection should be based on consent instead of opt-out. You could show a message on startup asking for consent, or even refuse to start unless a variable or command line flag is present. That is the only alternative to opt-in that ensures that no unwanted telemetry is collected. Adding opt-out mechanisms is bad, as many people likely wouldn't even know that there is telemetry and would accidentally leave it on.

If few people give consent, then it implies that this feature is not wanted and that is a risk that just needs to be taken.

[djsaunde]

Thanks for the feedback!

...even refuse to start unless a variable or command line flag is present

This is an interesting idea, and one we haven't considered. Definitely something to think about.

If few people give consent, then it implies that this feature is not wanted and that is a risk that just needs to be taken.

Totally understand this. We're trying to find the lowest friction way to understand how folks are using the software, so thanks for bearing with us :)

[invisietch]

+1 for the idea that an env variable should have to be present, informed consent is essential here. It may be worth splitting this into multiple env vars so the user can decide exactly which data to share.

If not present, present a warning with all the available options and exit without collecting data.

[fizzAI]

If you go that route, maybe make it configurable by config file too? So if you want to start a run the barrier isnt significantly higher than writing the rest of the config (it's not really an issue for some ppl but it can be a huge mental leap to have to configure ANOTHER thing outside of the regular config after installation)

[invisietch]

If you go that route, maybe make it configurable by config file too? So if you want to start a run the barrier isnt significantly higher than writing the rest of the config (it's not really an issue for some ppl but it can be a huge mental leap to have to configure ANOTHER thing outside of the regular config after installation)

Yeah, that would make sense as well. Would have to update all the existing example configs though.

[winglian]

It's worth pointing out that you're probably already sending data to HuggingFace, they just happened to have this implemented from the beginning. https://huggingface.co/docs/huggingface_hub/en/package_reference/environment_variables#hfhubdisabletelemetry

[fizzAI]

Now I agree with you in theory... but "these other people do it so it MUST be okay if we do it" is not the greatest defense policy xD

[mashdragon]

This is where Axolotl can shine. After you bring this up I feel like Huggingface has pulled a fast one on me. Axolotl can be transparent about everything and make it easy for me to opt out of Huggingface data collection.

Let's create a new config item, huggingface-telemetry-opt-out or something, make it easy for users to define their preferences there.

Maybe you can put axolotl-telemetry-opt-in in the default config files, at least users will SEE it and it be visible.

Axolotl needs to rise above Huggingface and have values that its users care about. Privacy is a big issue people value.

[mashdragon]

I am uncomfortable with sharing even the smallest amount of data. Thanks for alerting me to https://consoledonottrack.com/, at least that helps patch up the huggingface telemetry. But it is sad that as a user I have little control over my information, even basic device information, and have to blindly trust that open source projects and the telemetry providers like PostHog that serve them respect my preferences.

I understand the need and desire for data collection. It would be great to see beforehand what configurations go OOM and which ones don't. That would be super valuable, save countless hours and money, etc.

But unless it's extremely obvious to the users I would not support it. Like @fergusq says, refusing to start axolotl until the user has made a decision would be something I would be okay with. Like the Debian installer which requires users to choose one or the other:

[aronchick]

may i ask if you care that much, why not just fork the code and maintain your own branch?

[mashdragon]

In many cases I have to do that. I would prefer not another instance here. We already have to dodge enough tracking from companies, as an open source community we need to ensure we don't start taking away our core freedoms.

If you read what I am saying I am not opposed to this proposal as long as it's sufficiently clear and easy before even installing the package to opt out.

[aronchick]

100% agreed! I just think that - for better or worse - we should really empower the maintainers rather than ask them to jump through hoops when so few of the users need a feature (e.g. opt-in for anonymized data).

[invisietch]

100% agreed! I just think that - for better or worse - we should really empower the maintainers rather than ask them to jump through hoops when so few of the users need a feature (e.g. opt-in for anonymized data).

Judging by the comments here and on the Discord, a lot of users need this feature.

[aronchick]

I prefer the reverse. This is open source software and the environment variables could not be clearer how to opt out. The telemetry events are super super clear, and do not contain PII. I hate the idea that the maintainers on this project are being forced to do gymnastics to even understand what users are doing in order to build better products.

If you care about telemetry, then we should have a notice at the start that says how to opt-out which should be no more complicated than setting an environment variable. Let's not make the lives of the maintainers here harder!

There's even an industry standard for this! https://consoledonottrack.com/

[mashdragon]

Please note that this "industry standard" was entirely unknown to me before this discussion and it would be nice if Axolotl included clear easy instructions for opting out if @winglian goes this direction, already seems like he made up his mind and of course I respect the maintainers' decision.