Data Usage policy for pyOpenSci - what does our policy look like for data usage?

[lwasser]

Hi Everyone,
I know that this issue of data usage is an important one. As such i wanted to start an issue to capture community thoughts on the issue. Based upon our first iterations below is the language that we are using around data usage.

### Telemetry & user-informed consent

Your package should avoid collecting usage analytics. With
that in mind, we understand that package-use data can be invaluable for the
development process. If the package does collect such data, it should do so
by prioritizing user-informed-consent. This means that before any data are
collected, the user understands:

1. What data are collected
2. How the data are collected.
3. What you plan to do with the data
4. How and where the data are stored

Once the user is informed of what will be collected and how that data will be handled, stored and used, you can implement `opt-in` consent. `opt-in` means that the user agrees to usage-data collection prior to it being collected (rather than having to opt-out when using your package).

We will evaluate usage data collected by packages on a case-by-case basis
and reserve the right not to review a package if the data collection is overly
invasive.

There are some good, important points here regarding how maintainers collect data to support development

For maintainers

• The data can be useful to improve user experience
• The data can inform / focus development efforts

On the user end:
There is so much assumption today that collecting data is ok and is even a part of many organizations (think facebook) business model. In return the offer users a tool / service at now / low cost.
However there is a level of trust and ethical acknowledgement to consider. People should have some control about data derived from their activities.

We have a few models such as homebrew which up front is clear about collecting data upon install. the user can opt in or out there.
Is there a model for o(scientific) python that we could follow that would balance the needs of developers with the privacy of users?

References to two items:

• issue where we are reviewing a package that collects data
• Open pr with a new policy being drafted

pinging:

@sneakers-the-rat @NickleDave @tupui @stefanv @Batalex @yuvipanda @pradyun @choldgraf @skrawcz and anyone else who has some thoughts on what our policy looks like. Let's have an open, productive conversation here so we have a record of it!

[tupui]

Thank you for putting this together.

Some thoughts:

I am +1 on ethical data collection as described. Maybe add some wording like "ethical" and how data collection can be used. Also if we have practice we want to forbid, we should state it.e.g. not allow to be able to build user profiles, forbidden data to track, compliance with other laws like RGPD, etc.

One data point I think is Apple, devs are now requested to provide "card" explaining what their app is collecting etc. The card is normalised which is helpful for user as the information is easily accessible and understandable. I think we should do something similar, provide a template basically and tell how/where the info should be presented to the users. e.g. I would suggest having a section in the readme as it's important (also what Apple does, it's on the main page description of apps.)

I think we need to be careful here. Data collection is very sensible and if you do it wrong, liability can be at stake in some countries (and imagine if you leak sensitive info...). Might be worth to have some legal wording which would work in most countries (i.e. this need to be seen by some international lawyers IMHO.)

[Batalex]

As someone who has been involved in the financial & healthcare industries, two heavily regulated sectors, I agree with the community stance.
Open-source software is provided "as is," and developers would be well within their rights to include telemetry. However, no one likes a nosy neighbor coming unannounced to the party.
Moreover, I feel this would strain the trust the users may put in the community if their sysadmin starts slamming them for "unauthorized outgoing traffic". It is already hard enough to demonstrate the benefits of open-source software in an environment with a need for compliance. Let's not make it harder!

[tupui]

Agreed, the opt-in condition is here paramount and should be non-negotiable.

[NickleDave]

In the spirit of having an open discussion here, I want to add a link to the Command-Line Interface Guidelines that @pradyun helpfully shared in the pyOpenSci Slack, specifically the section on "analytics":
https://clig.dev/#analytics

(and maybe this will save @pradyun some typing 🙂 who I'm sure would also share it here)

Some quotes:

unlike websites, users of the command-line expect to be in control of their environment, and it is surprising when programs do things in the background without telling them.

Do not phone home usage or crash data without consent. Users will find out, and they will be angry. Be very explicit about what you collect, why you collect it, how anonymous it is and how you go about anonymizing it, and how long you retain it for.

Ideally, ask users whether they want to contribute data (“opt-in”). If you choose to do it by default (“opt-out”), then clearly tell users about it on your website or first run, and make it easy to disable.

Examples of projects that collect usage statistics:

• Angular.js collects detailed analytics using Google Analytics, in the name of feature prioritization. You have to explicitly opt in. You can change the tracking ID to point to your own Google Analytics property if you want to track Angular usage inside your organization.
• Homebrew sends metrics to Google Analytics and has a nice FAQ detailing their practices.
• Next.js collects anonymized usage statistics and is enabled by default.

Consider alternatives to collecting analytics.

• Instrument your web docs. If you want to know how people are using your CLI tool, make a set of docs around the use cases you’d like to understand best, and see how they perform over time. Look at what people search for within your docs.
• Instrument your downloads. This can be a rough metric to understand usage and what operating systems your users are running.
• Talk to your users. Reach out and ask people how they’re using your tool. Encourage feedback and feature requests in your docs and repos, and try to draw out more context from those who submit feedback.

Further reading: Open Source Metrics

Most of what's there aligns with what's already been said. And the fact that it's a resource provided by a community that's already done a lot of thinking about this reinforces those points.

I think it applies to research software more generally, not just CLIs.

In Slack, I think I'm the only one who voted for "either opt-in or opt-out". It seems like our community in particular is really sensitive to issues around analytics; it could look really bad if you found out your HPC cluster or whatever was sending off data to who knows where. Which makes me think we'd want to lean more towards "opt-in only", even if as devs we'd really love to have that usage info.

[stefanv]

In Slack, I think I'm the only one who voted for "either opt-in or opt-out". It seems like our community in particular is really sensitive to issues around analytics; it could look really bad if you found out your HPC cluster or whatever was sending off data to who knows where. Which makes me think we'd want to lean more towards "opt-in only", even if as devs we'd really love to have that usage info.

+1 With libraries, there is often no user interaction. That cannot be taken as implicit permission to send data!

[choldgraf]

I agree with everything in the policy as a first pass, except for the first sentence. I think we should encourage more practices where projects thoughtfully collect data about how people use their tools. What we want to avoid is people doing so unthoughtfully , in a way that has unintended consequences that conflict with the values pyOpenSci wants to promote. So I'd suggest adding "uninformed" to the first sentence and then making it a "must" rather than "should"

Also a meta point from a practicality perspective: I think it's important to consider that there are three things often true in projects:

1. They would benefit from and want basic information about how their users are using their tools and documentation
2. They have zero resources to collect it on their own with any kind of manual labor
3. They don't have many options to try

I think this is why everybody uses google Analytics - it's not because they want to per se, it's because it is their only option to realisticically solve a problem they have. I think that if policy and guidelines don't appreciate this nuance, then people will just not follow the policy, or they will do the bare minimum to follow the rules but not it's spirit. (Eg this is why everybody just ignores policy in universities - they are given a mandate but not empowered and resourced to follow it).

I appreciate the language in your proposal above that touches on this, I think it helps to say "we know this might be hard to follow and here are some things you can do to take minimal steps forward". I also think that if POS wants to mandate more ethical data collection, it should find ways to provide resources to projects to empower them to do it (eg, maybe it contracts with a UX firm to run user interviews as a service, maybe it runs its own plausible or motomo instances for people that want to use web tracking)

[NickleDave]

👍 for changing the first sentence to something like

Your package must avoid collecting usage analytics without informing people using it.

[tupui]

must+avoid is ambiguous. In this sentence it would be better to say: must not

[lwasser]

Good. morning everyone! I am going to remove the telemetry from our current PR and make a new PR so our process of deciding language is explicit. One question about this.

@choldgraf wrote above:

They would benefit from and want basic information about how their users are using their tools and documentation
They have zero resources to collect it on their own with any kind of manual labor
They don't have many options to try
I think this is why everybody uses google Analytics

Frankly this is exactly the issue that has been bothering me (and that i've been researching without finding a good path forward) for the past months. one note that @skrawcz mention was that developers ARE providing a service to users and that data (as chris points out) is very valuable. GA is free and easy to setup. it's also baked into numerous docs etc templates. plausible and matomo for web analytics are great but require infrastructure / $$. for our project i plan to pay for matomo but there is no way that in general we can expect volunteers to start to pick up bills for self hosted analytics. especially the smaller projects that we review.

So - my question for y'all is what are the options for actually implementing this (aside from a route like homebrew takes which IS opt out) ? I think homebrew does a good job of explaining what they collect and how.

[lwasser]

For reference: PR with telemetry policy only can be found here

[skrawcz]

Late to the party here. I will write more when I get more of a chance (hopefully next week), but here is a quick high level thought:

• No one is forcing anyone to use open source software.

Thus, I'm of the opinion that end users can ultimately make their own decision about what's best for them. But not everyone knows how to evaluate/understand implications, and thus they need a trusted entity to:

a) set clear and transparent guidelines
b) grade the project against those guidelines

With all that said, I think opt-in vs opt-out shouldn't be dictated by guidelines, instead the guidelines should specify what is or isn't acceptable under those different mechanism for getting telemetry.

[betatim]

As a user I dislike opt-out as a model, mostly because I tend to find out about it "by accident" and then am surprised. And negative surprise quickly turns into anger.

Yet I also think that opt-in is not so useful, almost to the point that I'd say it isn't worth discussing a policy around "opt-in" given how much work that would be, how much work it would be for library authors and how little insight comes out of it at the end.

Taking these two together I think there needs to be room for "opt-out" models and you need to find a way to remove the "surprise factor". Generally in life it seems most people are happy to do something if you explain to them upfront what it is, why you'd like to do it and how it would help you. However, they will not agree to exactly the same thing if they are surprised or otherwise feel tricked.

For example, I think I'd be totally happy to once a year let a library report the top N functions/classes/methods from its library used in the last week. Especially if you ask me beforehand and explain how this is useful to the library development. If you also want to report things about the function arguments, my system, a "unique" user ID, etc I'd probably quickly say "nope". In part because I'd start worrying that your reporting is too complex and hence the chances of you accidentally leaking some of my data (not just stats about it) are too high. And I don't see why you need to know this.

This means that I think finding a model that allows opt-out, collects data with a limited scope and doesn't surprise the user is worth the effort. A key challenge here is how a library can interact with the user (I think for applications like Jupyter it is ~trivial to see how the interaction would work).

---

The reason I think opt-in is nearly not worth the time to discuss is that I have no idea how I'd learn about a library having this functionality and then activating it. I'm happy to help you, but if I need to do a lot of legwork (even having to look up the name of that environment variable I'd have to set is probably too much) I'm likely to just forget about it.

[tupui]

Some of what you are saying is contradictory.

there needs to be room for "opt-out" models and you need to find a way to remove the "surprise factor"
...
Especially if you ask me beforehand

This is exactly what opt-in is.

Regardless, user data collection in any form is regulated and if we want to stay on the safe side, we must not do it in the background without prior explicit user consent.

Yes, for opt-in to be effective, people need to know it exists and activate the thing. This can be achieve by promoting the feature widely in the ecosystem, doing outreach, workshop explaining why we want that, etc. Even a few reports would already be more helpful than 0. And as the time goes the situation would improve.

I think it's better to be transparent and put ourselves out there for such things. Data collection is a very sensible topic nowadays and for it to not backfire, the communication needs to be rock solid. Yes it's very hard, and even big corps sometimes fail at this.