bpaf description seems a bit odd

[pacak]

In particular what are those rough edges and which parts of the API are confusing?

Comments I encounter on the Internet are mostly in favor of it as having small and flexible API and with derive macro until you start doing strange things the difference with clap is mostly with what you derive and how you run it. And once you do start doing strange things - everything comes from a combination of a few rules similarly to how you would chain iterators...

https://news.ycombinator.com/item?id=35734051

Oddly on the cli parsing thing, I did end up finding a library I like - bpaf. It's kind of abstract but seems more conceptually principled than the near-universal clap. The kind of abstraction where once you have grasped a small set of orthogonal concepts, you can just combine them at will, seems OK to me. As contrasted with the type which has dozens of small rules, and special cases that rely heavily on (human) memory and/or constant doc lookups.

[nicoburns]

For previous discussion, see #20 More feedback/input from the community (including yours) is always welcome.

[pacak]

Hmm... No concrete examples...

"For anything past that though, you need to be steeped in a Haskell mindset to figure out how things work."

Hmm... Let me write down a few examples then. I strongly suspect Rust mindset is sufficient.

"Every time I say "this can't be done with bpaf", the author replies back with some non-obvious application of the available functionality."

Hmm... I don't think that's bad though, only shows that if you want to parse something strange - you can do it. As for applications being non obvious - let me write down some examples...

use bpaf::*;
use std::path::PathBuf;

#[derive(Debug, Clone, Bpaf)]
#[bpaf(options)] // tag for top level parser, used in every derive example
pub struct Options {
    /// Increase verbosity
    verbose: bool,
                                                                                               
    #[bpaf(external)] // least logical part, explained in tutorial used in most examples     
    dependency: Dependency,                                                                        
}                                                                                                  
                                                                                                   
#[derive(Debug, Clone, Bpaf)]                                                                      
pub enum Dependency {                                                                              
    Local {                                                                                                 
        /// Add a local dependency                                                              
        path: PathBuf,                            
    },                                        
    Git {                                     
        /// Add a remote dependency           
        repository: String,                   
        #[bpaf(external)]                     
        rev: Rev,                             
    },                                        
}                                             
                                              
#[derive(Debug, Clone, Bpaf)]
// out of 19 currently present API names fallback seems most logical
// https://docs.rs/bpaf/latest/bpaf/trait.Parser.html#method.fallback
#[bpaf(fallback(Rev::DefaultRev))]
pub enum Rev {
    Branch {
        /// Use this branch
        branch: String,
    },
    Tag {
        /// Use this tag
        tag: String,
    },
    Revision {
        /// Use this commit hash
        rev: String,
    },
    #[bpaf(skip)]
    DefaultRev,
}
  
fn main() {
    println!("{:?}", options().fallback_to_usage().run());
}

This gives you as a programmer a good structure that is easy to consume and a good help to user:
program takes path or repository, with repository it takes optional branch, tag or revision.

$ add --help
Usage: add [--verbose] (--path=ARG | --repository=ARG [--branch=ARG | --tag=ARG | --rev=ARG])

Available options:
        --verbose         Increase verbosity
        --path=ARG        Add a local dependency
        --repository=ARG  Add a remote dependency
        --branch=ARG      Use this branch
        --tag=ARG         Use this tag
        --rev=ARG         Use this commit hash
    -h, --help            Prints help information

$ add --repository foo --tag bar --rev baz
Error: --rev cannot be used at the same time as --tag

Most confusing part is here probably use of extern, but it's documented in the tutorial.

Let's make things more interesting. Instead of taking a single dependency to add we take several. Quick search on Parser trait reveals many

#[derive(Debug, Clone, Bpaf)]
#[bpaf(options)] // tag for top level parser, used in every derive example
pub struct Options {
    /// Increase verbosity
    verbose: bool,

-   #[bpaf(external)]
-   dependency: Dependency,
+   #[bpaf(external, many)]
+   dependency: Vec<Dependency>,
}

Help for users automatically updates to indicate it takes multiple dependencies, programmer gets a vector in a field instead of a single struct.

All the new logic is external, many which you can also write as external(depdencency), many - is equivalent to iterator like chain dependency().many(), first consumes a single item, second changes it to consume a bunch of items.

Say we want the set to be deduplicated and stored as BTreeSet. Parser trait reveals map, a few type annotations and we can write this:

-   #[bpaf(external, many)]
-   dependency: Vec<Dependency>,
+   #[bpaf(external, many, map(|x| x.into_iter().collect()))]
+   dependency: BTreeSet<Dependency>,

Rust equivalent - just adding Iterator::map: dependency().many().map(|x| x.into_iter().collect()).

In currently unreleased version you can write collect directly

-   #[bpaf(external, many)]
-   dependency: Vec<Dependency>,
+   #[bpaf(external, collect)]
+   dependency: BTreeSet<Dependency>,

Say I want boxed string slices instead of strings. Same map gives just that

    Branch {
        /// Use this branch
        #[bpaf(argument::<String>("BRANCH"), map(Box::from))]
        branch: Box<str>,
    },

[pacak]

Now let's go into something more obscure, say we want to implement a blazingly fast dd clone.

Command line looks like this: dd if=/dev/sda of=/dev/sdb bs=4096 conv=noerror,sync

Best candidate to parse arbitrary item from a command line is any
As long as you remember that any is something like iterator you can keep chaining with other stuff from the type it returns and Parser trait... And there's a bunch of examples in the documentation too.

Combining all those ideas we get a function tag that describes dd style options and use it to describe fields in combinatoric style API: https://github.com/pacak/bpaf/blob/master/examples/dd.rs it would work with derive as well.

any is one of the stranger parts of the API which you don't really need until you start doing strange stuff.

[pacak]

"Every time I say "this can't be done with bpaf", the author replies back with some non-obvious application of the available functionality."

How do I parse this input=foo input=bar input=baz input=foo output=foo output=bar output=baz into this 🥺 ?

struct Options {
    io: LinkedList<Io>,
}
enum Io {
    Inputs(CustomHashSet<Box<str>>),
    Outputs(CustomHashSet<Box<str>>),
}

Which answer would you prefer?

• Sure, take this, this and this and you'll get there. Take a look at those examples too.
• This can't be done.

[epage]

Hmm... No concrete examples...

"For anything past that though, you need to be steeped in a Haskell mindset to figure out how things work."

Hmm... Let me write down a few examples then. I strongly suspect Rust mindset is sufficient.

@pacak I in the relevant comment, I quoted from you

Hmm... Maybe, but this naming seems more natural in Haskell. Parser is a proper Applicative you can compose, while OptionParser is just an object you can run or transform a bit, Parser is a very common name and most of them are monads or applicatives. Need to check how other libs are handling this in Rust.

With bpaf, you are very quick to jump into haskell-like ways of discussing things.

"Every time I say "this can't be done with bpaf", the author replies back with some non-obvious application of the available functionality."

Hmm... I don't think that's bad though, only shows that if you want to parse something strange - you can do it. As for applications being non obvious - let me write down some examples...

I don't have examples off the top of my head anymore but when I was coming to you saying "this can't be done", I don't think I was too far off the beaten path.

[pacak]

Hmm... Maybe, but this naming seems more natural in Haskell. Parser is a proper Applicative you can compose, while OptionParser is just an object you can run or transform a bit, Parser is a very common name and most of them are monads or applicatives. Need to check how other libs are handling this in Rust.

With bpaf, you are very quick to jump into haskell-like ways of discussing things.

Well, just for fun I decided to search for things like Applicative and Functor in the repo. Apart from category theory intro Applicative was used twice. Once by me in this context and once by a ticket author. Both in tickets that got resolved within a day. Haskell was mentioned 4 times, mostly by ticket authors but once by me explaining the limitations of derive API.

This specific quote comes from pacak/bpaf#50 while discussing why Parser is called a parser and not something else. There are multiple valid answers to this:

• Naming doesn't really matter, if you are using derive API and not doing anything strange you won't encounter either name anyway. If you are using combinatoric API or combination of both you will be using Parser more often and less typing is better I think.
• Closest analogy in the Rust world is Iterator trait which is also an Applicative Functor, also lazy but can be evaluated right away, with Parser you should to finalize it first so you can attach the documentation. You can run it directly, but there's a deprecation warning pointing to the documentation with a bunch of examples.
• Naming is not that confusing, there's 100+ projects on github using it. There are some private projects using it as well - otherwise I'm not sure how to explain hundreds of daily downloads. I checked source code for most of public projects - most of them seem fine to me. Ticket is still open with a bright label "collecting feedback" - got some useful one, no complains.

I don't have examples off the top of my head anymore but when I was coming to you saying "this can't be done", I don't think I was too far off the beaten path.

I can't find specific conversation but I think we've been talking about validation - some argument is required iff some other arguments are present and user gets back a product type (struct) with fields wrapped in Option such that restriction listed above holds.

In my opinion this is not something users should do. One of the reasons I'm using Rust (and Haskell before) is that I can express a lot of invariants about data I'm working with in the types. If some process can be modeled as being in one of several states with some variables - I'll use enum to represent that, etc. Same applies to inputs and outputs. App gets some data from the network as a slice of bytes - I parse it right away and complain if something is wrong. parse, don't validate. Same goes about command line options. Multiple cases - enum, fields that must be present together - struct. Or sum and product types in ADT terms.

What I was trying to explain is how to encode that invariant in the type system - nested enums where that conditional field goes into one branch with more variants and the rest go into separate branches. In my $dayjob we share bits of CLI parsers across multiple apps and this kind of representation is important so Rust compiler can tell you what needs to be updated. It is still possible to have a huge product type with optional fields and validation with guard on top inside the parser, it's just fragile.

Happy path I'm trying to get users into consists of

• Create data types that best describe the input state in terms of application domain types. enums and structs for overall shape, internal data types (parsed datatypes, for example if an app needs to take ISIN it will be a newtype around [u8; 12] and not a String)
• Add doc comments and annotations for external, fallback, etc.
• Add tests, optional autocomplete, generate documentation, etc.

[epage]

To set the tone: I think there are fascinating ideas in bpaf. I'm hopeful that the usability can be improved, whether directly in bpaf or if another comes along in the same vein.

I'm also wanting to be careful to that we don't go too far off in this discussion, taking away from the main point of this discussion. I believe the overall focus is on how appropriate it is to "bless" bpaf and I'll be focusing on that.

With bpaf, you are very quick to jump into haskell-like ways of discussing things.

Well, just for fun I decided to search for things like Applicative and Functor in the repo. Apart from category theory intro Applicative was used twice. Once by me in pacak/bpaf#120 (reply in thread) and once by a pacak/bpaf#168 (reply in thread). Both in tickets that got resolved within a day. Haskell was mentioned 4 times, mostly by ticket authors but once by me explaining the limitations of derive API.

I don't think the fact that a command-line argument parser has a category theory intro should be glossed over. My perception is also colored by the magazine article on the subject. As someone outside to the project, this helps further cement "this is haskell-complexity".

Naming doesn't really matter

Naming is the most important documentation for a project. For examples of where bad names can come into play, I highlight some in a post of mine

there's 100+ projects on github using it.

That doesn't necessarily mean it isn't confusing.

Evaluating by usage means there are other crates that would be considered first.

I have also seen comments from others who have tried it and given up on it. Granted, one case I'm thinking of they also gave up on clap.

I fully recognize that for trivial cases, for cases where someone is using something like argh, bpaf offers close to the same usability. Its any time you step off that golden path which happens quickly. I'm unsure about nico but for me, I see blessed as safe choices. That complexity cliff in bpaf would disqualify it in my mind from being a safe choice.

[pacak]

My perception is also colored by the magazine article on the subject

This was not a tutorial on bpaf though. This was based on a talk I gave about category theory and its practical applications. Types of abstractions, laws, etc. bpaf is there just to be an example. In fact the conclusion was:

While designing an Applicative style API requires some specialized knowledge - mostly what kind of laws implementation needs to obey and how to check them - using the resulting API does not. With help of the Rust’s type system it’s easy to make sure that for as long as the user’s code typechecks - it works and all compositions are correct.

That complexity cliff in bpaf would disqualify it in my mind from being a safe choice.

Where is this complexity? Sure, if you want to parse dd style, DOS or anything else obscure you'll have to define what they are first. But that's probably the same for any other general purpose parser, if it would let you to do that at all. For anything else it's defining data types and sprinkling some annotations on top of that.

If you set aside the category theory article and going off the documentation and examples - what makes it confusing? Ideally "I tried to do XXX expecting YYY, but got ZZZ instead". Or "I tried reading YYY and found the explanations unclear". I can't really fix "rough edges" and "confusing API" without knowing what those are. Is it just renaming Parser to SubParser? Something else?

[BurntSushi]

I agree with the description of bpaf here. I just spent ten minutes looking at its docs and I'm completely overwhelmed. There is no ready-to-run example in the top-level crate docs. I took a guess and thought maybe OptionParser was where to start, but I just got more confused. The docs say it's a "ready to run Parser," but it doesn't actually implement the Parser trait? And then it says it's created with Parser::to_options, but Parser is a trait... So how do I create it?

I had to click around for quite some time before finally finding a real example. I didn't go there originally because it's call "unusual" and I didn't want an unusual example. I just wanted a standard example.

The API surface area is huge and there are a lot of moving pieces. The API is further obscured, IMO, by the use of generics. The top-level crate docs start with a single sentence about what the crate does and then immediately launches into "Design considerations for types produced by the parser," and I have no idea what the heck that means. I go to expand "A few examples," and all I see are incomplete code snippets that don't use the library at all.

Where is this complexity?

I'd like to suggest you take this comment as an experience report about what I find complex rather than some objective claim over what is and isn't complex.

[pacak]

I'd like to suggest you take this comment as an experience report about what I find complex rather than some objective claim over what is and isn't complex.

Well... I will address at least some of the problems you listed. page on crates.io links to examples on github directly, I should keep something similar on the docs.rs as well. Will think about structuring documentation in a clearer way.

As for API size - there's a Parser trait with 19 functions, about 5 functions that can create it, 10 or so modifiers you don't really need until you want to deal with strange stuff and 14 methods on OptionParser so 48 items in total, is it really that big compared to clap's 142 methods on Command alone?

[BurntSushi]

I don't measure API size by counting API items. My measurement is subjective.

Listen, you asked for specifics. So I gave you ten minutes of my time to go through your crate docs and API and rendered an opinion about it. Take it or leave it. I'm not getting into a tit-for-tat with you about clap. I gave you feedback about your crate based on questions you asked. I didn't do a comparative analysis between your crate and clap.

[pacak]

I appreciate your feedback and will try to address it.

[pacak]

FWIW I updated the documentation and rewrote the tutorials.

A simple parser that parses a single flag -v to get started with looks like this:

fn main() {
    let r = short('v').switch().run();
    println!("{:?}", r);
}

[epage]

@pacak how about we table this until RustConf and power through some usability and documentation discussions then? I feel like there are some disconnects happening when talking about the docs and APIs that some in person, sync conversations might help with.

[pacak]

I feel like there are some disconnects happening when talking about the docs and APIs that some in person, sync conversations might help with.

Sure. I'll see what I can do about the navigation meanwhile.