Plan to proceed with the refactor of formatting functions

[mmatera]

Plan for refactor MakeBoxes and formatting

What we have learn about how formatting works in WMA

Differently to other parts of WMA, the way in which boxes works is particularly obscure, and is
hidden even when we look at "low-level" tracing tools, like TracePrint or trace debug. So to
establish how expressions must be formatted, or how MakeBoxes[...] expressions should be
evaluated relies mostly on doing experiments and guessing. These are some of my findings studing it:

• MakeBoxes[expr, fmt] only should allow a fmt listed in $BoxForms.

  In[1]:= MakeBoxes[a+b, OutputForm]
  MakeBoxes::boxfmt: OutputForm in MakeBoxes[a + b, OutputForm] is not a box formatting type. A box formatting type is any member of $BoxForms.
  Out[1]= MakeBoxes[a + b, OutputForm]
  In[2]:= MakeBoxes[a+b, OutputForm]:="x"
  In[3]:= MakeBoxes[a+b, OutputForm]
  MakeBoxes::boxfmt: OutputForm in MakeBoxes[a + b, OutputForm] is not a box formatting type. A box formatting type is any member of $BoxForms.
  Out[3]= MakeBoxes[a + b, OutputForm]
  In[4]:= Unprotect[Plus]; MakeBoxes[a+b, OutputForm]^:="x"
  In[5]:= MakeBoxes[a+b, OutputForm]
  MakeBoxes::boxfmt: OutputForm in MakeBoxes[a + b, OutputForm] is not a box formatting type. A box formatting type is any member of $BoxForms.
  Out[5]= MakeBoxes[a + b, OutputForm]
  In[6]:= $BoxForms
  Out[6]= {StandardForm, TraditionalForm}
  

  It is possible to extend $BoxForms by using AppendTo,

  In[7]:= AppendTo[$BoxForms, MyNewBoxForm]
  Out[7]= {StandardForm, TraditionalForm, MyNewBoxForm}
  In[2]:= $PrintForms
  Out[2]= {InputForm, OutputForm, TextForm, CForm, FortranForm, ScriptForm, MathMLForm, TeXForm, StandardForm, TraditionalForm, MyNewBoxForm}
  

  and automatically it is included in $PrintForm. However, we can not use it in a MakeBoxes Expression until we assign a ParentForm
  But we can only
  https://reference.wolfram.com/language/ref/message/ParentForm/deflt.html.en

  In[3]:= MakeBoxes[X, MyNewBoxForm]
  ParentForm::deflt: The ParentForm of ParentForm[MyNewBoxForm] is not defined on $BoxForms.
  Out[3]= X
  In[4]:= ParentForm[MyNewBoxForm]^=TraditionalForm
  Out[4]= TraditionalForm
  In[5]:= MakeBoxes[X, MyNewBoxForm]
  Out[5]= X	
  

  If fmt is in $PrintForms but not in $BoxForms, then the rules should be
  of the form MakeBoxes[Format[expr_,fmt], boxfmt_] or MakeBoxes[fmt[expr_], boxfmt_]:

  In[1]:= Format[F[x_],MyForm]:="?"
  In[2]:= $PrintForms
  Out[2]=  {InputForm, OutputForm, TextForm, CForm, FortranForm, ScriptForm, MathMLForm, TeXForm, StandardForm, TraditionalForm, MyForm}
  In[3]:= FormatValues[F]
  Out[3]= {HoldPattern[?] :> ?}
  In[4]:= Format[F[2],MyForm]
  Out[4]= ?
  In[5]:= Format[F[2],MyForm]//MakeBoxes
  Out[5]= InterpretationBox["?", ?, Editable -> True, AutoDelete -> True]
  

  It it useful to define a format rule when MyForm is the head of an expression. Then, by wrapping the
  expression with the form name, we get the correspoindig form, and the form appears in the output line:

  In[9]:= Format[MyForm[F[x_]]]:=Format[F[x],MyForm]
  In[10]:= F[3]//MyForm
  Out[10]//MyForm= ?
  

  If we now set a MakeBoxes rule for this symbol (as an upvalue), the rule only applies
  when the expression is wrapped in StandardForm:

  In[11]:= MakeBoxes[MyForm[F[x_]],StandardForm]^:="box"
  In[22]:= F[2]//MyForm
  Out[22]//MyForm= ?
  In[23]:= F[2]//MyForm//StandardForm
  Out[23]//StandardForm= box
  

  When an expression does not have a MyForm rule that match, OutputForm is used:

  In[24]:= In[33]:= a/b//MyForm
                          a
  Out[33]//MyForm= MyForm[-]
                          b
  

• InputForm, OutputForm and FullForm must be processed using FormatValues associated to them, but not
  using MakeBoxes. MakeBoxes OutputForm / FullForm #1163 and Independent FullForm #1316 goes in that line.

• MakeBoxes[] specific rules should be stored as format rules associated to the
  corresponding symbols, and not as downvalues rules associated to MakeBoxes.
  Notice for example that UpValues are not used in MakeBoxes:

  (*Set a MakeBoxes  rule as an UpValue of Plus*)
  In[1]:= Unprotect[Plus]; UpValues[Plus]:={HoldPattern[MakeBoxes[Plus[x__],StandardForm]]:>"PP"}	  
  Out[1]= {HoldPattern[MakeBoxes[+(x__), StandardForm]] :> PP}
  (*Not used in formatting or in evaluating a MakeBoxes expression:*)
  In[2]:= a+b//StandardForm									  
  Out[2]//StandardForm= a + b
  In[3]:= MakeBoxes[Plus[a,b],StandardForm]							  
  Out[3]= RowBox[{a, +, b}]
  (*But the rule matches....*)
  In[4]:= Hold[MakeBoxes[a+b,StandardForm]]/.HoldPattern[MakeBoxes[Plus[x__],StandardForm]]:>"PP"	  
  Out[4]= Hold[PP]
  (*UpSet set a FormatValue to Plus:*)
  In[5]:= MakeBoxes[Plus[x__],StandardForm]^:="PP"
  In[6]:= FormatValues[Plus]
  Out[6]= {HoldPattern[MakeBoxes[+(x__), StandardForm]] :> PP}
  (*Now the rule is applied*)
  In[7]:= a+b//StandardForm									  
  Out[7]//StandardForm= PP									  
  

  This also allows to remove the special case in mathics.core.builtin.Builtin.contribute
  that requires that a MakeBoxes definition be loaded before any other.
  The only special case must be the "general" rule MakeBoxes[expr_, fmt_]
  that look for format rules and apply them to expr.

  Makeboxes rules are applied with the following priority:

  • UpValues rules are applied before than downvalues:

    In[1]:= MakeBoxes[F[x_],StandardForm]:="O";
    In[2]:= MakeBoxes[F[x_],StandardForm]^:="Q";
    In[3]:= FormatValues[F]                                                         
    Out[3]= {HoldPattern[MakeBoxes[F[x_], StandardForm]] :> Q}
    In[4]:= FormatValues[MakeBoxes]
    Out[4]= {HoldPattern[MakeBoxes[F[x_], StandardForm]] :> O}
    In[5]:= F[32]//StandardForm
    Out[5]//StandardForm= Q (*The upvalue Q is used*)
    

  • User defined rules are applied over builtin rules:

  In[1]:= Unprotect[Plus]; MakeBoxes[Plus[x___],StandardForm]:="P"
  In[2]:= a+b //StandardForm
  Out[2]//StandardForm= P
  In[3]:= Clear[MakeBoxes]
  In[4]:= a+b //StandardForm
  Out[4]//StandardForm= a + b
  In[5]:= Unprotect[Plus]; MakeBoxes[Plus[x___],StandardForm]^:="P"
  In[6]:= a+b //StandardForm
  Out[6]//StandardForm= P   
  

• Assignment to Format with a single argument defines both a generic MakeBoxes rule and a Format rule:

  In[1]:= Format[F[x__]]:=Q[x,1]
  In[2]:= FormatValues[F]//FullForm
  Out[2]= { HoldPattern[MakeBoxes[F[Pattern[x, BlankSequence[]]], Pattern[FormatType, Blank[]]]]:> Format[Q[x, 1], FormatType],
            HoldPattern[Format[F[Pattern[x, BlankSequence[]]]]]:> Q[x, 1]]}
  In[3]:= F[2]//StandardForm
  Out[3]//StandardForm= Q[2, 1]
  

  The first rule handles a $BoxForm as it was a $PrintForm. The second rule applies to any $PrintForm. Notice that
  by setting a MakeBoxes as downvalue we can not modify the result:

  In[4]:= MakeBoxes[F[x__], StandardForm]:="---"
  In[5]:= F[2]//StandardForm
  Out[5]//StandardForm= Q[2, 1]
  

  By assigning as UpValue, then we get the effect:

  In[6]:= MakeBoxes[F[x__], StandardForm]^:="---"
  In[7]:= F[2]//StandardForm
  Out[7]//StandardForm= ---
  

  Notice now that if we overwrite the standard rule for MakeBoxes, all the outputs tagged as
  StandardForm are affected:

  In[8]:= DownValues[MakeBoxes]:={HoldPattern[MakeBoxes[expr_,StandardForm]]:>"generic rule for StandardForm"}
  Out[8]= {HoldPattern[MakeBoxes[expr_, StandardForm]] :> generic rule for StandardForm}
  In[9]:= F[2]//StandardForm
  Out[9]//StandardForm= generic rule for StandardForm
  In[12]:= BesselJ[n,x]//StandardForm
  Out[12]//StandardForm= generic rule for StandardForm
  

  but OutputForm is not affected:

  In[14]:= F[2]//OutputForm
  Out[14]//OutputForm= Q[2, 1]
  In[15]:= BesselJ[n,x]
  Out[15]= BesselJ[n, x]
  

  InputForm, FullForm and any other $PrintForm are neither affected:

  In[16]:= F[2]//InputForm
  Out[16]//InputForm= F[2]
  In[17]:= Format[g[expr_],MyForm]:="myform..."
  In[18]:= F[21]//MyForm
  Out[18]//MyForm= MyForm[Q[21, 1]]
  In[19]:= Format[F[21],MyForm]
  Out[19]= Q[21, 1]
  

  This is because $PrintForms do not make use of MakeBoxes rules, but just the Format[...] rules.

• Assignment to MakeBoxes should store FormatValues to MakeBoxes (if a DownValue)
  and to the Head of the first element (MakeBoxes[f[x_],fmt_]^:=formatfunc[x,ftm]
  must be stored as a FormatValue of f).

How I guess we should proceed

To get better compliance with WMA, I think we should introduce changes in sequence:

• First, add specific routines to implement FullForm, InputForm and OutputForm
  formatting, without relying on MakeBoxes. I already started to do this.
• Change the MakeBoxes[expr, form] rules for forms not in $BoxForms to
  MakeBoxes[form[expr], boxform_]. This includes MathMLForm and TeXForm.
• Move specific rules for MakeBoxes from downvalues of MakeBoxes to FormatValues
  to the corresponding symbols, and add an eval function that applies these rules.
  This would be the larger change.
• Modify the assignment code to modify where MakeBoxes rules are stored in assignments.

Of course, there are more, but once we reach this point, the following changes will be smaller, and can also be implemented as autoload WL modules.

[rocky]

I have been going over the information here and thinking about this over the last couple of days.

Even though these thoughts appear after other discussions such as in PRs, those discussions have been motivated by trying to wrap my head around what has been proposed here.

I think what we should do try to narrow the initial work to just fixing up MakeBoxes. Let's defer revising any of the forms like Fullform or adding Character-based 2D rendering for later.

I think there is enough to do in just getting MakeBoxes, in the areas it currently handles, to conform better to WMA.

And I suspect making even small changes to MakeBoxes is going to drastically change output that a lot of our tests are going to fail. If this is the case, then to mitigate this we can create a new makeboxes routine NMakeBoxes, and have the two work side by side.

There is an art to trying to simplify and narrow a big problem. In particular one needs to make sure that the simplification we goes with the natural grain or groove of the problem. The analogy I've given in the past is in solving a Rubik's cube by its underlying subgroups instead of the visual proximity of a corner piece and its neighboring 3 edge pieces.

So the specific details given here should help ensure that we are not going against the natural grain in reimplementing MakeBoxes.

To start with something simple, consider this.

WMA:

In[1]:= MakeBoxes[a b]                                                                               
Out[1]= RowBox[{a,  , b}]  (* There is a space character, " ", between the two commas, but rendering drops double quotes *)
In[2]:= MakeBoxes[a + b]                                                                             
Out[2]= RowBox[{a, +, b}]

Mathics3:

In[1]:= MakeBoxes[a b]
Out[1]= RowBox[{Times, [, RowBox[{a, ,, b}], ]}]
In[2]:= MakeBoxes[a + b]
Out[2]= RowBox[{Plus, [, RowBox[{a, ,, b}], ]}]

Can we write an NMakeBoxes function that does not violate the operational details outlined here and that makes this work?

[mmatera]

The difference between the WMA and Mathics behavior is that MakeBoxes does not apply first the format rules, as we do in format_element. But yes, I can prepare an implementation of NMakeBoxes, following that behavior.

[rocky]

I'd greatly appreciate it if we had a simple implementation - simple as possible, but no simpler - of NMakeBoxes that fixes this behavior.

To start, NMakeBoxes can just handle StandardForm. Then after seeing how much we can cover with that, we can expand to the other forms in $BoxForms. And then after that, the user-defined $PrintForms as given in the example before.

But to make it easy to see what's going on, let's start very simple, keeping in mind caveats that we know that will come up, and steering clear of things that would make handling those harder. when we get to them. Thanks!