Top-level specification of programming language is broken

[pdawyndt]

TESTed is supposed to support test suites that are specific for a single programming language, by specifying a top level programming_language attribute. However, this functionality does not seem to work. Here's an example of a Java-specific test suite

namespace: "Numbers"
programming_language: "java"
units:
- unit: "Numbers.oddValues"
  cases:
    - expression: "Numbers.oddValues(new int[]{1, 2, 3, 4, 5, 6, 7, 8})"
      return: [1, 3, 5, 7]

This does not work, and should be a shorthand for

namespace: "Numbers"
units:
- unit: "Numbers.oddValues"
  cases:
    - expression: !programming_language
        java: "Numbers.oddValues(new int[]{1, 2, 3, 4, 5, 6, 7, 8})"
      return: [1, 3, 5, 7]

which is more verbose, but also more versatile as it allows to specifiy language-specific statement for individual expressions and statements, also for multiple programming languages. This works, but the former should work as well.

[niknetniko]

I did some investigation, and the bug was introduced in #526. In that PR, the jsonschema checking was made much more strict; one of the improvements was checking the Python syntax of the "TESTed" expression literals, instead of raising later during execution.

The problem is that the jsonschema is too strict. The relevant excerpt is:

"expressionOrStatement" : {
    "oneOf" : [
      {
        "type" : "string",
        "format" : "tested-dsl-expression",
        "description" : "A statement of expression in Python-like syntax as YAML string."
      },
      {
        "description" : "Programming-language-specific statement or expression.",
        "type" : "object",
        "minProperties" : 1,
        "propertyNames" : {
          "$ref" : "#/definitions/programmingLanguage"
        },
        "items" : {
          "type" : "string",
          "description" : "A language-specific literal, which will be used verbatim."
        }
      }
    ]
  },

When encountering a string, the string is marked as having the format tested-dsl-expression, which triggers the Python syntax checking.

This is annoying to fix I think. Currently, I see this as the "proper" solution:

Make the json schema much more complex, by introducing different possibilities based on the presence of the programming_language attribute at a level above. E.g. if we encounter this attribute on the tab level, we should add a tab variant to the jsonschema where the expressions are normal strings, but we also need this for every other level where this is supported. Together with the existing support for tabs/units this will result in a rather complex and long json schema.

So, conceptually, instead of having a jsonschema like this:

oneOf:
- tabs
- units
....

we will need:

oneOf:
- tabsWithLanguage
- tabsWithoutLanguage
- unitsWithLanguage
- unitsWithoutLanguage
....

(and this on every level that supports the attribute)