Add more Metrics & Specs (#2)

- added fake code coverage - added example custom metrics - added fake specs
testspace-com · Aug 6, 2021 · 39ee9fe · chouhan50 · Jan 18, 2022 · 39ee9fe
1 parent 1e9e45b
commit 39ee9fe
Show file tree

Hide file tree

Showing 16 changed files with 220 additions and 10 deletions.
diff --git a/.github/workflows/auto.yml b/.github/workflows/auto.yml
@@ -0,0 +1,14 @@
+name: Automate
+on:
+  repository_dispatch:
+    types: [auto]
+jobs:
+  setup:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - name: Automation here
+        run: |
+          echo "Call a script to do something .."
+          echo "Name     = " ${{ toJson(github.event.client_payload.name) }}
+          echo "Password = " ${{ toJson(github.event.client_payload.password) }}
diff --git a/.github/workflows/setup.yml b/.github/workflows/setup.yml
@@ -13,8 +13,6 @@ jobs:
         run: |
           curl -fsSL https://testspace-client.s3.amazonaws.com/testspace-linux.tgz | sudo tar -zxvf- -C /usr/local/bin
           testspace config url s2.${{ github.event.inputs.domain }}.com
-        env:
-          DOMAIN: ${{ secrets.TESTSPACE_DOMAIN }}
       - name: Setup failure tracking
         run: |
           ./push.setup.sh
diff --git a/.gitignore b/.gitignore
@@ -1 +1 @@
-.SAVE
+_SAVE
diff --git a/README.md b/README.md
@@ -3,6 +3,9 @@ This demo Testspace project is available at [testspace-com:demo](https://s2.test
 
 > https://s2.testspace.com/projects/testspace-com:demo
 
+<br>
+
+[![Space Metric](https://s2.stridespace.com/spaces/20907/badge?token=ce9d082b5169ca25479d33d3f8aa033d90fe6e45)](https://s2.stridespace.com/spaces/20907?utm_campaign=badge&utm_medium=referral&utm_source=test "Test Cases") [![Space Metric](https://s2.stridespace.com/spaces/20907/metrics/42956/badge?token=4bb36aff66be5954de502b76895309d72b83662f)](https://s2.stridespace.com/spaces/20907/current/Code%20Coverage?utm_campaign=badge&utm_medium=referral&utm_source=coverage "Code Coverage (lines)") [![Space Metric](https://s2.stridespace.com/spaces/20907/metrics/42957/badge?token=dbb8ccc83e9336439d0478074ea1e1c285d3fb28)](https://s2.stridespace.com/spaces/20907/current/Issues?utm_campaign=badge&utm_medium=referral&utm_source=issues "Issues")
 
 ## Fake Results
 The fake results contained in the `testspace-com:demo` project are for *demonstration purposes* only. There is a set of `pre-canned result` XML files under the *fake.results* used for pushing content to Testspace. 
@@ -15,7 +18,7 @@ The *fake results* highlight the following:
 - `Grouping Similar Failure Types`
 - and more
 
-For general information on Testspace, refer to the [help documentation](https://help.testspace.com).
+For general information on Testspace publishing, refer to the [help documentation](https://help.testspace.com/publish/overview).
 
 ### Scripts 
 
@@ -29,3 +32,27 @@ Use the two GitHub Workflows to manual execute the corresponding shell scripts v
 - `setup.yml`
 
 The GitHub workflows use the `inputs` to set the domain. The default is `testspace`. 
+
+## Specs
+The repo also contains a set of *fake* test specs for *demonstration purposes only*. The *fake specs* highlight the following:
+
+- `Specs` ca be co-located with automated tests 
+- Manual results are `carried` from previous session(s) when new test automated results are published
+
+
+For general information on Testspace Manual Test Framework, refer to the [help documentation](https://help.testspace.com/manual/overview).
+
+### Desktop Preview
+To preview the `specs` on console:
+
+Install:
+```
+cd specs
+bundle install
+```
+
+Run:
+```
+cd specs
+bundle exec jekyll serve
+```
diff --git a/fake.results/CodeCoverage.xml b/fake.results/CodeCoverage.xml
@@ -0,0 +1,98 @@
+<?xml version="1.0" ?>
+<!DOCTYPE coverage
+  SYSTEM 'http://cobertura.sourceforge.net/xml/coverage-03.dtd'>
+<coverage line-rate="0.733">
+  <packages>
+    <package name="Controller">
+      <classes>
+        <class filename="controlStart.cpp">
+          <methods/>
+          <lines>
+            <line branch="false" hits="1" number="1"/>
+            <line branch="false" hits="1" number="2"/>
+            <line branch="false" hits="1" number="3"/>
+            <line branch="false" hits="0" number="4"/>
+          </lines>
+        </class>
+        <class filename="controllerExec.cpp">
+          <methods/>
+          <lines>
+            <line branch="false" hits="1" number="1"/>
+            <line branch="false" hits="1" number="2"/>
+            <line branch="false" hits="0" number="3"/>
+            <line branch="false" hits="0" number="4"/>
+          </lines>
+        </class>
+          <class filename="controllerShutdown.cpp">
+          <methods/>
+          <lines>
+            <line branch="false" hits="1" number="1"/>
+            <line branch="false" hits="1" number="2"/>
+            <line branch="false" hits="0" number="3"/>
+            <line branch="false" hits="0" number="4"/>
+            <line branch="false" hits="1" number="5"/>
+            <line branch="false" hits="1" number="6"/>
+          </lines>
+        </class>
+      </classes>
+    </package>
+    <package name="Model">
+      <classes>
+        <class filename="modelInit.cpp">
+          <methods/>
+          <lines>
+            <line branch="false" hits="1" number="1"/>
+            <line branch="false" hits="1" number="2"/>
+            <line branch="false" hits="0" number="3"/>
+            <line branch="false" hits="0" number="4"/>
+          </lines>
+        </class>
+        <class filename="modelStart.cpp">
+          <methods/>
+          <lines>
+            <line branch="false" hits="1" number="1"/>
+            <line branch="false" hits="0" number="2"/>
+            <line branch="false" hits="0" number="3"/>
+            <line branch="false" hits="0" number="4"/>
+          </lines>
+        </class>
+      </classes>
+    </package>
+    <package name="Lib">
+      <classes>
+        <class filename="LibInit.cpp">
+          <methods/>
+          <lines>
+            <line branch="false" hits="1" number="1"/>
+            <line branch="false" hits="1" number="2"/>
+            <line branch="false" hits="0" number="3"/>
+            <line branch="false" hits="0" number="4"/>
+            <line branch="false" hits="0" number="5"/>
+            <line branch="false" hits="0" number="6"/>
+          </lines>
+        </class>
+        <class filename="LibSweep.cpp">
+          <methods/>
+          <lines>
+            <line branch="false" hits="1" number="1"/>
+            <line branch="false" hits="0" number="2"/>
+            <line branch="false" hits="1" number="3"/>
+            <line branch="false" hits="1" number="4"/>
+          </lines>
+        </class>
+        <class filename="LibCache.cpp">
+          <methods/>
+          <lines>
+            <line branch="false" hits="1" number="1"/>
+            <line branch="false" hits="0" number="2"/>
+            <line branch="false" hits="0" number="3"/>
+            <line branch="false" hits="0" number="4"/>
+            <line branch="false" hits="1" number="5"/>
+            <line branch="false" hits="1" number="6"/>
+            <line branch="false" hits="0" number="7"/>
+          </lines>
+        </class>
+      </classes>
+    </package>
+  </packages>
+</coverage>
diff --git a/fake.results/Timing.csv b/fake.results/Timing.csv
@@ -0,0 +1 @@
+Timing Metrics, 134, 65, 165
diff --git a/fake.results/Timing.log b/fake.results/Timing.log
@@ -0,0 +1,5 @@
+File Timing.log: contains some simple timing metrics.
+
+The delay before start was 134 ms 
+The latency from start to actual start was 65 ms
+The duration from actual start to stop was 165 ms
diff --git a/fake.results/result.pass.txt b/fake.results/result.pass.txt
@@ -1,3 +1,5 @@
 "[API Tests]$RESULTS_PATH/API*.pass.xml"
 "[E2E Tests]$RESULTS_PATH/E2E*.pass.xml"
 "[UI Tests]$RESULTS_PATH/UI*.pass.xml"
+"$RESULTS_PATH/CodeCoverage.xml"
+"$RESULTS_PATH/Timing.log{$RESULTS_PATH/Timing.csv: Timing throughput}"
diff --git a/fake.results/result.set1.txt b/fake.results/result.set1.txt
@@ -4,4 +4,6 @@
 "[E2E Tests]$RESULTS_PATH/E2ESecurity.fail.xml"
 "[UI Tests]$RESULTS_PATH/UIBack.pass.xml"
 "[UI Tests]$RESULTS_PATH/UIFront.fail.xml"
-"[UI Tests]$RESULTS_PATH/UISide.fail.xml"
+"[UI Tests]$RESULTS_PATH/UISide.fail.xml"
+"$RESULTS_PATH/CodeCoverage.xml"
+"$RESULTS_PATH/Timing.log{$RESULTS_PATH/Timing.csv: Timing throughput}"
diff --git a/fake.results/result.set2.txt b/fake.results/result.set2.txt
@@ -4,4 +4,6 @@
 "[E2E Tests]$RESULTS_PATH/E2ESecurity.fail.xml"
 "[UI Tests]$RESULTS_PATH/UIBack.pass.xml"
 "[UI Tests]$RESULTS_PATH/UIFront.pass.xml"
-"[UI Tests]$RESULTS_PATH/UISide.fail.xml"
+"[UI Tests]$RESULTS_PATH/UISide.fail.xml"
+"$RESULTS_PATH/CodeCoverage.xml"
+"$RESULTS_PATH/Timing.log{$RESULTS_PATH/Timing.csv: Timing throughput}"
diff --git a/fake.results/result.set3.txt b/fake.results/result.set3.txt
@@ -4,4 +4,6 @@
 "[E2E Tests]$RESULTS_PATH/E2ESecurity.fail.xml"
 "[UI Tests]$RESULTS_PATH/UIBack.pass.xml"
 "[UI Tests]$RESULTS_PATH/UIFront.fail.xml"
-"[UI Tests]$RESULTS_PATH/UISide.fail.xml"
+"[UI Tests]$RESULTS_PATH/UISide.fail.xml"
+"$RESULTS_PATH/CodeCoverage.xml"
+"$RESULTS_PATH/Timing.log{$RESULTS_PATH/Timing.csv: Timing throughput}"
diff --git a/fake.results/result.set4.txt b/fake.results/result.set4.txt
@@ -4,4 +4,6 @@
 "[E2E Tests]$RESULTS_PATH/E2ESecurity.fail.xml"
 "[UI Tests]$RESULTS_PATH/UIBack.fail.xml"
 "[UI Tests]$RESULTS_PATH/UIFront.fail.xml"
-"[UI Tests]$RESULTS_PATH/UISide.fail.xml"
+"[UI Tests]$RESULTS_PATH/UISide.fail.xml"
+"$RESULTS_PATH/CodeCoverage.xml"
+"$RESULTS_PATH/Timing.log{$RESULTS_PATH/Timing.csv: Timing throughput}"
diff --git a/specs/Manual Tests/Sign in.md b/specs/Manual Tests/Sign in.md
@@ -1,6 +1,12 @@
 ---
 testspace:
-title: Sign in
+title: Sign-in
+before:
+  name: github::auto
+  description: SETUP FIXTURE
+  input:
+    name: Sam
+    password: 1234
 description: Checks the sequence of signing into the system. 
 ---
 
@@ -9,6 +15,7 @@ description: Checks the sequence of signing into the system.
 # {{ spec.title }}
 {% include example-intro.md %}
 
+> USES A AUTOMATED FIXTURE 🎁
 
 ## Credentials
 Confirm credentials are valid.

diff --git a/specs/Manual Tests/Sign up.md b/specs/Manual Tests/Sign up.md
@@ -1,6 +1,6 @@
 ---
 testspace:
-title: Sign up
+title: Sign-up
 description: Checks the sequence of signing up for using the system. 
 ---
 

diff --git a/specs/Manual Tests/Turn on power.md b/specs/Manual Tests/Turn on power.md
@@ -0,0 +1,50 @@
+---
+testspace:
+title: Turn on power
+---
+
+{% if page %} {% assign spec = page %} {% endif %}
+
+# {{ spec.title }}
+This tests different startup stages of the application when turning on the power. It will eventually be transitioning to an `idle` state. 
+
+## Check Transitions
+The display will be changed based on the following configurations: 
+
+Mode 1.a | Mode 2.c
+-------- | --------
+ State 1 | State n
+
+- Check for the first transition after power-up
+- Check next step for **id=17.xx**. Should display the local URL. 
+
+Now review the states based on following diagram:
+
+![states](./states.png "State machine")
+
+## Power Scenario
+The power scenario requires quick cycling, going over each scene in less than `10` seconds.
+
+- Going online **without** system configuration set to `ABC`
+- Verify system turns on and display *blinking every 3 seconds*
+
+## Switch On
+The switch-on will be very rapid based on the `Mode 2.c`. 
+
+- Go back on line to check for turn-off switch
+- Check UI mode display
+
+
+## Turn off delay
+The turn off delay verification is used for input configuration.
+
+- Go back on line to check for switch-on
+- Check UI mode display
+
+## Full Idle Mode
+The idle can switch automatically based on previous configuration.
+
+
+- Go back on line to check for switch-off
+- Check UI mode display
+
diff --git a/specs/Manual Tests/states.png b/specs/Manual Tests/states.png