Automate Manual System Tests Hackathon

[JackEAllen]

As a developer, I would like a Hackathon to take place to begin automating manual system tests which we perform as part of every release. 👨‍💻

Moving towards more automation as part of a hackathon would greatly reduce the time required to test IBEX as part of a release and therefore reduce the time and effort required to create and deploy a release.

Automating manual system tests will also reduce the potential for human error during testing for a release.

In order to automate as many of the manual system tests as possible, Issues will need to be created for each system test in advance of a hackathon. This will allow a developer to just pick up an issue and get on with automating system tests without needing to plan how exactly to do so first.

Could I ask for help in writing up issues for each system test in preparation for proposing a hackathon and then post a link to them below?

Once an issue has been written up for each system test that can be automated, I will convert this discussion into an issue and propose for the next hackathon.

[JackEAllen]

System Test Issue Checklist

As issues are created for each system test, please update this checklist and link the new issue

1. The server starts in an appropriate way
   • start_inst runs:
   • ISISDAE_01 runs (open console and type ENTER, verify its log is updated.):
   • BLOCKSVR runs (as above):
2. GeniePython Autocomplete
   • In both scripting window and genie_python console type 'load_script("' then press tab:
3. The GUI start in an appropriate way
   • Launch GUI:
4. Dashboard Perspective
   • Open Dashboard:
5. Blocks Perspective
   • Load Blocks:
6. Buttons perspective is working
   • Can switch between perspectives:
7. Alarms Perspective
   • Opens when the button is selected:
   • Open alarms tree:
8. Beam Status Perspective
   • Switch to beam status perspective:
   • Check beam information panel:
   • Check MCR news panel:
   • Check Beam current panel:
9. DAE Perspective
   • Opens when the button is selected:
   • Start a run:
   • End a run:
   • pause run (start first):
   • Resume run (pause first)
   • Abort run
   • Cancel the abort
   • Switch between tabs (and sub-tabs) when the instrument is running
   • Can switch between tabs (and sub-tabs) when the instrument is not running
   • Check Experiment Setup sub-tabs while instrument is running
   • Check Experiment Setup sub-tabs while instrument is NOT running
   • Run Summary Tab: specify a filter value in the drop-down menu
   • Experiment Setup, Time Channels: select a new time channel file without clicking 'Apply changes'
   • Experiment Setup, Time Channels: select a new time channel file
   • Experiment Setup, Data Acquisition: select a new wiring/detector/spectra table file without clicking 'Apply changes'
   • Experiment Setup, Data Acquisition: select a new wiring/detector/spectra table file
   • Experiment Setup, Periods: select select a new period file without clicking 'Apply changes'
   • Experiment Setup, Periods: select a new period file.
   • Experiment Setup, Periods: content of "Period Settings" group changes appropriately depending on "Period Type" Selection
   • Experiment Setup, Periods: (only applicable for hardware period types) window shows appropriate controls depending on selected period source
   • When you have no TCB files in your settings then the check the TCB file dropdown
   • 6-22 for wiring tables (Error message points to the correct location to place files)
   • 6-22 for detector tables (Error message points to the correct location to place files)
   • 6-22 for spectra tables (Error message points to the correct location to place files)
   • 6-22 for period files (Error message points to the correct location to place files)
   • Create a block called [PVPREFIX]:DAE:TITLE:DISPLAY. Check its value when the "Show DAE title on web dashboard" box is ticked/unticked in the DAE run summary
10. Experiment Details Perspective
    • Switch to Experiment Details Perspective
    • Set a RB number of an actual Experimental Team
    • Set a fake RB number
    • Search for an RB number by user name
    • Alter the Team information
    • check parameters table (All entries in the Parameter tables are relevant and have name and units)
    • Alter the name information to include utf8 characters in the surname string "₣·₤·₥·₦·₧·₨·₩·₪·₫·₭·₮·₯·₹·ø" (Name is utf8 string in both Users (in banner) and Table for both client A and B)
    • Alter the name information to include utf8 character in the forenames string "₣·₤·₥·₦·₧·₨·₩·₪·₫·₭·₮·₯·₹·ø surname" (Name is utf8 string in Table for both client A and B)
    • Alter the institute to include the utf8 characters "₣·₤·₥·₦·₧·₨·₩·₪·₫·₭·₮·₯·₹·ø" (Institute is utf8 string in Table for both client A and B)
11. IOC log perspective
    • Select perspective
    • Show info messages
    • Limit the messages to content value
    • Limit the messages to a date range
    • Limit the messages to a sender
    • Limit the messages to a sender host (the sender host column can be enabled by right clicking on the columns of the ioc log table)
    • Limit the messages to a severity
    • Limit the messages to a Type
    • Limit the messages to an Application ID (the Application ID column can be enabled by right clicking on the columns of the ioc log table)
    • Clear the search criteria
    • Use the message simulator (found in the developers wiki repsoitory in a folder next to the system testing spreadsheet) to create messages of different types.
12. Log plotter perspective
    • Select no blocks
13. Motors Perspective
    • Select perspective
    • Switch between the tabs
    • Check motors
    • Double click a motor
    • Open a motor OPI from the motors perspective. Switch instrument
14. Scripting Perspective
    • Select perspective
    • Wait for console to load
    • Read a block using cget in the scripting console
    • Set a block in the scripting console
    • Read a block using cshow
    • Read a disconnected block using cget
    • Read a disconnected block with cshow
    • start a run from the scripting console
    • end a run from the scripting console
    • pause a run from the scripting console (start first)
    • resume a run from the scripting console (pause first)
    • Select perspective
    • Wait for console to load
    • Read a block using cget in the scripting console
    • Set a block in the scripting console
    • Read a block using cshow
    • Read a disconnected block using cget
    • Read a disconnected block with cshow
    • start a run from the scripting console
    • end a run from the scripting console
    • pause a run from the scripting console (start first)
    • resume a run from the scripting console (pause first)
15. Synoptic Perspective
    • Select perspective
    • Select a synoptic
    • Browse the synoptic navigation
    • Open an OPI (from a synoptic element)
16. Web Links Perspective
    • Switch to web links perspective
17. Switch Instruments
    • Switch instruments by typing a known name
    • Switch instruments by typing an unknown name and giving it the correct prefix
18. Configs
    • Edit the current config (add an IOC), and save it
    • Edit the current config by adding an IOC and cancel the dialog
    • Edit the current config (add a block), and save it
    • Edit the current config by adding a block and cancel the dialog
    • Edit the current config (add a group), and save it
    • Edit the current config by adding a group and cancel the dialog
    • Edit the current config (select a component), and save it
    • Edit the current config by adding a component and cancel the dialog
    • Edit the current config (set IOC macro), and save it.
    • Edit the current config by setting an IOC macro and cancel
    • Create a new config. Click 'Save As'. Enter the name of the current configuration.
    • Create a new config. Click save. Enter the name of another configuration.
    • Edit a config (not current) by setting some IOC macros. Load the config.
    • Edit the current config (IOC PV values), and save it.
    • Edit the current config (IOC PV values) and cancel
    • Edit the current config (summary: description and synoptic), and save it
    • Edit the current config and cancel (summary: description and synoptic)
    • Edit an existing config, and save it
    • Create a new config, and save it
    • Delete a config
    • Delete multiple configs simultaneously
    • Load a config
    • Create a new config and save as component
    • Create a new component
    • Edit a component
    • Delete a component
    • Add a block, verifying that the PV search works
    • Edit a block
    • Delete a block
    • A block can be created with run control
    • Check new block log settings
    • A block can be created with logging
    • Run control can be turned off for a block in the config
    • Logging can be turned off for a block
    • Run control can be reinstated for a block via the config editor
    • Logging can be reinstated for a block via the config editor
    • Add a group
    • Add blocks to the group
    • Remove blocks from the group
    • Remove a group
    • Add components to the config
    • Remove components from the config
    • Add some values to the IOC macros
    • Add a default synoptic to a config
    • Edit the default synoptic of a config
    • Remove the default synoptic of a config
    • Create a block with a pv which has a field (VAL, HLM on a motor)
19. Configs within the system
    • Set 1 IOCs to Auto Start, 1 IOC Auto restart, 1 IOC to both, 1 IOC to neither in a config. Load that config.
    • Edit a component of the current config, get an idea of length of time for config to reload
    • Reload the config whilst an IOC not in the config is running
20. Configs
    • Create a new configuration. Click save as. Try adding spaces (start, middle and end of name) and special characters.
    • 1. Create a configuration. Add an IOC. Set it to auto-start and put it in recsim mode. Save the config. 2. Edit the config, change the sim level to None. Save the config.
    • Create a config with an 2 IOCs in. Load it; Remove an ioc and add a different IOC
    • Add an IOC with custom settings (e.g. recsim) to a component; Add the component to a config; Load the config and run the IOC
    • Create a configuration. Set a default synoptic and save
    • Create a synoptic. Create a new configuration with that synoptic as default. Save. Delete the synoptic
    • Change the description in the current config, save.
    • Change blocks in the current config, save.
21. Synoptics
    • Create a synoptic
    • Edit a synoptic
    • Delete a synoptic
    • Add a component to a synoptic
    • Delete a component from a synoptic
    • Copy a component on a synoptic
    • Delete multiple components from a synoptic
    • Copy multiple components on a synoptic
    • Add a component and select a target (e.g. Analyser)
    • Using the Analyser*, alter the component target details to be something else(*or whichever you picked in the previous test)
    • Confirm you can set a custom icon on the Analyser*, which persists when you change the target. Confirm this persists after saving and reopening the dialog.
    • On the Analyser*, confirm the "Clear" button works (i.e. resets to an undefined component) and that subsequently selecting a target sets the correct default icon.
    • Attempt to create a synoptic with a name of greater than 30 characters
    • Add a new PV to a component
    • Attempt to add multiple components with the same name to a synoptic
22. Synoptics within the system
    • Create a synoptic that shows the beam, load it
    • Create a synoptic that doesn't show the beam, load it
    • Once synoptic is saved it is available immediately in the drop down in the synoptic perspective
    • A synoptic can be edited after saving using "save as"
    • Add a component to a synoptic. Change its type at least twice.
23. Synoptics
    • Open the Synoptic perspective. Check synoptic selected in drop-down
    • With the synoptic perspective open, switch configurations
    • In the synoptic perspective,push the load button
24. IOC Start/Stop
    • Start an IOC from the menu
    • Using the console, stop an IOC set to auto-restart in the config.
    • Stop an IOC
    • Press the "Restart" button in the start/stop IOC dialog both for IOC with and without "auto-restart" enabled.
    • Add the Simple IOC to a test configuration with Auto-start and auto-restart ticked. Boot the Simple IOC. From an epics terminal issue the command caput %MYPVPREFIX%SIMPLE:CRASHVALUE 1 to crash the IOC
25. Run Control
    • View the run control settings
    • Edit the run control settings
    • Restore the configuration values for run control
26. OPIs and IOCs
    • With an IOC started, open OPI, get an idea of length of time before blocks are connected
    • With an OPI open , Start IOC, get an idea of length of time before pvs are connected
    • Start IOC, open OPI, Restart IOC, get an idea of length of time
    • Start IOC, open OPI, Stop IOC, Wait 2 minutes, Restart IOC, get an idea of length of time before pvs reconnect
    • Open OPI, Wait 2 minutes, Start IOC, get an idea of length of time
27. Blocks
    • Create a block
    • Disconnect a connected block
    • Reconnect a disconnected block (check various disconnection times up to 5mins)
    • Start up SIMPLE IOC and put a block on VALUE1. Write a 4 into the PV to cause a MINOR-alarm state
    • Take a block under run control and remove run control.
    • Write a 1 to the VALUE1 PV to cause a MAJOR alarm.
    • Block pointing at ai: 0 is displayed as 0.000 (PREC 3) and 0.00000 (PREC 5)
    • Put block into an INVALID-alarm state (.SEVR field)
    • Block pointing at ai: 0.1 is displayed as 0.100 (PREC 3) and 0.10000 (PREC 5)
    • Block pointing at ai: 0.01 is displayed as 0.010 (PREC 3) and 0.01000 (PREC 5)
    • Block pointing at ai: 0.001 is displayed as 0.001 (PREC 3) and 0.00100 (PREC 5)
    • Block pointing at ai: -0.001 is displayed as -0.001 (PREC 3) and -0.00100 (PREC 5)
    • Block pointing at ai: 0.001234 is displayed as 0.001 (PREC 3) and 0.00123 (PREC 5)
    • Block pointing at ai: -0.001234 is displayed as -0.001 (PREC 3) and -0.00123 (PREC 5)
    • Block pointing at ai: 0.0009 is displayed as 9.000E-4 (PREC 3) and 9.00000E-4 (PREC 5)
    • Block pointing at ai: -0.0009 is displayed as -9.000E-4 (PREC 3) and -9.00000E-4 (PREC 5)
    • Block pointing at ai: 15 is displayed as 15.000 (PREC 3) and 15.00000 (PREC 5)
    • Block pointing at ai: 999999 is displayed as 999999.000 (PREC 3) and 999999.00000 (PREC 5)
    • Block pointing at ai: -999999 is displayed as -999999.000 (PREC 3) and -999999.00000 (PREC 5)
    • Block pointing at ai: 1000000 is displayed as 1.000E6 (PREC 3) and 1.00000E6 (PREC 5)
    • Block pointing at ai: -1000000 is displayed as -1.000E6 (PREC 3) and -1.00000E6 (PREC 5)
    • Block pointing at ai: 1234567 is displayed as 1.235E6 (PREC 3) and 1.23457E7 (PREC 5)
    • Block pointing at ai: -1234567 is displayed as -1.235E6 (PREC 3) and -1.23457E7 (PREC 5)
    • Block pointing at longin: never displayed as scientific notation using the following script:
      from time import sleep for val in [0, 15, 999999, 1000000, 1234567]: g.cset(b.LONG_VAL, val) sleep(0.5) g.cset(b.LONG_VAL, -val) sleep(0.5)
    • Create a block. Right click on it in the blocks dashboard and select Edit host configuration. Change the block. Click cancel.
    • Create a block. Right click on it in the blocks dashboard and select Edit host configuration. Change the block. Click ok.
    • Create a hidden block. Right click on any group and confirm that you can switch between showing and hiding the block.
    • Create a hidden block. Right click on the block and confirm that you can switch between showing and hiding the block.
28. Groups
    • Create a group containing ONLY hidden blocks.
29. Instrument List
    • Open instrument list (IBEX > Switch instrument)
    • When on an instrument machine for an instrument in CS:INSTLIST, open the instrument list.
    • Can switch from local host to other instrument
    • Can switch between two listed instruments
    • Can switch to a custom instrument not on the list
30. genie_python
    • While the DAE is in a state it can begin a run: Clone the developers manual wiki and load the script "genie_python_dae.py" found and in the testing folder from the scripting perspective using g.load_script(<script_name>). Run script
    • test 25-1 in a genie_python console
    • Create a config with SIMPLE IOC and blocks TEST_BLOCK_NUM (SIMPLE:VALUE:P5) and TEST_BLOCK_STR (SIMPLE:HELLO). Run the SIMPLE IOC. Run script "genie_python_blocks.py" from the testing area of the developer manual.
    • test 25-3 in a genie_python console
31. Device screen
    • Add a device screen and select 'Remove this device when IBEX is closed'. Restart the client
    • Attempt to add/edit/delete Device Screens
    • Add a device screen and select 'save this device screen'. Restart the client
    • Confirm that device screens can not be saved if one or more items is not pointing at a target OPI.
32. Menu Items
    • Verify that the manager mode dialog can be opened first time. From GUI go to IBEX menu -> Manager mode.
33. JournalViewer
    • Check the JournalViewer can establish a connection to the database
34. Script Generator
    • Add some actions and then switch between script definitions
    • Add an action then duplicate it. Edit the contents of the first action to be valid and the contents of the second action to be invalid with different values. After checking the expected outcome, delete one of the actions.
    • Add an action and make it valid, then duplicate it. The validity column should have a tick and be green. Click the generate script button.
    • Add a couple of actions and make them invalid.
    • Add 3 actions, make one of them valid and duplicate it. Edit the contents of the duplicated action and one of the invalid actions. Move the actions up and down in the table.
35. Disabled GUI tests
    • In the GUI repository, look for any disabled tests in https://github.com/ISISComputingGroup/ibex_gui/blob/master/base/uk.ac.stfc.isis.ibex.client.tycho.parent/pom.xml and run them manually from eclipse by right clicking on the test project and saying run as -> junit test