Hi, I'm COSTA (Credibility Of Simulation Tool Assistant)! I'm your companion in assessing the credibility of your simulation!
Here is the list of main chapters, which you can use to navigate
⚡ The term credibility generally describes the quality to elicit trust in the results. ⚡
I think you already know this, but to fully unleash the benefits of Modeling and Simulation (M&S) and to enable simulation-informed decision making in product development, the credibility of the M&S results has to be established and appropriately communicated between decision makers and simulation engineers. Trust in M&S can be built and increased by a comprehensible credibility assessment - and this is exactly where I can support you with! I was developed to help you in assessing the credibility of your M&S results.
So, this is how we will do it:
-
The assessment contains three Chapters and I will ask you specific questions which I would like you to answer.
-
For some questions you only need to enter the info in the text box
-
For some other questions (especially in Chapters Verification and Validation) I also need you to tick off the corresponding check box to determine the status level of credibility.
- a: No SQA procedures are documented
- b: SQA procedures from the vendor are referenced
- c: An audit is conducted with the vendor to confirm that quality procedures are conducted and documented during the software development process
- d: Benchmark verification test cases, provided by the vendor, are run on the user's computer platform. The results are compared to vendor results and documented
-
Last but not least, as same as for the status I need you to determine the target level of credibility, too.
- a: No SQA procedures are documented
- b: SQA procedures from the vendor are referenced
- c: An audit is conducted with the vendor to confirm that quality procedures are conducted and documented during the software development process
- d: Benchmark verification test cases, provided by the vendor, are run on the user's computer platform. The results are compared to vendor results and documented
-
That's it! Once we are done with all the questions, you will receive a credibility spider which you can use for documentation and communication of your M&S results.
Enough chitchat for now! Let's get started! :-)
⚡ Rule #1: "There is no credibility without context!" ⚡
Without any context information, there is very little I can do. So, let's collect them together!
You already know who I am and now I would like to know more about you. So, before we take-off, with whom do I have the pleasure to collaborate? Please tell me your name(s):
John Doe
Nice to meet you!
First things first: Please tell me what product it is we're talking about
e-machine
and/or the project name this credibility assessment is required for
W-V
Now, please describe the question of interest, i.e., the specific question, decision, or concern that is being addressed:
What is the reliability of a design element of an e-machine at external excitations?
Good! And what does this all have to do with M&S? Please let me know, what the simulation task or the simulation goal is for the specific question of interest.
What are the local accelerations at a design element of an e-machine at external excitations using M&S?
The more I know, the better I can support you! Please tell me the related simulation domain(s):
solid mechanics, fluid dynamics
Almost there! Now, I need you to specify the context of use (COU) for M&S. The COU defines the specific role and scope of the computational model used to address the question of interest. It should include a detailed statement of what will be modeled and how the outputs from the computational model will be used to answer or inform the question of interest.
Finite element analysis (FEA) will be used to determine the transfer function to convert external excitations into local accelerations at a design element. The obtained local accelerations are then used as inputs for reliability analysis.
Can you tell me the quantities of interest (QOI) which are the relevant outputs to inform a decision?
accelerations
⚡ "Did we implement the M&S right?" ⚡
The objective of verification is to ensure that the mathematical model is implemented correctly and then accurately solved. Verification is composed of two activities: code verification and calculation verification, which we will discuss now in more detail. There we have following topics. Click on the list, you will jump to the specific chapter.
The objective is to identify and to remove errors in the source code and numerical algorithms of the computational software.
The goals of code verification are to identify and remove errors in the source code and numerical algorithms of the computational software. Documented results from verification studies conducted by the software developer may be referenced to support code verification. However, the verification studies from the software developer may not encompass all aspects of the software used for the COU, and thus additional code verification specific to the COU may be required. Code verification activities include software quality assurance and numerical code verification.
Apply Software Quality Assurance (SQA) procedures and document the results to ensure that the software is functioning correctly and produces repeatable results on a specified computer resource in a specified software environment.
The objective of SQA is to ensure that the software is functioning correctly and produces repeatable results on a specified computer resource in a specified software environment. Types of computational model software include, but are not limited to, off-the-shelf (OTS), modified off-the-shelf (MOTS), and user-developed. SQA is achieved through software validation on OTS and MOTS software and software quality development assurance on MOTS and user-developed software.
- a: No SQA procedures are documented.
- b: SQA procedures from the vendor are referenced.
- c: An audit is conducted with the vendor to confirm that quality procedures are conducted and documented during the software development process.
- d: Benchmark verification test cases, provided by the vendor, are run on the user's computer platform. The results are compared to vendor results and documented.
- a: No SQA procedures are documented.
- b: SQA procedures from the vendor are referenced.
- c: An audit is conducted with the vendor to confirm that quality procedures are conducted and documented during the software development process.
- d: Benchmark verification test cases, provided by the vendor, are run on the user's computer platform. The results are compared to vendor results and documented.
Apply Numerical Code Verification (NCV) to demonstrate correct implementation and functioning of the numerical algorithms
The objective of NCV is to demonstrate correct implementation and functioning of the numerical algorithms. NCV relies on careful investigation of numerical aspects, such as spatial and temporal convergence rates, spatial convergence in the presence of discontinuities, independence to coordinate transformations, and symmetry tests related to various types of system conditions. NCV is typically conducted by comparing numerical solutions to exact benchmark solutions that are analytical or semi-analytical in nature, as might be generated using the method of manufactured solutions.
- a: NCV is not performed.
- b: The numerical solution was compared to an accurate benchmark solution from another verified code.
- c: Discretization error was quantified by comparison to an exact solution as the discretization was refined, i.e. the numerical solution asymptotically approaches the exact solution as the discretization is refined.
- d: In addition to the quantification of discretization error and the execution of a grid convergence study as described in level c, the observed order of accuracy was quantified and compared to the theoretical order of accuracy.
- a: NCV is not performed.
- b: The numerical solution was compared to an accurate benchmark solution from another verified code.
- c: Discretization error was quantified by comparison to an exact solution as the discretization was refined, i.e. the numerical solution asymptotically approaches the exact solution as the discretization is refined.
- d: In addition to the quantification of discretization error and the execution of a grid convergence study as described in level c, the observed order of accuracy was quantified and compared to the theoretical order of accuracy.
The objective is to estimate the numerical error in the quantities of interest (QOIs) due to spatial and temporal discretization of the model.
Calculation verification helps to ensure that the spatial and temporal convergence behavior of the solution of the computational model is analyzed and quantified by refining the discretization parameters and solver convergence tolerances. Additionally, it helps to ensure that practitioner errors are not corrupting the simulation results. Calculation verification involves the estimation of discretization error, numerical solver error, and identification of use error.
Determine the Discretization Error. This error is associated with solving continuous computational problems at a finite number of spatial and/or temporal grid points.
Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.
- a: No grid or time-step convergence analysis was performed to estimate the discretization error.
- b: Applicable grid or time-step convergence analyses are performed and documented and their respective convergence behaviors are observed to be stable, but the discretization error is not estimated.
- c: Applicable grid or time-step convergence analyses are performed and documented and discretization error is estimated.
- a: No grid or time-step convergence analysis was performed to estimate the discretization error.
- b: Applicable grid or time-step convergence analyses are performed and documented and their respective convergence behaviors are observed to be stable, but the discretization error is not estimated.
- c: Applicable grid or time-step convergence analyses are performed and documented and discretization error is estimated.
Determine the Numerical Solver Error. Numerical solver error refers to the errors originating from the numerical solution based on the selection of solver parameters (e.g., convergence tolerance(s)).
- a: No solver parameter sensitivity is performed.
- b: No solver parameter sensitivity is performed. Solver parameters are established based on values from a previously verified computational model. This is documented in a sufficiently traceable manner.
- c: Problem-specific sensitivity study is performed and documented on solver parameters, confirming that changes in simulation results due to changes in the solver parameters are negligible relative to the modeling & simulation accuracy goal.
- a: No solver parameter sensitivity is performed.
- b: No solver parameter sensitivity is performed. Solver parameters are established based on values from a previously verified computational model. This is documented in a sufficiently traceable manner.
- c: Problem-specific sensitivity study is performed and documented on solver parameters, confirming that changes in simulation results due to changes in the solver parameters are negligible relative to the modeling & simulation accuracy goal.
Identify, mitigate and document User Errors which can occur in the simulation results by the practitioner (e.g. typographical errors).
- a: User errors are not analyzed.
- b: Key user errors are identified, and the practitioner knows about basic mitigation concepts .
- c: Key user errors are identified, appropriate mechanisms for detection and mitigation are implemented and reviewed.
- a: User errors are not analyzed.
- b: Key user errors are identified, and the practitioner knows about basic mitigation concepts .
- c: Key user errors are identified, appropriate mechanisms for detection and mitigation are implemented and reviewed.
⚡ "Did we implement the right M&S?" ⚡
Validation is the process of assessing the degree to which the computational model is an appropriate representation of the reality of interest. Therefore, validation activities are principally concerned with demonstrating the correctness of the underlying model assumptions and the degree to which sensitivities and uncertainties of the computational model and the associated comparator(s) are understood. Validation is generally demonstrated by comparing the computational model predictions with the results from the comparator(s) (e.g., physical experiments and/or other already validated simulation results). Therefore, appropriate validation activities require attention to both the computational model and the comparator, with an appropriately rigorous evaluation of the simulation results.
No worries! I will not let you down - we'll do it together!
The two credibility factors for the computational model are model form and model inputs, which encompass four components of a computational model: governing equations, system configuration, system properties, and system conditions.
Analyze the Model form which refers to both the conceptual and mathematical formulation of the computational model.
Model form includes not only the form of the governing equations but also the form of the system configuration, system properties, and system conditions. Model form is established or selected based on assumptions that will enable the computational model to achieve the desired predictions within the context of use (COU). The assumptions that give rise to a model form may be evaluated by preliminary modeling studies to identify the important contributors to model form uncertainty. Any prior knowledge on the success or limitations of the selected model form for the problem types/physics relevant to the COU may be referenced.
- a: Influence of model form assumptions are not explored.
- b: Influence of expected key model form assumptions are explored and documented, to indicate that major effects of the regarded system for the COU with the QOIs are represented by the model equations or the model approaches.
- c: Comprehensive evaluation and documentation of the model form assumptions and their implementation, that determine the governing equations and model approaches, to prove that the model represents in a well-defined manner and with sufficient accuracy the regarded system for the COU with the QOIs.
- a: Influence of model form assumptions are not explored.
- b: Influence of expected key model form assumptions are explored and documented, to indicate that major effects of the regarded system for the COU with the QOIs are represented by the model equations or the model approaches.
- c: Comprehensive evaluation and documentation of the model form assumptions and their implementation, that determine the governing equations and model approaches, to prove that the model represents in a well-defined manner and with sufficient accuracy the regarded system for the COU with the QOIs.
Examine and document the degree to which the computational model outputs are sensitive to the model inputs & parameters.
- a: Sensitivity analysis is not performed.
- b: Sensitivity analysis on expected key parameters is performed and documented.
- c: Comprehensive sensitivity analysis is performed and documented.
- a: Sensitivity analysis is not performed.
- b: Sensitivity analysis on expected key parameters is performed and documented.
- c: Comprehensive sensitivity analysis is performed and documented.
Examine and document the degree to which known or assumed uncertainties in the model inputs are propagated to uncertainties in the simulation results.
Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.
- a: Uncertainties are not identified.
- b: Uncertainties on expected key inputs are identified and qualitatively assessed w.r.t. the simulation results.
- c: Uncertainties on a subset of expected key inputs are identified and quantified and are propagated to quantitatively assess the effect on the simulation results. The not quantified key inputs are qualitatively assessed.
- d: Uncertainties on all key inputs are identified and quantified, and are propagated to quantitatively assess the effect on the simulation results.
- a: Uncertainties are not identified.
- b: Uncertainties on expected key inputs are identified and qualitatively assessed w.r.t. the simulation results.
- c: Uncertainties on a subset of expected key inputs are identified and quantified and are propagated to quantitatively assess the effect on the simulation results. The not quantified key inputs are qualitatively assessed.
- d: Uncertainties on all key inputs are identified and quantified, and are propagated to quantitatively assess the effect on the simulation results.
Comparators provide the data against which simulation results are evaluated. Comparators can be pyhsical tests and/or other simulation results. The comparator might be designed or selected to optimize a balance of resources and relevance to the COU.
The credibility ascpect comparator addresses the test samples and the test conditions. Both test samples and test conditions are further subdivided into the following four components: quantity, range of characteristics, measurements, and measurement uncertainty. The measurements made to characterize the comparator test samples and test conditions may be used as inputs to the computational model. The measurement data also enable quantification of the uncertainty in the computational model inputs, thereby enabling quantification of the uncertainty in the computational model outputs. The measurement data may also be used to examine the equivalency of the inputs used in the computational model and comparator during the validation assessment. Each component of test samples and test conditions impacts the extent to which the comparator may support model credibility and should be considered separately.
Determine and evaluate the quantity, the range of characteristics of the test samples the rigor with which the measurement data characterize each test sample. Here, the number of samples used as well as the range of each test sample charcateristic of interest included in the comparator study is examined. Increased credibility is generally achieved with a larger number of samples and with a broader range of test sample characteristics.
Example: If the length of the test sample is a characteristic of interest, this factor addresses the range of the lengths studied.
Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.
- a: No test sample is used.
- b: A single sample is used, with a single set of characteristics.
- c: Multiple samples are used representing a range of characteristics near nominal. Some of the key characteristics of these test samples are measured. The number of samples is not enough to be statistically relevant.
- d: A statistically relevant number of samples is used that represent the nominal and the expected extreme values of the parameters. All key characteristics of the samples are measured.
- a: No test sample is used.
- b: A single sample is used, with a single set of characteristics.
- c: Multiple samples are used representing a range of characteristics near nominal. Some of the key characteristics of these test samples are measured. The number of samples is not enough to be statistically relevant.
- d: A statistically relevant number of samples is used that represent the nominal and the expected extreme values of the parameters. All key characteristics of the samples are measured.
Examine the uncertainty associated with the tools and methods used to obtain the measurements characterizing the samples.
Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.
- a: Samples are not characterized or are characterized with gross observations, and measurement uncertainty is not addressed.
- b: Uncertainty analysis incorporates instrument accuracy only.
- c: Uncertainty analysis incorporates instrument accuracy and repeatability (i.e., statistical treatment of repeated measurements).
- d: Uncertainty analysis incorporates a comprehensive uncertainty quantification, which includes instrument accuracy, repeatability, and other conditions affecting the measurements.
- a: Samples are not characterized or are characterized with gross observations, and measurement uncertainty is not addressed.
- b: Uncertainty analysis incorporates instrument accuracy only.
- c: Uncertainty analysis incorporates instrument accuracy and repeatability (i.e., statistical treatment of repeated measurements).
- d: Uncertainty analysis incorporates a comprehensive uncertainty quantification, which includes instrument accuracy, repeatability, and other conditions affecting the measurements.
For a given test method, determine and evaluate the number and the range of test conditions conditions included in the comparator study as well as the rigor with which the measurement data characterize the test conditions. Increased credibility is generally achieved with a larger number and a broader range of test conditions.
Example: The test method could specify measuring the test sample strength at multiple strain rates under external loads at multiple temperatures. If the test condition is temperature, this factor addresses the range of temperatures studied.
- a: No test condition is used.
- b: A single condition is used, with a single set of characteristics.
- c: Multiple conditions are used representing a range of characteristics near nominal. Some of the key characteristics of these test conditions are measured. The number of conditions is not enough to be statistically relevant.
- d: A statistically relevant number of conditions is used that represent the nominal and the expected extreme values of the parameters. All key characteristics of the conditions are measured.
- a: No test condition is used.
- b: A single condition is used, with a single set of characteristics.
- c: Multiple conditions are used representing a range of characteristics near nominal. Some of the key characteristics of these test conditions are measured. The number of conditions is not enough to be statistically relevant.
- d: A statistically relevant number of conditions is used that represent the nominal and the expected extreme values of the parameters. All key characteristics of the conditions are measured.
Examine the uncertainty associated with the tools and methods used to obtain the measurements characterizing the test conditions.
Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.
- a: Test conditions are not characterized or are characterized with gross observations, and measurement uncertainty is not addressed.
- b: Uncertainty analysis incorporates instrument accuracy only.
- c: Uncertainty analysis incorporates instrument accuracy and repeatability (i.e., statistical treatment of repeated measurements).
- d: Uncertainty analysis incorporates a comprehensive uncertainty quantification, which includes instrument accuracy, repeatability, and other conditions affecting the measurements.
- a: Test conditions are not characterized or are characterized with gross observations, and measurement uncertainty is not addressed.
- b: Uncertainty analysis incorporates instrument accuracy only.
- c: Uncertainty analysis incorporates instrument accuracy and repeatability (i.e., statistical treatment of repeated measurements).
- d: Uncertainty analysis incorporates a comprehensive uncertainty quantification, which includes instrument accuracy, repeatability, and other conditions affecting the measurements.
Applicability is the relevance of the validation activities to support the use of the computational model for a context of use (COU). The applicability of the validation activities is governed by two factors: the relevance of the quantities of interest (QOIs) used in the validation activities to the QOIs of the COU, and the relevance of the validation conditions relative to those of the COU.
Note, that the measured QOIs of the validation activities are not always identical to the QOIs for the COU because the QOIs for the COU are not always directly measurable, might not be measured without unduly perturbing the intended test conditions, and/or might not be obtained within acceptable ranges of uncertainty and error. Therefore, the measured QOIs of the validation activities may be surrogates for the QOIs for the COU, with varying degrees of applicability.
The aspects associated with establishing applicability of the computational model to the COU are the relevance of the validation activities to the COU and the relevance of the validation QOIs.
Evaluate the QOIs (Quantity of Interests) for validation relevance, completeness, comparability (simulation results and comperator) and document that appropriately. Compare the QOIs from the validation activities to the QOIs for the COU.
- a: From validation activities, no QOIs are related to those for the COU.
- b: The QOIs from the validation activities are related, though not identical, to those for the COU.
- c: A subset of the QOIs from the validation activities are sufficiently relevant, complete, comparable to those for the COU.
- d: All QOIs from the validation activities are sufficiently relevant, complete, comparable to those for the COU.
- a: From validation activities, no QOIs are related to those for the COU.
- b: The QOIs from the validation activities are related, though not identical, to those for the COU.
- c: A subset of the QOIs from the validation activities are sufficiently relevant, complete, comparable to those for the COU.
- d: All QOIs from the validation activities are sufficiently relevant, complete, comparable to those for the COU.
Evaluate relative proximity of the context of use (COU) to the validation ranges and document that appropriately.
- a: There is no overlap between the validation ranges and the COU.
- b: There is partial overlap between the validation ranges and the COU.
- c: The COU encompass all validation ranges, and the validation ranges span the entire COU space.
- a: There is no overlap between the validation ranges and the COU.
- b: There is partial overlap between the validation ranges and the COU.
- c: The COU encompass all validation ranges, and the validation ranges span the entire COU space.
An assessment of the accuracy of the simulation output can be performed after the outputs from the V&V activities are obtained and compared.
We are almost done! There is just one final step missing: The output comparison between simulation and comparator.
Compare outputs of simulation and comparator.Determine how well the simulation results and the comparator match and document that appropriately.
- a: No assessment.
- b: Visual or qualitative comparison concludes sufficient agreement. What sufficient means is to be defined contextually by experts.
- c: Differences between computational results and comparator data (e.g. experimental) are typically less than 5%. Includes consideration of relevant uncertainty, but statistical distributions for uncertainty quantification are unknown.
- d: Differences between computational results and comparator data (e.g. experimental) are typically less than 5%. Statistical distributions for uncertainty quantifications are known.
- a: No assessment.
- b: Visual or qualitative comparison concludes sufficient agreement. What sufficient means is to be defined contextually by experts.
- c: Differences between computational results and comparator data (e.g. experimental) are typically less than 5%. Includes consideration of relevant uncertainty, but statistical distributions for uncertainty quantification are unknown.
- d: Differences between computational results and comparator data (e.g. experimental) are typically less than 5%. Statistical distributions for uncertainty quantifications are known.
You have just crossed the finish line! Now, I will create the credibility spider for you...