Progress

docs/conclusion.md

@@ -2,10 +2,11 @@
 
 ## RQ1
 
-We conclude that the survey currently used
-is not derived from evidence-based best practices
-and instead is established in more informal ways.
-Not of the current survey, nor of the surveys it is based on,
-has it been published how the survey questions were developed,
-nor by which criteria the best questions were selected.
+From our literature search,
+we conclude that the survey currently used
+is not derived from evidence-based best practices.
+Neither of the current survey, nor of the surveys it is based on,
+has it been published (1) how its questions were developed,
+(2) by which criteria the best questions were selected.
+
 

docs/discussion.md

@@ -12,3 +12,28 @@ a formal academic paper is written.
 To us, it seems more likely that only a shallow research search has been
 performed, hence that there was little awareness of the evidence-based
 literature at the time of writing.
+
+
+## Notes
+
+
+
+## Epilogue
+
+We know that teachers reflecting on their work is one of the
+best ways to increase his/her teaching quality.
+Or: 'student ratings can only become a tool for enhancement when they
+feed reflective conversations about improving the learning process and when
+these conversations are informed by the scholarship of teaching and
+learning `[Roxå et al., 2021]`.
+The other best way for teachers to improve is to do peer observations.
+Note that neither practice needs an evaluations.
+
+If we really care about teaching quality, shouldn't we encourage
+doing the things that actually work?
+
+## References
+
+- `[Roxå et al., 2021]` Roxå, Torgny, et al. 
+  "Reconceptualizing student ratings of teaching to support quality discourse
+  on student learning: a systems perspective." Higher Education (2021): 1-21.

docs/index.md

@@ -23,135 +23,3 @@ An evaluation of the ELIXIR course evaluation.
 - Appendix
     - [The current ELIXIR short-term evaluation](elixir_evaluation.md)
 
-### How good are these questions?
-
-The question is: how good are these questions?
-To determine this, I'll first describe how these
-questions are developed. After that, I will describe
-the written-down goal of the questions and each of these in
-detail.
-
-### How were these questions developed?
-
-To get a first idea of how good these evaluation questions are can be
-obtained by reading how these were developed.
-We can read in `[Gurwitz et al., 2020]` how this process went:
-'These metrics were developed out of those already collected by ELIXIR
-training providers, as well as from discussions with stakeholders,
-external training providers, and literature review', with references to
-two papers.
-The first paper described the demograpics and short-term (i.e. directly
-after the course) questions used by DataCarpentries `[Jordan et al., 2023]`,
-where the second is about long-term impact `[Brazas & Ouellette, 2016]`.
-At a first glance, this looks like a proper procedure.
-
-### What are the goals of these questions?
-
-```mermaid
-flowchart TD
-  goal_1[ELIXIR: Determine training quality]
-  goal_2[NBIS: Find out how participants have used the skills and knowledge they gained]
-  goal_3[NBIS: Improve the course and the materials we deliver]
-  goal_1 <-.-> |similarish| goal_3
-  goal_2 <-.-> |dissimilar| goal_3
-```
-
-To get a second idea of how good these evaluation questions are can be
-obtained by reading the goal of the questions.
-
-Here is the ELIXIR goal `[Gurwitz et al., 2020]`: 
-
-> We were interested in participant satisfaction as a reflection on training
-> quality in order to be able to inform best practice for ELIXIR training.
-
-Here the intention of the NBIS short-term evaluation form is quoted:
-
-> The intention of the STF survey is to find out how participants have used the
-> skills and knowledge they gained through participating in the NBIS course.
-
-To me, this seems like a copy-paste error from the long-term survey
-by NBIS...? Section 1 seems to be more in line with a short term
-evaluation:
-
-> It is really important to us in order to continually
-> improve the course and the materials we deliver
-
-So it seems the goal of the STF is to improve the course and its materials.
-
-
-### Missing questions
-
-There are some questions that were removed from the Data Carpentries
-evaluations.
-
-### How to evaluate these questions?
-
-The goal is this exercise is:
-
-- To evaluate the ELIXIR course evaluation
-- If possible, propose a better one
-
-To achieve this goal, these are the steps:
-
-- Collect evaluation questions that were thought of as 'good'
-- Of these and the NBIS SFT evaluation questions,
-  collect reasons why this would be a 'good' or 'bad' question
-- Of the questions and reasonings, assign a rating per question
-
-## Epilogue
-
-We know that teachers reflecting on their work is one of the
-best ways to increase his/her teaching quality.
-Or: 'student ratings can only become a tool for enhancement when they
-feed reflective conversations about improving the learning process and when
-these conversations are informed by the scholarship of teaching and
-learning `[Roxå et al., 2021]`.
-The other best way for teachers to improve is to do peer observations.
-Note that neither practice needs an evaluations.
-
-If we really care about teaching quality, shouldn't we encourage
-doing the things that actually work?
-
-## Notes
-
-I think this is a useful evaluation using learning objectives:
-- https://uppmax.github.io/bianca_workshops/evaluations/20241111/
-
-I think this is a useful evaluation for improving as a teacher:
-- https://uppmax.github.io/programming_formalisms/reflections/2024_autumn/20241122_richel/
-
-Here is an example of how useful questions can be determined:
-
-- https://github.com/UPPMAX/R-python-julia-matlab-HPC/blob/main/reflections/20241024_richel/README.md#evaluation-results
-- https://uppmax.github.io/programming_formalisms/reflections/2024_autumn/20241122_richel/
-
-## References
-
-
-
-
-- `[Ang et al., 2018]` Ang, Lawrence, Yvonne Alexandra Breyer, and Joseph Pitt.
-  "Course recommendation as a construct in student evaluations:
-  will students recommend your course?." Studies in Higher Education 43.6
-  (2018): 944-959.
-- `[Brazas & Ouellette, 2016]`
-  Brazas, Michelle D., and BF Francis Ouellette.
-  "Continuing education workshops in bioinformatics positively impact
- research and careers." PLoS computational biology 12.6 (2016): e1004916.
-- `[Gurwitz et al., 2020]`
-  Gurwitz, Kim T., et al. 
-  "A framework to assess the quality and impact of bioinformatics training
-  across ELIXIR." PLoS computational biology 16.7 (2020): e1007976.
-  [website](https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1007976)
-- `[Jordan et al., 2023]`
-  Jordan, Kari, François Michonneau, and Belinda Weaver.
-  "Analysis of Software and Data Carpentry’s pre-and post-workshop surveys."
-  Software Carpentry. Retrieved April 13 (2018): 2023.
-- `[Roxå et al., 2021]` Roxå, Torgny, et al. 
-  "Reconceptualizing student ratings of teaching to support quality discourse
-  on student learning: a systems perspective." Higher Education (2021): 1-21.
-- [Uttl et al., 2017]
-  Uttl, Bob, Carmela A. White, and Daniela Wong Gonzalez.
-  "Meta-analysis of faculty's teaching effectiveness:
-  Student evaluation of teaching ratings and student learning are not related."
-  Studies in Educational Evaluation 54 (2017): 22-42.

docs/introduction.md

@@ -32,6 +32,24 @@ where the literature is searched for how these questions
 came to be and by which criteria the best were selected,
 in the hope of establishing the usefulness of this survey.
 
+???- question "Which alternatives are suggested?"
+
+    One alternative that is suggested is to use learning outcomes:
+
+    - [An example evaluation using learning outcomes](https://uppmax.github.io/bianca_workshops/evaluations/20241111/)
+
+    Another alternative that is suggested is to ask for written feedback
+    only to rate teachers:
+
+    - [An example reflection that used written feedback for its teachers](https://uppmax.github.io/programming_formalisms/reflections/2024_autumn/20241122_richel/)
+
+    Also, there is a clear rule to assess the usefulness of an evaluation
+    questions: 'An evaluation question is useful if it is used in a reflection'.
+    Reflections that apply these rules:
+
+    - [Example 1, where some questions are assessed to be useless](https://github.com/UPPMAX/R-python-julia-matlab-HPC/blob/main/reflections/20241024_richel/README.md#evaluation-results)
+    - [Example 2, where all questions are used](https://uppmax.github.io/programming_formalisms/reflections/2024_autumn/20241122_richel/)
+
 Besides discussing the current survey question,
 this paper is the first to give a fully transparent process
 on how, with the same goals in mind, a similar set of

docs/results_1.md

@@ -1,6 +1,47 @@
 # Results of RQ1: What is the history of the ELIXIR evaluation questions?
 
-The NBIS short-term evaluation questions are based on the ones by ELIXIR.
+## What is the ancestry of the NBIS questions?
+
+The paper where these questions were described first
+in `[Gurwitz et al., 2020]` . We can read that these questions
+are based on `[Jordan et al., 2023]` and `[Brazas & Ouellette, 2016]`.
+These last two papers do not reference any academic papers on where
+their questions originated from.
+
+## What are the goals of these questions?
+
+```mermaid
+flowchart TD
+  goal_1[ELIXIR: Determine training quality]
+  goal_2[NBIS: Find out how participants have used the skills and knowledge they gained]
+  goal_3[NBIS: Improve the course and the materials we deliver]
+  goal_1 <-.-> |similarish| goal_3
+  goal_2 <-.-> |dissimilar| goal_3
+```
+
+To get a second idea of how good these evaluation questions are can be
+obtained by reading the goal of the questions.
+
+Here is the ELIXIR goal `[Gurwitz et al., 2020]`: 
+
+> We were interested in participant satisfaction as a reflection on training
+> quality in order to be able to inform best practice for ELIXIR training.
+
+Here the intention of the NBIS short-term evaluation form is quoted:
+
+> The intention of the STF survey is to find out how participants have used the
+> skills and knowledge they gained through participating in the NBIS course.
+
+To me, this seems like a copy-paste error from the long-term survey
+by NBIS...? Section 1 seems to be more in line with a short term
+evaluation:
+
+> It is really important to us in order to continually
+> improve the course and the materials we deliver
+
+So it seems the goal of the STF is to improve the course and its materials.
+
+## Development of the questions
 
 ELIXIR developed these evaluation questions to,
 as quoted from `[Gurwitz et al., 2020]`:
@@ -58,14 +99,9 @@ Neither does the referred literature:
   Also here, it is not described how the evaluation questions
   came to be and with which reasoning the best were selected
 
-We may conclude that for neither the ELIXIR evaluation questions,
-nor for its ancester questions,
-it is not described how the evaluation questions
-came to be and with which reasoning the best were selected.
-
-This paper is a first to give a fully transparent process
-on how evaluation questions came to be from an ELIXIR node
-and with which reasoning the best were selected.
+Taking a closer look at the evaluation questions of `[Jordan et al., 2023]`,
+we see that some questions of its questions were not used.
+The reasoning behind this is absent in `[Gurwitz et al., 2020]`.
 
 ## References