FAMULUS
(Funding Period: 2017 - 2020)

Fostering diagnostic competence in medical and teacher education via adaptive online-case-simulations

Motivation

Diagnostic reasoning is a key competence in many professions. Examples are diagnosing diseases and suitable therapies in medicine and diagnosing whether pupils with learning difficulties suffer from a disability in educational settings. A good university curriculum should thus not only educate students regarding factual knowledge but also support them in developing diagnostic skills.

The interdisciplinary FAMULUS project aims to study how online case simulations that provide automatic adaptive feedback can foster students' diagnostic skills. To generate automatic feedback, we will develop novel methods for identifying and evaluating diagnostic reasoning (e.g. hypothesis generation, evidence generation and evaluation, hypothesis acceptance or rejection) in student essays. The effect of such feedback on the development of diagnostic skills will then be evaluated in a study conducted with students from medicine and education.

Goals

The FAMULUS project aims to answer the following questions regarding diagnostic essays written by students in medicine and education:

  • How do different formats of online case simulations (e.g. “whole case” versus “serial cue”) affect the diagnostic reasoning of students with different levels of background knowledge? [study conducted by project partners]
  • Is it possible to automatically and reliably detect diagnostic reasoning in student essays?
  • How can the quality and suitability of diagnostic reasoning in an essay be automatically evaluated to generate adaptive feedback?
  • How useful is the automatic feedback for students compared to providing a model essay? [study conducted by project partners]
  • Does diagnostic reasoning differ when performed individually as compared to cooperatively? [study conducted by project partners and usage of automatic analysis from 2.]
  • Can the methods developed for the automatic analysis and evaluation of diagnostic reasoning also be applied to essays of students from other fields of study?

Partners

This project was completed in cooperation with:

Institute for Medical Education (Ludwig-Maximilian University Munich), Prof. Dr. Martin Fischer, Principal Investigator

Chair of Education and Educational Psychology (Ludwig-Maximilian University Munich), Prof. Dr. Frank Fischer, Principal Investigator

Rust, Phillip ; Pfeiffer, Jonas ; Vulić, Ivan ; Ruder, Sebastian ; Gurevych, Iryna (2021):
How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models.
In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 3118-3135,
Association for Computational Linguistics, 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP 2021), virtual Conference, 01.-06.08.2021, [Conference or Workshop Item]

Pfeiffer, Jonas ; Rücklé, Andreas ; Poth, Clifton ; Kamath, Aishwarya ; Vulić, Ivan ; Ruder, Sebastian ; Cho, Kyunghyun ; Gurevych, Iryna (2020):
AdapterHub: A Framework for Adapting Transformers.
In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing : System Demonstrations, pp. 46-54,
Association for Computational Linguistics, EMNLP 2020 : Conference on Empirical Methods in Natural Language Processing, virtual Conference, 16.-20.11.2020, ISBN 978-1-952148-62-0,
DOI: 10.18653/v1/2020.emnlp-demos.7,
[Conference or Workshop Item]

Pfeiffer, Jonas ; Vulić, Ivan ; Gurevych, Iryna ; Ruder, Sebastian (2020):
MAD-X: An Adapter-based Framework for Multi-task Cross-lingual Transfer.
In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7654-7673,
Association for Computational Linguistics, 2020 Conference on Empirical Methods in Natural Language Processing, virtual Conference, 16.-20.11.2020, ISBN 978-1-952148-60-6,
DOI: 10.18653/v1/2020.emnlp-main.617,
[Conference or Workshop Item]

Bauer, Elisabeth ; Sailer, Michael ; Kiesewetter, Jan ; Shaffer, David Williamson ; Schulz, Claudia ; Pfeiffer, Jonas ; Gurevych, Iryna ; Fischer, Martin R. ; Fischer, Frank (2020):
Pre-Service Teachers’ Diagnostic Argumentation: What is the Role of Conceptual Knowledge and Cross-Domain Epistemic Activities?
In: The Interdisciplinarity of the Learning Sciences: 14th International Conference of the Learning Sciences (ICLS) 2020: Conference Proceedings, pp. 2399-2400,
International Society of the Learning Sciences, 14th International Conference of the Learning Sciences, Virtual Conference, 19.-23.06.2020, ISSN 1573-4552, ISBN 978-1-7324672-9-3,
[Conference or Workshop Item]

Simpson, Edwin ; Pfeiffer, Jonas ; Gurevych, Iryna (2020):
Low Resource Sequence Tagging with Weak Labels.
In: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 8862-8869,
AAAI Press, 34th AAAI Conference on Artificial Intelligence (AAAI 2020), New York, USA, 07.-12.02.2020, e-ISSN 2374-3468,
DOI: 10.1609/aaai.v34i05.6415,
[Conference or Workshop Item]

Pfeiffer, Jonas ; Simpson, Edwin ; Gurevych, Iryna (2020):
Low Resource Multi-Task Sequence Tagging - Revisiting Dynamic Conditional Random Fields.
In: arXiv-Computer Science, In: Computation and Language, (Preprint), [Article]

Bauer, Elisabeth ; Sailer, Michael ; Kiesewetter, Jan ; Schulz, Claudia ; Pfeiffer, Jonas ; Gurevych, Iryna ; Fischer, Martin R. ; Fischer, Frank (2019):
Using ENA to Analyze Pre-service Teachers' Diagnostic Argumentations: A Conceptual Framework and Initial Applications.
In: Advances in Quantitative Ethnography, pp. 14-25,
Springer, 1st International Conference on Quantitative Ethnography (ICQE 2019), Madison, USA, 20.-22.10., ISBN 9783030332310,
DOI: 10.1007/978-3-030-33232-7_2,
[Conference or Workshop Item]

Schulz, Claudia ; Meyer, Christian M. ; Gurevych, Iryna (2019):
Challenges in the Automatic Analysis of Students’ Diagnostic Reasoning.
In: Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence (AAAI-19), pp. 6974-6981,
Hawaii, USA, Thirty-Third AAAI Conference on Artificial Intelligence (AAAI-19), [Conference or Workshop Item]

Alhindi, Tariq ; Pfeiffer, Jonas ; Muresan, Smaranda (2019):
Fine-Tuned Neural Models for Propaganda Detection at the Sentence and Fragment levels.
In: 2nd Workshop on Natural Language Processing for Internet Freedom: Censorship, Disinformation, and Propaganda, pp. 98-102,
EMNLP-IJCNLP 2019, Hong Kong, China, 03.11.2019--07.11.2019, [Conference or Workshop Item]

Pfeiffer, Jonas ; Meyer, Christian M. ; Schulz, Claudia ; Kiesewetter, Jan ; Zottmann, Jan ; Sailer, Michael ; Bauer, Elisabeth ; Fischer, Frank ; Fischer, Martin R. ; Gurevych, Iryna (2019):
FAMULUS: Interactive Annotation and Feedback Generation for Teaching Diagnostic Reasoning.
In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations, pp. 73-78,
The 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations, Hong Kong, China, 03.11.2019-07.11.2019, [Conference or Workshop Item]

Schulz, Claudia ; Meyer, Christian M. ; Kiesewetter, Jan ; Sailer, Michael ; Bauer, Elisabeth ; Fischer, Martin R. ; Fischer, Frank ; Gurevych, Iryna (2019):
Analysis of Automatic Annotation Suggestions for Hard Discourse-Level Tasks in Expert Domains.
pp. 2761-2772, The 57th Annual Meeting of the Association for Computational Linguistics (ACL 2019), Florence, Italy, 28.07.2019-02.08.2019, [Conference or Workshop Item]

Schulz, Claudia ; Meyer, Christian M. ; Sailer, Michael ; Kiesewetter, Jan ; Bauer, Elisabeth ; Fischer, Frank ; Fischer, Martin R. ; Gurevych, Iryna (2018):
Challenges in the Automatic Analysis of Students' Diagnostic Reasoning.
In: ArXiv e-prints, ISSN 1811.10550,
[Article]

Schulz, Claudia ; Sailer, Michael ; Kiesewetter, Jan ; Bauer, Elisabeth ; Fischer, Frank ; Fischer, Martin R. ; Gurevych, Iryna (2018):
Automatic Recommendations for Data Coding: a use case from medical and teacher education.
In: Proceedings of the 14th eScience IEEE International Conference, pp. 364-365,
Amsterdam, Netherlands, 29.10.2018--01.11.2018, DOI: 10.1109/eScience.2018.00100,
[Conference or Workshop Item]

Schulz, Claudia ; Kiesewetter, Jan ; Sailer, Michael ; Bauer, Elisabeth ; Fischer, Martin R. ; Fischer, Frank ; Gurevych, Iryna (2018):
The Theory of Scientific Reasoning and Argumentation in Practice.
Bridging the Gap between Formal Argumentation and Actual Human Reasoning, Bochum, 4.-5. Oktober 2018, [Conference or Workshop Item]

Schulz, Claudia ; Sailer, Michael ; Kiesewetter, Jan ; Meyer, Christian M. ; Gurevych, Iryna ; Fischer, Martin R. ; Fischer, Frank (2018):
Automatische Analyse von Diagnosekompetenzen in Fallsimulationen.
In: 6te Jahrestagung der Gesellschaft für Empirische Bildungsforschung (GEBF 2018), pp. 543-544,
Basel, Switzerland, [Conference or Workshop Item]

Schulz, Claudia ; Sailer, Michael ; Kiesewetter, Jan ; Meyer, Christian M. ; Gurevych, Iryna ; Fischer, Frank ; Fischer, Martin R. (2017):
Fallsimulationen und automatisches adaptives Feedback mittels Künstlicher Intelligenz in digitalen Lernumgebungen.
In: e-teaching.org Themenspecial „Was macht Lernen mit digitalen Medien erfolgreich?“, pp. 1-14. [Article]

Sailer, Michael ; Kiesewetter, Jan ; Meyer, Christian M. ; Fischer, Martin R. ; Gurevych, Iryna ; Fischer, Frank (2017):
Förderung von Diagnosekompetenzen durch simulationsbasiertes Lernen im Lehramtsstudium.
In: Educational Research and Governance. Abstractband zur Tagung der Sektion „Empirische Bildungsforschung“ (AEPF), p. 189,
Tübingen, Germany, [Conference or Workshop Item]

Funding

The project was funded by the German Federal Ministry of Education and Research (BMBF) from 2017 – 2020 as part of the program “Digital Higher Education” (Digitale Hochschullehre).