The global research question should be more clearly stated in the text. This should also help with connecting the two parts of the dissertation.
The section "Structure of this dissertation" has been edited to explicitly mention a global research question, and a research question for each chapter.
There should be more focus on the lessons learned when building an assessment platform like Dodona.
By adding a research question and conclusion to chapters 2, 3, and 4, this remark should be handled.
Chapter 6 is of lower quality than the rest of the chapters.
The work on the included article was continued after the dissertation was submitted, and that article has now been submitted to the Journal of Artificial Intelligence in Education. This version of the article is now included, which should solve the remarks on this chapter.
It would be good to compare Dodona and other modern platforms, perhaps via a comparative table.
This comparative table has already been created by Sven Strickroth at https://systemscorpus.strickroth.net/. I tried including a selection from this table in the dissertation, but found it did not add much to the text.
The final chapter is short and could be expanded upon.
The possibilities for future work have each been expanded to include more detail.
At the end of the chapter an overview of the releases is given, while informative it would have been fine to move this part to the appendix or simply point to the repository.
This has been moved to an appendix.
While the text reads good, it might have been more logic to group the work around TESTEd together with the explanation of how judges are implemented.
I tried this out, but the text didn't really work well with this change applied, so I reverted it.
The explanation of the R judge could have also been a bit more in depth.
While short, the interesting technical challenges in the R judge have been explained fully. Therefore I did not expand on the section.
The wording used throughout the dissertation is of high quality. One suggestion for further improvement is to eliminate contractions like "isn't" and "can't," given that contractions are considered informal language.
These contractions have been removed.
Acronyms are not always written out in full upon first usage (e.g., SPOJ, MOSS, and so on).
I checked all acronyms, and this should now be fixed.
In the introduction of each chapter, it would be good to provide the complete bibliographic details of the paper(s) upon which the chapter is based.
This has been done.
Nitpicking: there is a difference between ‘amount of’ and ‘number of’. ‘Number’ is used for items that are countable, while ‘amount’ is used for items that are uncountable.
I check all occurrences, this has been fixed.
I would recommend adding a conclusions section, with summarizing take-home messages, to Chapters 2, 3, and 4.
This has been done.
... to build knowledge progressively. -> ... to build knowledge and skills progressively.
This suggestion was applied.
It may be useful to include an appendix with an example that illustrates the templating system adopted by TESTed.
While this would be interesting, TESTed is not the main focus of this PhD. For more detail on this, I would like to refer to the PhD of my friend and colleague Niko Strijbol, which goes much more in detail on TESTed.
The fact that the fourth series is an exception can perhaps also be attributed to students facing the combined use of conditional and repetitive execution for the first time, which challenges their skills in terms of keeping track of the flow of execution.
While true that this is challenging for students, most exercises in the third series (while repetitive execution is introduced) also contain conditional execution. Therefore I did not change the text related to this.
No suggestions for modification.
The bibliography section may require a bit of clean up for consistency of the references.
This has been done.
In the list of publications on pages ix and x it is not very clear what the difference between the different publications is (1 and 6, 2 and 8, 4 and 5). Some seem to have the same title but published elsewhere? Is it really a different publication then?
Some of these were publications of conference posters; these have been removed. The others are in fact different publications.
In chapter 4 before explaining all implementation choices and details of TESTed in section 4.4 maybe first show a motivating example first to keep the reader's attention. (For example the example of Listing 4.5 could come much earlier.)
This has been done.
Another recurrent (and easy to fix) remark is that the captions of most images/tables are way too big. It is better to have small captions that summarise the essence of a figure and to put all the detailed text that is now in the caption in the running text that refers to the figure.
I have gone through all the captions and edited them where possible.
Most of the document is (well-explained) running text, but some more rigorous explanation (formalism, pseudocode, formulas) would have been welcome at several places. (For example, on pages 50 and 109.)
The mathematics has been taken out of the running text and placed in its own environment.
LLMs are becoming more important, and the possible link with auto-graders is clear. You do mention them to some extent in chapters 6 and 7 but a slightly more thorough section on that would have been preferable. (This remark is a suggestion and not a blocking factor.)
The possibilities for future work have been expanded upon in the final chapter, including the LLM sections.
The differences and similarities between learning analytics and educational data mining could be more clearly described, both theoretically and in the practical section.
Definitions of LA and EDM have been added to the introduction to make clear what is meant by these terms in this PhD.
It would be interesting to know more about the division of labor between Van Petegem and the rest of the Dodona team.
For each project, it is now described who contributed and what their role was (including me).
Clearer argumentation of ethical issues of Dodona would further strengthen the work.
A section on this has been added to the introduction.