RandD

Table of contents

1. Goals
2. TBS
3. TA-Bot_V1
4. TA-Bot_V2
5. Future Work

Goals

The creation of TA-Bot relied and built upon prior research in the Automated Assessment Tool field. Each design decision was made as an extension of prior work, or as a novel research experiment to a problem that was not addressed in prior work. At the onset of development, the system had several goals, but throughout it's creation, the system has continued to evolve to address the needs of educators and students.

Goals of the web version: Educators

• Reduce the complexity of creating test cases and projects.
  • Bash TA-Bot relied on a series of shell scripts each relying on several files for a single test case. The creation of these test directories was a time consuming and error prone process.
  • project creation, and test cases can be created via UI, reducing the need for users to have a robust understanding of the underlying system.
• Significantly reduce the time commitment in grading student assignments.
• Provide both high and level course analytics to instructors.
  • This feedback enables instructors to identify which students are struggling
  • Insights on how the class as a whole is performing
  • Help in identifying which students attend office hours or help desk hours the most, or least.
• Provide a secure environment to run students code.

Goals of the web version: Students

• Provide students with a more detailed feedback on their submissions.
  • Students get a detailed report of their submission, including the output of their code diffed against the expected output.
  • Students get code Style feedback via the use of Pylint, or another languages linter.
    • A custom Github Repo was created to reduce the complexity in understanding Pylint errors.
    • This can be found: here
• Provide an incentive to motivate students to start, and complete assignments earlier.
  • While there is no way to force students to start assignments earlier, we created a novel rate limiting system called Time Between Submissions(TBS) which yielded positive results. This concept is described in detail bellow.
    • Link to Poster: #TODO
    • Link to Paper: #TODO
    • Link to Paper: #TODO
• Provide an incentive to motivate students to attend help desk hours

Time Between Submissions(TBS):

The time between submissions(TBS) is a novel rate limiting system that was created to encourage students to start assignments earlier. The scheme was created for weekly assignments. A students rate limit starts at 5 minutes, but as the deadline approaches, the rate limit gradually increases as shown in the figure bellow. This encourages students to start assignments earlier, as they will be able to submit more frequently. In general, even with TBS enabled, students are able to receive feedback far more often than they would with a traditional TA-Bot system. This rate limiting scheme was created to work in conjunction with novel research experiments aimed at encouraging students to start assignments earlier, finish assignments sooner, and attend help desk hours more often.

While we highly encourage educators to use this, or another rate limiting scheme, it is not required. TA-Bot can be configured to run without TBS enabled. This is discussed in detail in the Educator Guide.

2021-2022 "Pylint Score TBS Incentive":

The 2021 semester was the first time Web-TA-Bot was used in an academic setting. Each submission was graded on a 100 point scale, with 60 points being allocated to test cases, and 40 points being allocated to a students Pylint score based on the following logic. Ideally, we hoped to encourage students to produce code that is more readable.

def pylint_score_finder(error_count):
    if error_count <= 10 and error_count > 7:
        return 25
    if error_count <= 7 and error_count > 5:
        return 30
    if error_count <= 5:
        return 40
    else:
        return 10

Their resulting Pylint score was combined with their test-case results, which were distributed evenly across the test-cases for a particular assignment.

Should a student get a score of at least 75 on week 𝑋 ’s assignment would enable a student to unlock a day of reduced TBS for week 𝑋 + 1, This reward was redeemable on the third day of an assignments lifespan, as denoted by green in the image above.

This resulted in significant improvements to student habits, code style, and students finishing assignments sooner, theses results are discussed in detail in the following paper:

• Link to paper: #TODO

2023-Current "Help Desk TBS Incentive:

In the 2023 semester, Marquette launched a Computer Science help desk. This help desk is staffed by Graduate and undergraduate TA's who are available to help students with their assignments. The help desk is open M-F from 10:00am-5:00pm.

A student unlocks a reduced rate limit should they attend these office hours to ask questions.

To see more about how this system operates, please refer to the Educator Guide, or for the student perspective, please refer to the Student Guide.

Research is ongoing, but at a minimum, this helps provide new insights on how many students are attending office hours or visiting a help desk.

Future work:

The following is a list of future work that is planned for TA-Bot, or is currently being developed

1. Project analytics and advanced early warning detection for at-risk students.
2. Automating the creation of test cases given a solution file.
3. Creating a robust scheduling platform designed to help students find mentors, schedule appointments, and attend help desk hours.