AIM Question format

Theodore Kolokolnikov , Norbert Van den Bergh

Introduction

Overview and examples of questions

Detailed description of flags:

Computing and displaying output: h>, p> and t> flags
Specifying a grading scheme: a> flag or s> flags.
Specifying a solution: sb>.. se> flag block.
Specifying question type: the c> flag
Hints and subquestions: the sq> ... end> flag block
Including another question: the i> flag
Greek letters, subscripts and other math typograpy
Other flags
Auxiliary functions and global variables
How to monitor student's progress

Introduction

The purpose of AIM web server is to use Maple and the web to automatically administer and grade homeworks and tests. In a typical setup, the students take tests over the web. The computer then automatically stores and grades their answers. At a later date the computer can generate a solution to the quiz for students, and a class grades report for the teacher.
The task of preparing a new course for the AIM Server involves creating:
Usually creating a quiz definition is nothing more than choosing a name for a quiz and, if needed, changing the default question order. By far the biggest amount of time is spent preparing questions. The advantange of separating quizzes from questions is that the questions themselves can be reused, and selected at random for a given test.

Overview and examples of questions.

Question file contains a definition of questions that are later on compiled into a database from which the quizzes are constructed. Each statement of the question file begins with a flag, followed by some text. Depending on the flag, the text can be a Maple statement or just plain text.
Let's have a look at the most basic "toy" example:
# A question on addition.
t> What is 2+2?
a> 2+2
end>
This example creates a question which aks "what is 2+2?" then reads the answer. Here is the meaning of each flag, line-by-line:
The next example illustrates how to randomize questions and how to use 'procedures' to test answers.
Question definition What student sees
h> p_ := x^4+
randpoly(x, degree=3, coeffs=rand(-3..3));
t> Evaluate the following integral:
p> Int(p_, x)
s> (ans)->Testzero(diff(ans, x)-p_),
int(p_, x)
end>		 Evaluate the following integral:

/
| 4 3 2
| x - x + x - 2 x + 2 dx
|
/
This example also introduces the following new flags:
One can also include multiple-choice questions. For example:
Question definition What student sees
t> The following figure:
p> plots[display](plottools[torus](
[0,0,0], 1, 2), scaling=constrained);
t> is equivalent to...
c> MC("coffee cup", teapot, "wine glass")
a> "cofee cup"
end>		 The following figure:
is equivalent to...

The flag c> ("check type") has two different uses. In this question, it is used to specify that the question is of type 'multiple choice', as well as a Maple sequence of possible choices. The other use is to specify the expected Maple type of the answer, if an answer is to be a Maple expression. Here, we used p> flag to draw a Maple graphic. Finally the a> flag specifies the correct answer.

It is preferable to use multiple-response questions instead of multiple choice, the difference being that more than one choice may be correct, and students have to choose all correct choices to get full marks. The reason is that there are 2n possible answers compared to just n for multiple choice. Here is an example:

Question definition What student sees
# A multiple-response question.
t> Consider the following matrix:
p> A = matrix(2, 2,
[cos(theta),-sin(theta),
sin(theta), cos(theta)]);
t> Which of the following properties does this matrix posess?
c> MR(symmetric, "anti-symmetric", orthonormal, singular, "non-singular")
a> {orthonormal, "non-singular"}
end>

Another advantange of MR questions is that they have partial grading built in, without any extra assistance from the teacher's behalf. For instance, if the student were to choose "anti-symmetric" (which is wrong) and "orthonormal" and "non-singular" (which are right), he would have gotten 50%. See here for more information on grading MR questions.

A recent addition to AIM is its ability to provide conditional hints. This feature is both another method of providing part marks (students are penalized for viewing hints) as well as an attempt to provide more feedback to the student. See here for an example.

Detailed description of flags

Computing and displaying output: h>, p> and t> flags

These three flags are used most commonly. Use h> to include hidden Maple commands, p> to print a math expression (or a Maple graph) and t> to type text, html tags or math expression.
Flag
Description
examples
h> cmd
Execute a Maple cmd. The output is hidden from the student. cmd is assumed to be a valid Maple command.
h> n_ := rand(5..9)();
t> What is $n_<sup>2</sup>?
a> n_^2
end>
p> stuff
Prints stuff using Maple's prettyprinter. This is useful for displaying mathemetics (for instance matrices) in two-dimensional format, or to include graphs: if stuff is a Maple plot (or a Maple plotting command), then the plot will be displayed as a gif graphic.
Math example (display a matrix):
h> A_ := matrix(2,2,[1,2,
3,4]);
p> A_
a> -2
end>

Graph example:
p> plot3d(sin(x)*cos(y),x=-4..4, y=-4..4);
t> text
Type out text followed by a new line.
Variables or greek letters may be included by proceeding them with a $ sign.
 
t> What is 2+2?

h> n_ := rand(10..20)();
t> What is $n_ mod 3?

t> Greek 'Gamma' looks like this: $Gamma.
end>

Specifying a grading scheme: a> flag or s> flags.

These flags are used to provide a grading scheme as well as a model answer. Use a> ("answer") flag when the right answer is mathematically unique; use s> ("solution") when there is more than one correct answer, or to assign partial marks.
Syntax for a> flag
a> ex
Description
ex is a maple expression which is the solution to the question. When this flag is read, the student is prompted for an answer (see also ap> flag). If his answer is mathematically equivalent to ex then the student receives full marks, otherwise he receives zero.
This flag can also be used when the the type of the question is MR (Multiple Response). In this case a different grading procedure is used (see c> flag for more information).
This flag also signals the end of a part of the question, unless followed by i>, sb>...se>, or sq>...end> block in which case the end of the question part comes after it.
Example

t> What is 2+2?
a> 4
end>
See also:
s>, c> ap>, end>, `aim/Testzero`
Syntax for s> flag
s> fcn, sol
Description
Grades using a solution procedure fcn. Use this flag instead of a> if a question has more than one solution or to give partial marks. fcn is a Maple procedure that accepts two arguments and returns either true/false or a number between 0 and 1. The first argument passed to fcn is the student's answer. The second (optional) argument is sol which is assumed to be a correct solution to the question.
If fcn returns true then full marks are given. If fcn returns false then no marks are given. If fcn returns a number between 0 and 1 then the grade assigned is that number multiplied by the value of the question (see v> flag). If fcn causes an error (such as division by zero) then a grade of zero is assumed. If fcn returns FAIL then no grade is assumed (this is different then a zero grade; see section on penalty).
This flag also signals the end of a part of the question, unless followed by i>, sb>...se>, or sq>...end> block in which case the end of the question part comes after it.
Example
t> What is 2+2?

#The following procedure gives part marks
#if the answer is correct to within 10%.
s> proc(chk, sol)
if evalf(chk-sol)=0 then
1;
elif evalf(abs((chk-sol)/sol)) < 0.1 then
0.5
else
0
fi;
end, 4
See also:
v>, a>, ap>, sb>...se>, end>

Specifying a solution: sb>..se> flag block.

It is possible to include a detailed solution to a question by putting it between sb>..se> flags. A solution will only be shown after the due date (if specified) and after the student presses "show solution" button).
Syntax:
sb>
[sequence of flags]
se>
Description
The sequence of flags is executed only when the solution is to be shown (after the due date and after the student uses "show solution" button). Note that this is different from hints (see sq> flag).
Example

t> What is the area under the curve
p> y=x^2
t> for x between 0 and 1?
a> 1/3
sb>
t> The answer to this question can be found by evaluating
p> Int(x^2, x=0..1)
se>
end>
See also:
sq>, end>

Specifying question type: the c> flag

This flag is used to specify the answer type. The answer can be either an arbitrary Maple type or one of predefined types described below. Predefined answer types also select the default grading procedure (which may be changed).
Answer type Description Default grading procedure Example
unspecified Any valid Maple expression will be accepted for an answer. Algebraic comparison: Substract teacher's answer from student's answer and test if the result is mathematically zero using Maple's simplification machinery. If the result is zero then the student gets full marks; otherwise the student gets zero. t> Find the integral of
p> Int(sin(x)^2,x=0..t)
a> int(sin(x)^2,x=0..t)
MC(c1,c2,...) Multipe choice question. c1,c2,... are allowed responses. The student must select one of c1,c2,... If the answer selected by the student corresponds to teacher's answer, the student gets full marks. Otherwise the student gets zero. t> What is 2+2?
c> MC(1,2,3, "none of above")
a> "none of above"
MR(c1,c2,...) Multipe response question. The student must choose all correct answers from the list c1, c2, ... Grade out of 1 = max(0,(Rs-Ws)/R)
where Rs is the number of right choices and Rs (Ws) is the number of right (wrong) choices that the student had selected.		 t> Which of the traits below apply to Napoleon?
c> MR(Short, Tall, Italian, French)
a> {Short, French}
matrix(n,m) The answer is an n by m matrix. Entry-by-entry algebraic comparison. t> What does a 2x2 identity matrix look like?
c> matrix(2,2)
a> matrix(2,2,[1,0,0,1])
[t1,t2,...,tn] The answer is a list of n elements. The first element must be of Maple type t1, the second of t2 etc. Entry-by-entry algebraic comparison. h> x_ := rand(-5..5)();
y_ := rand(-5..5)();
t> Give the x and y coordinates of the minimum of the curve
p> y=expand((x-x_)^2+y_)
c> [anything, anything]
a> [x_,y_]
string The answer will be treated as a Maple string. Unlike any other type, a string does not have to be a maple expression; it could be an English sentence for instance. String comparison. No mathematical verification or Maple simplification is performed. t> Simplify 12/15 as much as possible. Give your answer as a/b where a,b are two whole numbers. Do not use decimal point.
c> string
a> "4/5"
string(n) Same as string except a text box of height n lines will be provided for input (the input may be longer than n lines; the text will wrap). This type is primarily used for providing comments in evaluation surveys, or for questions that are hand-marked. t> What is the meaning of life?
c> string(20)
a>
string(tags) Same as string but use custom html tags for input box. See string(n) t> Give your comments.
c> string("cols=80,rows=40")
constant The expected answer is constant. Accept approximate numeric answer. Anything that is within 2% of the true answer will be accepted as correct. t> What is 2+2?
# Accepted answers are 4
# or 4.0 or 4.01
c> constant
a> 4
Any Maple type The exected answer must be the given Maple type. If the answer is not of the given type then a syntax error is returned.The student must try again but no grade is given. Otherwise an algebraic comparison is done. t> Give an equation of a quadratic going through points (-1,1),(0,0)and (1,1). Your answer must of of the form y=a*x^2+b*x+c.
c> AND(`=`, dependent(x,y))
a> y=x^2
If a question type is NOT specified but a> flag is used and the teacher's answer is of Maple type constant then the question type is assumed to be constant.

Hints and subquestions: the sq> ... end> flag block

A subquestion is a sequence of flags delimited between sq> and end> flags. It is used to include hints (with partial grading). Hints may themselves include questions or hints for students to answer. If a hint includes an a> or an s> flag, a grade is assigned to it and is then factored into the question total grade using the formula described later in this section. By default, the content of a hint is not displayed; it is displayed only if the student chooses to see it. This is because the teacher can assign a penalty for allowing to view a hint. Conditional hints are displayed whenever a teacher-assigned condition is fulfilled.

Hints are a way of giving part marks to poor students without burdening a good student with unnecessary work. This is because hints are optional: a good student should be able to answer the question straight away whereas a worse student may need to do it in steps.

Syntax:

sq> penalty=p, value=v, condition=c
body
end>
Description:
Any arguments following the sq> flag are optional and can be present in any order. p and v must be numbers from 0 to 1 representing a fraction of the question value (both are assumed zero if not specified). c must be a boolean expression returning either true or false. body is a sequence of question flags.

If condition=c is present then the body of the subquestion will be executed if c evaluates to true. If condition=c is not present, a checkbox is displayed, along with penalty and value of the subquestion. If the student checks the checkbox and validates the quiz then the subquestion is expanded and the body is executed.

An execution of a subquestion may affect the total grade of the question if p or v are nonzero (both are assumed zero if not specified). v indicates the value assigned to the subquestion, as a fraction of the question value. p is the penalty given for executing the subquestion. For example, suppose a question is graded out of 1 and the student gets a grade q (out of 1) on the question and a grade h (out of v ) for the hint whose penalty was p .Then the student's final grade for the whole question would be (1-p ) (q+ h/v)/(1+v ) out of 1. A similar grading procedure is applied if a question contains more than one hint. In this case the penalties are additive.

Hints may also include subhints, in which case the above grading scheme is applied recursively.

If a question contains more than one hint, only one unexpanded hint will be shown at any time. The hints are shown and expanded in the order in which they appear.

Once a hint is expanded, it cannot be collapsed again. This is also true for conditional hints, even if c changes from true to false.

Examples:
What teacher types What student sees - before hint expansion What student sees - after hint expansion
h> f_ := `*`(op(combinat[randcomb](
[exp(-x), sin(2*x), cos(3*x)], 2)))
t> Evaluate the following integral:
p> Int(f_, x)
s> (ans)->`aim/Testzero`(diff(ans, x)-f_), int(f_,x)

sq> penalty=0.2
t> Use integration by parts.
end>
Evaluate the following integral
/
|
| exp(-x) sin(2 x) dx
|
/

Answer:
Click here: for a hint. Penalty=20%, value=0% of question value.

Evaluate the following integral
/
|
| exp(-x) sin(2 x) dx
|
/

Answer:

Hint (Penalty=20%, value=0% of question)

Use integration by parts.
If the student gave a correct answer without expanding the hint, he would get 100%, but if he used the hint, then he would obtain only a maximum of 80%.

A useful way of giving hints without much extra effort is to include other questions using the i> flag. This way the teacher can re-use the "easy" questions when asking "harder" questions. Another advantange is that the included question may also have hints and so on. The next example demonstrates this.
Question definitions Second question, before expansion looks like this: Second question, when expanded, looks like something like this:
k> 1
t> Find the derivative of
p> x^2*sin(x)
a> 2*x*sin(x)+x^2*cos(x)
end>

k> 2
t> Find a derivative of
p> x^2*sin(x)
t> at a point x=$Pi/2.
a> Pi
sq> penalty=0.1, value=0.5
i> keyword=1
end>
Find a derivative of 
             2
            x  sin(x)
at a point x= p /2.
Answer:
Click here: for a hint. Penalty=10%, value=50% of question value.
Find the derivative of 
             2
            x  sin(x)
at a point x= p /2.
Answer:
Hint (Penalty=10%, value=50% of question)
Find the derivative of 
         2
        x  sin(x)
Answer:

If the student gives a correct answer to both the question and its hint (as done above), he would get 90% on the question. If the student answers the hint correctly and the question incorrectly, his grade would be 30%. Finally if the student answers the question correctly and the hint incorrectly, he would get 60%.

Including another question: the i> flag

You can use the i> flag to reference other questions and subquestions.

Syntax

i> file_list, options

file_list is a comma-separated sequence of question files, and options is a comma-separated sequence of equalities option=value as described below. All arguments are optional.

Description
If no options are given, then one question will be selected at random from one of the files given by file_list. An empty file_list is equivalent to the file_list which contains only the current question file name.
Options are used to further refine the question choice. They may be one of the following: Each question or a part of a question can contain at most one "a>" or "s>" flag. If both the original and the included questions contain these then an error will be issued.
Examples
Suppose you have two question files, the first one called "q1" and the second "q2" with the following contents:
Contents of q1 Contents of q2
k> include_me
t> This is question 1 from q1
end>

k> include_me
t> This is question 2 from q1
end>

t> This is question 3 from q1. It has no keywords.

 t> This is question 1 from q2. It includes a random question from file q1.
i> q1
end>

t> question_2
t> This is question 2 from q2. It includes a random question from this file (but will not inlcude itself)
i> selfun= ((v, k) -> not has(k, question_2)
end>

t> This is question 3 from q2. It includes a random question from file q1 with keyword include_me.
i> q1, keyword=include_me
end>

A quiz that contains all questions from q2 might look something like this:

Question 1 (0 marks)
This is question 1 from q2. It includes a random question from q1. This is question 3 from q1. It has no keywords.
Question 2 (0 marks)
This is question 2 from q2. It includes a random question from this file (but will not inlcude itself). This is question 3 from q2. It includes a random question from q1 with keyword include_me. This is question 2 from q1.
Question 3 (0 marks)
This is question 3 from q2. It includes a random question from q1 with keyword include_me. This is question 2 from q1.

Other flags

Flag
Description
example
end>
Signals the end of one question and a beginning of another. Note that each question may have more than one part. An end of a question part is signalled by a> or s>flags.
t> What is 2+2?
a> 4
end>

t> What is 2+3?
a> 5
end>
forbid> names
Forbids names from appearing in the student's answer. names is a sequence of Maple symbols.
p> `Evaluate`, Int(x^2, x)
forbid> int, Int
a> x^3/3
k> keywords
Specifies additional keywords related to this question. keywords is a sequence of Maple names. Note that the name of the question file is automatically added to the list of the keywords, even if no k> flag is specified. If the list contains an element of the form QID = question_identifier then a log will be kept in the student's marks file of the results obtained for the question "question_identifier". Statistics about the number of times these questions were attempted and about their average results can be obtained from the teacher's "statistics" page.
k> addition, arithmetic
t> What is 2+2?
a> 4
v> max
Specifies the value of the question. By default the value is 1.
# Easy question with (default) value 1:
t> What is 2+2?
a> 4
end>

# More difficult question, value 2 :
v> 2
t> What is 20+20?
a> 40
ap> text
Change the prompt used when asking a question. The flag has the same syntax as the t> flag.
t> Give the x and y coordinate
of the minimum of
p> y=x^2
ap> x-coordinate:
a> 0
ap> ycoordinate:
a> 0

Greek letters, subscripts and other math typography

It is possible to include Greek letters by using their English transliteration. This can be done in two ways.
To include subscripts in your output, simply enclose them in square brackets [ ]. For example, t> a[b] will appear as ab in your output.
One can use any other html tags explicitly. This is useful to include superscripts. For example:
t> <i>Hint</i>: consider <b>R</b><sup>3</sup>.
which will appear as
Hint: consider R3.
t> re&euml;el proced&eacute
will appear as : reëel procedé

Auxiliary functions and global variables

Several Maple functions defined by AIM can be used within quizzes, especially within s> flag. These are: `aim/Testzero`,`aim/TestNumeric`,`aim/p` and `aim/t`.
`aim/Testzero`
extends the functionality of Maple's Testzero function to matrices and lists. In addition, it will try to simplify the input using trig identities. Fore xample `aim/Testzero`(sin(x)^2+cos(x)^2-1) will return true.This routine is invoked by default by the a> flag.
`aim/TestNumeric`
tests whether the two numeric values are approximatively equal. This routine is also used by default by the a> flag when c> constant is specified. It accepts two numeric arguments and returns true iff they are within 5% of each-other. For example, all of the following return true: `quiz/TestNumeric`(10, 9.81), `quiz/TestNumeric`(3, Pi), `quiz/TestNumeric`(1.4, sqrt(2)).
`aim/t`, `aim/p`
are equivalent to t> or p> flags. The argument to `aim/t`must be a string. For example `aim/t`("What is $a_+$b_?") is equivalent to t> What is $a_ +$b_?

AIM also provides several useful global variables. These are ta (teacher's answer) and sa (student's answer). These variables refer to the current question. They are set to NULL at the top of the question.

How to monitor student's progress


AIM provides extensive facilities for administering questions and quizzes. All of these facilities are available through a web interface by going to teacher's page, and are explained there. Here we will discuss mainly how to monitor student's progress. For this, AIM provides four different tools: the ability to "spy" and change student's answers, grade reports, aggregate statistics, and log files. Of these, the "spy" tool is the most used.

When the teacher creates a course he/she chooses his/her own password. This password overwrides individual student's passwords: you can log in as any student by using teacher's password and that student's ID. The teacher then can, just like a student, modify any of the student's answers. In addition the teacher can change student's penalties (which are not accessible to student). By logging in with teacher's password one can also view model answers for each student (which may be different from student to student because of randomisation) before the due date. This makes it easy to respond to the student's emails and is also useful to detect any bugs in the questions.

AIM records student's answers to the quizzes only if the student logs in under a name (which can be arbitrary, or which can be restricted to a list of registered students), and regenerates a quiz each time the student reloads it. No recording happens if the student logs in with a blank name: in that case then also no log is kept of the responses to the questions which are tagged with a "QID = ...." keyword.
Before questions are chosen for a given test, the random number generator is set to some value that depends only on the student ID (unless no ID is specified -- in which case the number generator is not reset). Once the questions are chosen but before each question is executed, the random number generator is set to some value that depends only on the student ID and the question number. This means that as long as the student uses the same ID, the test he gets will be the same everytime he logs in (you can override this on the teacher's quiz-page by setting randomisation to "always", so that also a fixed set of registered students always gets new questions when they log in; this is mainly useful if you are interested in the results obtained for questions tagged with a QID.
Bear in mind that, when randomisation is "off", you should not add any new questions during the period that the tests are open to students, or change how the question is randomized; otherwise the student's answers may get out of sync with the test.

However you can modify questions while the students are taking the tests, as long as your modifications don't change how the question is randomized (for instance it is always OK to modify the text of the question, but not ok to add or delete a new random variable). If you correct a bug which changes the grading procedure, you might want to re-grade all the students (option available on teacher's page).

The log and grade reports are updated everytime the student presses "validate" or "grade" button of the test. See "other options" on teacher's page to change how the overall grade is computed.

The teacher can also generate statistics for the whole class, showing how many students answered what for each question of a given quiz. This is especially useful for making electronic surveys, and less useful for quizzes that have a lot of randomisation. For the latter a useful feature is the possibility of obtaining statistics about the use (and the results) of questions carrying a "QID = ..." tag: just add the keyword "QID = question_identifier" after the k> flag and obtain a list of the number of times this question was attempted and of the average mark. This only works however for students who did not log in with a blank name.

One of the goals when designing AIM was to be able to use it for administering graded homeworks and not just self-assessment. To this end the teacher can set up a due date. Up to the due date, the students can enter their answers and correct them, but they will not be able to get the correct solutions from the computer. The teacher can opt either for giving feedback for each question separately (with a possible exception for the multiple choice questions) or, when "feedback" is off and when "minimal nr of answers" is on, for giving a total mark of the quiz only.
After the due date they cannot modify their answers but the model solution to the homework is made available (however if you login with the teacher's password, you can always modify answers or view solutions). Of course AIM can also be used for self-assessement, simply by removing the due date.

The students may communicate with the teacher by email, through a link conveniently placed on every quiz. When you set up a course, make sure you set the email address correctly by choosing "other options" from teacher's page and changing the "test end" string.