Rework how reducedScoring is computed and stored.

[somiaj]

This is paired with removing reducedScoring from PG. Instead webwork should handle taking the raw score from PG and adjusting the score based on reducedScoring options.

This adds a method apply_reduced_scoring_adjustment to ProblemUtils. This method needs to be run after the raw score is returned from PG, but before writing it to the database to apply any reduced scoring adjustments to the raw score. The method updates the current problem right before writing it to the database.

This also changes how the columns status and sub_status in the problem_user table are used. status still contains the actual adjusted grade used by webwork, while sub_status now stores the raw non adjusted grade. Provided the reducedScoringValue has not changed, it is possible to use the previous status and sub_status to recover the portion of the problem that was done before the reduced scoring date and the portion that was done after, so only the part of the problem done after the reduced scoring date is adjusted.

This needs to be merged with openwebwork/pg#650 and stems from the discussion openwebwork/pg#648.

Now that sub_status contains the raw score, it is now possible to address some of the concerns mentioned by @Alex-Jordan for places that should check the raw score and not the adjusted score when determining what to due (such as conditional release or indicating that a problem has been completed).

[somiaj]

With changing how sub_status is used in the database, there could be a possible upgrade issue from 2.16 for any current courses. If sub_status < status, it should be safe to assume this is an upgrade, so we can recreate the new sub_status from this (provided the instructor hasn't changed the reducedScoringValue):

sub_status += (status - sub_status) / reducedScoringValue

If sub_status == status this means the problem was done outside of the reduced scoring period, and these two fields will be the same both before the upgrade and after the upgrade, so there is nothing to be done.

In the old setup sub_status <= status, while in the new setup sub_status >= status, so the above should cover all possible cases. Thus to ensure a safe upgrade from <= 2.16, we should check if sub_status < status and then update sub_status if needed. I think this should be done in the database module when the data is retrieved from the database.

[somiaj]

I was thinking it is possible at the time an instructor changes the reducedScoringValue to loop through the problem_user table and update the status field based on sub_status, the old value, and the new value. This is probably a good idea to avoid any strange effects of changing this value in the math used to keep track of the full portion and reduced portion of the grade. Though it will be slow, as that table could be fairly big. I suspect this won't happen that much and then it would be like the weight, if you change it, it effects all assignments.

As for what to do if an instructor enables/disables reduced scoring. I don't see any way to nicely deal with being able to turn this on/off/on and keep track of the data. My initial thought is leave it alone, and in this case students could go increase their grade by earning full credit on any open assignments, but any past assignments will remain as they are. Another option is to update the status field if it goes from on to off, but this is one way, there would be no way to get the data back after that. Yet another possibility could be leave the table alone, but make the various places webwork reports the grade use the sub_status field instead of status if reduced scoring is disabled. This way if it is enabled again, all the data is still there. This still has the same issue is any open assignments students are working on will loose any past reduced scoring data once a new grade is submitted and the status field gets updated.

Since we aren't storing time stamps with grade attempts, there is probably nothing that can be done automatically if a reduced scoring date is changed.

[somiaj]

In testing this just noticed another issue with the manual problem grader. It is using the raw scores from PG, so when you first submit a quiz in the reduced scoring date, everything looks like it is 100%, but if you leave the quiz and come back, the reduced scoring has been correctly applied.

Edit: Fixed. Though the manual grader still shows the raw score on question parts.

[drgrice1]

This is certainly going to need careful scrutiny before it goes in.

I am not sure that the way that the apply_reduced_scoring_adjustment method changes the database records is a good approach.

I haven't looked particularly close at this yet though.

[somiaj]

@drgrice1 I think it will be useful to save the raw grade in the database, which currently isn't done. To try to clarify the change.

Current method: status = adjusted grade, sub_status = grade earned before the reduced scoring date. This was so the reduced scoring only applied to anything earned after the reduced scoring date.

Suggested change: status = adjusted grade (stays the same), sub_status = raw score. I then recover the grade earned before the reduced scoring date (vs storing it). Works provided reducedScoringValue doesn't change.

So the change just stores the raw score so it can be used other places which will be helpful to determine conditional release or when a problem is completed. It could be possible to compute the raw score from the curent sub_status and reducedScoringValue, I just think overall having the raw score being stored will be more beneficial than scoring just the portion earned before the reduced scoring date.

[drgrice1]

This pull request will need to be completely redone. The current formula that you have implemented is in no way correct. Basically, it is possible with this formula for a student to always increase their grade as long as they have submissions available. The limit of that possible grade is 1 with this formula. That limit can not be achieved, but in just a few iterations it can be made pretty close to that, and can easily exceed the reduced credit the student is currently allowed.

For a simple test to see that this is the case, create an assignment that only has one problem. To make this easy use the attached problem AlwaysCorrect.pg.txt. This problem is always correct not matter what you enter (even if you enter nothing).

Now enable reduced scoring for the course and the set, and set the open date and reduced scoring date to a date before today, and the due date to a date after today. So the students will all have a status and sub_status of 0, and will be in the reduced scoring period from the start. Also set the reduced scoring value to 0.75 (the default).

Now go to this problem in the assignment and submit any answer.
Your implementation will set the status to 0.75 and the sub_status to 1.
Now submit any answer again. Now the status is 0.91667 and the sub_status of course is still 1 and it will never change again.
Submit again. Now the status is 0.972222.
Submit again. Status is 0.990741.
Submit again. Status is 0.996914.
Note that these are the values seen in the database for this user, but you can see the rounded results on the grades page (or elsewhere).

Of course if you start this test all over with the current develop branch and the current reduced scoring implementation, you see that after the first attempt the status is set to 0.75 and never changes again, and the sub_status never changes from 0.

[somiaj]

Thanks for catching this. I was testing with 50% reduced scoring, and it all worked, I needed to divide by (1 - reducedScoringValue), not reducedScoringValue, but at 50% it didn't matter. I'm not able to increase the score now by just submitting again and again.

[drgrice1]

I have left some comments. This is just a start on what will need to be done here to get this ready to go in.

[somiaj]

I have drastically cleaned things up and changed how and when the reduced scoring is applied thanks to @drgrice1's comments. My tests seem to be positive so far, but I do think the major down side of this approach is that the reduced scoring computation may not be correctly inverted if the reduced scoring value is changed or grades are manually changed.

So this leaves a few key questions:

Is it important to save the raw score in the database? There have been various reasons to support this mentioned in openwebwork/pg#648 (comment).

If it is important to save the raw score, should another column be added to the database? This approach doesn't add another column, but there will be some edge cases that the before/after reduced scoring portion can no longer be correctly computed.

There might also be other options I have not thought of to store the raw score in the database without adding more columns.

At this point is won't take long to change this to how reduced scoring was originally computed, I was just trying to explore the possibility of using sub_status for the raw score.

[drgrice1]

Okay, so here is your next rework. This approach of saving the pg score in the database is just not going to reliably work. Adding another column to the database simply isn't needed either.

Here is what is needed. This should be redone to simply move the pg code that performed the reduced scoring to the webwork2 side, but using the same formulas as that code and using the same meaning of the values in the database. The thing is that that will actually work to fix all issues that were brought up in openwebwork/pg#648. It will take a bit more care in some cases, but is the only reliable way to have the score before the reduced scoring period begins. Furthermore, manual grading of problems will not mess it up. Instead of unreliably trying to recover the score before the reduced scoring period begins from the currently saved pg result score and reduced score, you recover the last pg result score from the currently save reduced score. That is much more reliable, and failure is not as critical as the places where this is needed as requested by @Alex-Jordan in openwebwork/pg#648 are only UI display things. This is also much simpler for those issues because you really don't need the actual pg result score, but you just need to know how the saved score compares to the maximum possible score in the reduced scoring period, and this is easily computed with the current information in the database.

Perhaps you should have heeded what @pstaabp said in openwebwork/pg#648 (comment). He told me to hold off on this, and you probably should have too.

[drgrice1]

I think I need to reword the beginning of that last comment. Instead of calling it your next rework, I think what needs to be said here is put this pull request on hold a bit until further discussion can be had.

[Alex-Jordan]

Some references above are pointing to an old PR in webwork2. The pg issue is here:
openwebwork/pg#648

I am about to go add an item to the list.

[drgrice1]

Thanks @Alex-Jordan. I forgot to prefix those with the pg repository. That works if the issue is from the same repository, but that one is not.

[somiaj]

This approach won't work in the long run. Closing.