CFEngine API

org.recommender.algorithms
Class AutoTester

java.lang.Object
  |
  +--org.recommender.algorithms.AutoTester

public class AutoTester
extends java.lang.Object

A class containing test code to be run immediately upon server loading. Evaluates the effectiveness of an implemented algorithm using metrics defined in TestStats. Includes a rating test for generic training/test splits and an all-but-one test, for comparing to published papers that use all-but-one tests. Testing for top-N recommendations is in progress, but doesn't yet work very well so it has been commented out. Note: to run the autotest mode, we need a different table format from the server mode: autotest mode need four columns in the RATING_TABLE: UserID, ItemID, Rating and RatingSet.

Author:
Daniel Lowd <dlowd@cs.orst.edu>, Shuzhen Nong <nong@cs.orst.edu>

Field Summary
(package private) static int givenNparam
           
(package private) static int NUM_RECS
           
(package private) static int numParam
          Test the quality of an algorithm's Top N recommendations.
(package private) static int reduceSetOpt
           
 
Constructor Summary
AutoTester(DataManager dataMgr, java.lang.Class algorithm)
          Construct an AutoTester object to test the provided algorithm using data from the provided data manager.
 
Method Summary
protected  void addCachedRatings(java.util.Iterator ratings)
          Add ratings to cache.
 void allButOne(java.util.Iterator ratings)
          Run an all-but-one test, given an iterator of ratings representing the test set.
protected  void removeCachedRatings(java.util.Iterator ratings)
          Remove ratings from cache
static void runTests(DataManager dataMgr, java.lang.Class algorithm, java.lang.String whereClause)
          Run automated tests and return the results using the provided algorithm, data manager.
protected  void testCachedRatings(java.util.Iterator ratings, TestStats testStats)
          Run test for one user - the one just added in cache, and add results to testStats.
 void testRatings(java.util.Iterator iter)
          Run a training/test split rating test, given an iterator of ratings representing the test set.
 void testRecommendations2(java.util.Iterator iter1)
           
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

numParam

static final int numParam
Test the quality of an algorithm's Top N recommendations.

See Also:
Constant Field Values

reduceSetOpt

static final int reduceSetOpt
See Also:
Constant Field Values

givenNparam

static final int givenNparam
See Also:
Constant Field Values

NUM_RECS

static final int NUM_RECS
See Also:
Constant Field Values
Constructor Detail

AutoTester

public AutoTester(DataManager dataMgr,
                  java.lang.Class algorithm)
Construct an AutoTester object to test the provided algorithm using data from the provided data manager.

Parameters:
dataMgr - data manager to provide ratings and other support services.
algorithm - algorithm to test
Method Detail

runTests

public static void runTests(DataManager dataMgr,
                            java.lang.Class algorithm,
                            java.lang.String whereClause)
Run automated tests and return the results using the provided algorithm, data manager.

Parameters:
dataMgr - data manager to provide ratings and other support services.
algorithm - algorithm to test
whereClause - place to load the testset

allButOne

public void allButOne(java.util.Iterator ratings)
Run an all-but-one test, given an iterator of ratings representing the test set. All ratings for any given user must be in a single block for this to work properly. That is, ratings must be grouped by UserID. In testing predictions, this method loads all ratings in the test set from a given user, then removes each one temporarily for predicting it. Ratings for that user that are already cached may be used by the algorithms as training data, but are not removed or predicted. When all the user's ratings in the test set have been predicted, these ratings are removed and the next user is tested. The overall Mean Average Error (MAE) is printed out after every 1000 predictions. When testing is complete, more thorough results are displayed to System.out in a format determined by TestStats.

Parameters:
ratings - Iterator of all ItemRatings in the test set, grouped by UserID.

addCachedRatings

protected void addCachedRatings(java.util.Iterator ratings)
Add ratings to cache.

Parameters:
ratings - the tested user's iter of ItemRating

removeCachedRatings

protected void removeCachedRatings(java.util.Iterator ratings)
Remove ratings from cache

Parameters:
ratings - the tested user's iter of ItemRating

testCachedRatings

protected void testCachedRatings(java.util.Iterator ratings,
                                 TestStats testStats)
Run test for one user - the one just added in cache, and add results to testStats.

Parameters:
ratings - the tested user's ItemRating
testStats - place to put test results

testRatings

public void testRatings(java.util.Iterator iter)
Run a training/test split rating test, given an iterator of ratings representing the test set. It is presumed that all ratings in the training set are already loaded into the data manager. All ratings for any given user should be in a single block for some metrics to work properly. That is, ratings ought to be grouped by UserID. This allows data to be collected on the average mean error per user, in addition to the overall mean rating error. This method tests ratings from the test set one at a time. If a rating in the test set is for a user with fewer than 5 ratings loaded by the data manager, then the rating is skipped. None of the ratings in the test set are ever added to the database to be used as training data. Thus, it is important that the test set consist of additional ratings for users who are also in the training set. The overall Mean Average Error (MAE) is printed out after every 1000 predictions. When testing is complete, more thorough results are displayed to System.out in a format determined by TestStats.

Parameters:
iter - Iterator of all ItemRatings in the test set, grouped by UserID.

testRecommendations2

public void testRecommendations2(java.util.Iterator iter1)

CFEngine API

Copyright © 2003 - Oregon State University www.orst.edu