Skip to end of metadata
Go to start of metadata

Overview

The test harness in it's current format is run as an Ant project, with the required LAMS sequences residing in the same directory. Other than that, the only requirement is the lams server. There are 5 configuration files governing the operation of creating users/courses, author accounts, monitor accounts, and learner accounts.

Download and run

shell> cvs -d :pserver:anonymous@code.lamsfoundation.org:/usr/local/cvsroot export -Dnow TestHarness4LAMS2
shell> cd TestHarness4LAMS2
shell> ant run

That's the quick guide. You'll probably want to edit the configuration files.

Operations

Individual settings are explained in the 5 properties files themselves, so the following will simply discuss some high level issues. Refer to the comments in the properties files for specific instructions.

Generally the test harness is limited to the following order: first it will optionally create a course and users first, then proceed to import the chosen sequence, then mix requests by the learners and monitors to the server according to min and max time delays.

The admin part can create users en masse, as well as courses as required. Since this isn't a common operation in practise, it's best to do a run of the test harness which creates the users and course you will need, then reconfigure the test harness to do the 'real' run of learners and monitors hitting the server. You can configure multiple test run configurations by suffixing a different number to each file; see test.properties for details.

Not all tools are supported; the ones that are are forum, noticeboard, notebook, multiple choice, Q&A, grouping.

Common settings

Number of learners

In learnerTest1.properties the NumberOfLearners setting will set the number of learners you want to use/create in the test harness.

We recommend a maximum of 100 users per Test Harness run. If you need to simulate more than 100 users, we suggest you use a second desktop/laptop to run the test harness from.

The 100 maximum is because we have detected that an average desktop has troubles handling more than this number. In effect, this means that your desktop is running 100 browser instances at once which can be a bit heavy.

Nevertheless, you can run the test harness from many desktops at the same time.

Minimum and maximum delays

There are several .properties files that you can configure. However, one of the most important settings is the minimum and maximum delays for each request to LAMS. Therefore we suggest you try to be as consistent with these delays. Specially for the learnerTest1.properties as these will simulate how long students take to perform an action (answer a question, post a message in a forum, etc).

We usually set these for learner to:

#------------------------------------------------------------                                                                
# Set delays (in secounds) which are the amount of idle time between calls to the server                                     
# Delay is used to emulate the time users spent on going to toilet during composing sequences ;-)                          
# delay = MinDelay*1000 + random.nextInt((MaxDelay-MinDelay+1)*1000)                                                         
# If not specified, 0 will be used for both.                                                                                 
#------------------------------------------------------------                                                                
MinDelay = 15                                                                                                               
MaxDelay = 90  

A minimum delay of 15 and a max of 90 means that each of our simulated students will take a minimum of 15 seconds to perform an action and a maximum of 90 seconds. Each simulated users will get a value between 15 to 90 seconds at random.

For monitor, we suggest you pick a min of 45 and a max of 120 seconds.

LAMS 2.4+ and test harness

When using the test harness with LAMS version 2.4+, ensure you the system configuration setting "Force mobile devices to use learner interface without Flash" is set to false. See LDEV-2919 for further details.

LAMS 2.4.1+ and test harness

We have introduced a new feature that enforces consistent username, names and email. As the Test harness creates random names with special characters, these would fail so you need to make sure you turn these options OFF in the sysadmin menu (Edit config settings):

  • Enforce properly formatted emails
  • Enforce first and last name validation
  • Enforce username validation

Reuse the same users

You probably don't want to recreate a whole stack of users every time you run the test harness, so the best thing to do is to reuse the same users you created (with the test harness) before.

To do so, you will need to make a few changes to the .properties files:

authorTest1.properties

#-----------------------------------------------------------------                                                           
# This LearningDesignFile is uploaded to import a Learning Design                                                            
# It could be relative path or absolute path or any valid URL                                                                
#----------------------------------------------------------------- 
# LearningDesignFile = 20nbs.zip                                                                                              

#-----------------------------------------------------------------                                                           
# If LearningDesignId is set,                                                                                                
# All the settings above will be ignored                                                                                     
# This property is used to test against an existing learning design                                                          
#-----------------------------------------------------------------                                                           
LearningDesignId = 1

Where LearningDesignId is an actual learning design ID from the the lams_learning_design table. Note that the LearningDesignFile has been commented out!

adminTest1.properties

#-----------------------------------------------------------------                                                           
# If CourseId is set,                                                                                                        
# All the settings regarding course creation will be ignored                                                                 
# This property is used to test against an existing course                                                                   
#-----------------------------------------------------------------                                                           
CourseId = 8

#-----------------------------------------------------------------                                                           
# If UserCreated is set to true, all the settings regarding user creation will be ignored                                    
# If not specified, the value of the property will be false.                                                                 
# If CourseId is not set, this value will be always considered false                                                         
#-----------------------------------------------------------------                                                           
UserCreated = true

In adminTest1.properties, you will need to give the value for the CourseId you've been using in your previous test (see the _lams_organisation to get the appropriate id.

Additionally, set the UserCreated setting to true so the Test Harness doesn't attempt to recreate the users.

monitorTest1.properties

#------------------------------------------------------------------                                                          
# Set the user id of the monitor.                                                                                            
# It should be set if and only if LessonId is not set and UserCreated is true in adminTest                                   
#------------------------------------------------------------------                                                          
UserId = 10

For monitorTest1.properties, you need to find out what the user for monitoring your lesson is. To do this, you can login into
your LAMS server as sysadmin, go to the sysadmin menu and use "Find a user". Then search for a "monitor" and you'll see a user called something like XXXYYYZZ_monitor, get the ID for this user and enter it in the UserId setting as shown above.

learnerTest1.properties

#------------------------------------------------------------------                                                          
# Set the number of learners to be created or updated for this test                                                          
# If it's not specified, 1 will be used                                                                                      
#------------------------------------------------------------------                                                          
NumberOfLearners = 15

...                                                                                               

#------------------------------------------------------------------                                                          
# UserIdOffSet set the start userId. It should be set when                                                                   
# UserCreated set to be true in AdminTest while LessonId is not set                                                          
# in MonitorTest                                                                                                             
#------------------------------------------------------------------                                                          
UserIdOffset = 11

The NumberOfLearners is as we mentioned above the total number of learners that we going to use in an test harness run. If you have create a 100 users in your first run, for your second run you can reduce the figure if you like (for instance only 50 users).

The UserIdOffset is very important to set up right. It indicates what the user_id is for the first learner. So look into your lams_user table and find the first Learner1 user created by the test harness. For instance:

mysql> select user_id, login from lams_user order by user_id;
+---------+----------------------+ 
| user_id | login | 
+---------+----------------------+ 
| 1 | sysadmin | 
| 2 | test | 
| 3 | lamskh01 | 
| 4 | mmm | 
| 5 | test1 | 
| 6 | test2 | 
| 7 | test3 | 
| 8 | test4 | 
| 9 | aptop.local_1_Author | 
| 10 | ptop.local_1_Monitor | 
| 11 | top.local_1_Learner1 | 
| 12 | top.local_1_Learner2 | 
| 13 | top.local_1_Learner3 | 
| 14 | top.local_1_Learner4 | 
| 15 | top.local_1_Learner5 | 
| 16 | top.local_1_Learner6 | 
| 17 | top.local_1_Learner7 | 
| 18 | top.local_1_Learner8 | 
| 19 | top.local_1_Learner9 | 
| 20 | op.local_1_Learner10 | 
+---------+----------------------+ 
20 rows in set (0.00 sec) 

Here you'll see that the user_id for our first learner created by the test harness is:

| 11 | top.local_1_Learner1 | 

So the UserIdOffset must be set to 11.

How to interpret the Test Harness results

When a test harness run finishes you will get an output like this:

     [java] **********************************Brief Report*******************************************
     [java] * 
     [java] * Disclaimer:
     [java] *   This program is created in the hope that it will help estimate how many concurrent
     [java] *   users a LAMS 2.x server can handle, but WITHOUT ANY GARANTEE  the server can support
     [java] *   that number of users in service use.
     [java] * 
     [java] *   This program is more a load test tool than a functional test tool, 
     [java] *   so it does NOT GARANTEE there is no functional bug in the target server.
     [java] * 
     [java] * Test Result Summary:
     [java] *   1 test suite(s) launched. 1 test suite(s) finished, and 0 test suite(s) aborted.
     [java] *   Test Suite 1 finished, in which
     [java] *   adminTest1 finished, authorTest1 finished, monitorTest1 finished, learnerTest1 finished
     [java] *   In learnerTest1, 100 learner(s) attended, 100 finished and 0 aborted.
     [java] * 
     [java] * Refer to the formal test report document for the details.
     [java] * 
     [java] *****************************************************************************************
     [java] 
     [java] 10:23:13,607 INFO [main] TestReporter - Generating the formal test report document...
     [java] 10:23:13,609 INFO [main] TestReporter - Total response time: 8106.001 seconds
     [java] 10:23:13,609 INFO [main] TestReporter - Average response time: 1.1027072507141886 seconds
     [java] 10:23:13,609 INFO [main] TestReporter - Average response time of joinLesson calls: 0.45319000000000004 seconds
     [java] 10:23:14,039 INFO [main] Main - It's done, anyway

The main result here is the Average response time. This means how long LAMS took to return a page after a request. In this example (which is based on Clustering for 2000 concurrent users), this means that when there's 2000 learners in LAMS at the same time, the average time for LAMS to return a page was 1.1 seconds.

Take into account that if you are running the test harness from Australia and your LAMS server is in Canada, then there will be latency in your response. The test harness cannot tell you whether this is due to network latency or actual server response. So we suggest you run the test harness from the same network as your server whenever possible.