The test harness in it's current format is run as an Ant project, with the required LAMS sequences residing in the same directory. Other than that, the only requirement is the lams server. There are 5 configuration files governing the operation of creating users/courses, author accounts, monitor accounts, and learner accounts.
Download and run
That's the quick guide. You'll probably want to edit the configuration files.
Individual settings are explained in the 5 properties files themselves, so the following will simply discuss some high level issues. Refer to the comments in the properties files for specific instructions.
Generally the test harness is limited to the following order: first it will optionally create a course and users first, then proceed to import the chosen sequence, then mix requests by the learners and monitors to the server according to min and max time delays.
The admin part can create users en masse, as well as courses as required. Since this isn't a common operation in practise, it's best to do a run of the test harness which creates the users and course you will need, then reconfigure the test harness to do the 'real' run of learners and monitors hitting the server. You can configure multiple test run configurations by suffixing a different number to each file; see test.properties for details.
Not all tools are supported; the ones that are are forum, noticeboard, notebook, multiple choice, Q&A, grouping.
Number of learners
In learnerTest1.properties the NumberOfLearners setting will set the number of learners you want to use/create in the test harness.
We recommend a maximum of 100 users per Test Harness run. If you need to simulate more than 100 users, we suggest you use a second desktop/laptop to run the test harness from.
The 100 maximum is because we have detected that an average desktop has troubles handling more than this number. In effect, this means that your desktop is running 100 browser instances at once which can be a bit heavy.
Nevertheless, you can run the test harness from many desktops at the same time.
Minimum and maximum delays
There are several .properties files that you can configure. However, one of the most important settings is the minimum and maximum delays for each request to LAMS. Therefore we suggest you try to be as consistent with these delays. Specially for the learnerTest1.properties as these will simulate how long students take to perform an action (answer a question, post a message in a forum, etc).
We usually set these for learner to:
A minimum delay of 15 and a max of 90 means that each of our simulated students will take a minimum of 15 seconds to perform an action and a maximum of 90 seconds. Each simulated users will get a value between 15 to 90 seconds at random.
For monitor, we suggest you pick a min of 45 and a max of 120 seconds.
LAMS 2.4+ and test harness
When using the test harness with LAMS version 2.4+, ensure you the system configuration setting "Force mobile devices to use learner interface without Flash" is set to false. See LDEV-2919 for further details.
LAMS 2.4.1+ and test harness
We have introduced a new feature that enforces consistent username, names and email. As the Test harness creates random names with special characters, these would fail so you need to make sure you turn these options OFF in the sysadmin menu (Edit config settings):
- Enforce properly formatted emails
- Enforce first and last name validation
- Enforce username validation
Reuse the same users
You probably don't want to recreate a whole stack of users every time you run the test harness, so the best thing to do is to reuse the same users you created (with the test harness) before.
To do so, you will need to make a few changes to the .properties files:
Where LearningDesignId is an actual learning design ID from the the lams_learning_design table. Note that the LearningDesignFile has been commented out!
In adminTest1.properties, you will need to give the value for the CourseId you've been using in your previous test (see the _lams_organisation to get the appropriate id.
Additionally, set the UserCreated setting to true so the Test Harness doesn't attempt to recreate the users.
For monitorTest1.properties, you need to find out what the user for monitoring your lesson is. To do this, you can login into
your LAMS server as sysadmin, go to the sysadmin menu and use "Find a user". Then search for a "monitor" and you'll see a user called something like XXXYYYZZ_monitor, get the ID for this user and enter it in the UserId setting as shown above.
The NumberOfLearners is as we mentioned above the total number of learners that we going to use in an test harness run. If you have create a 100 users in your first run, for your second run you can reduce the figure if you like (for instance only 50 users).
The UserIdOffset is very important to set up right. It indicates what the user_id is for the first learner. So look into your lams_user table and find the first Learner1 user created by the test harness. For instance:
Here you'll see that the user_id for our first learner created by the test harness is:
So the UserIdOffset must be set to 11.
How to interpret the Test Harness results
When a test harness run finishes you will get an output like this:
The main result here is the Average response time. This means how long LAMS took to return a page after a request. In this example (which is based on Clustering for 2000 concurrent users), this means that when there's 2000 learners in LAMS at the same time, the average time for LAMS to return a page was 1.1 seconds.
Take into account that if you are running the test harness from Australia and your LAMS server is in Canada, then there will be latency in your response. The test harness cannot tell you whether this is due to network latency or actual server response. So we suggest you run the test harness from the same network as your server whenever possible.