Loadrunner is perhaps the oldest of these three testing tools that I am going to cover. It was created by Mercury Interactive in 1994, sold to Hewlett Packard in 2006 and eventually resold to Micro Focus in 2016. This software tool supports quite a few different protocols.
- .Net
- Citrix ica
- DNS
- Flex
- FTP
- IMAP
- Java over http
- LDAP
- MAPI
- MQTT
- ODBC
- POP3
- RDP
- RTE
- Sap web
- SMTP
- Truclient
- Web http/html
- Web services
Loadrunner is an integrated suite of programs which let you program your test. Three different programs in this suite mirror the process.
Virtual User Generator – develop a test
Controller – run the test and gather statistics
Analysis – reporting on test statistics
Virtual User Generator
The “VuGen” program reminds me a bit of Arduino IDE. Because you are not creating your own stand alone program there is essentially a harness that you are fitting your tests into. Each new Test program which is referred to as a “Solution” in LoadRunner has an initialization and clean up routing. You are then able to add as many Actions (source files) as you see fit. Loadrunner has its own main routine for the program and so you create your method and it is run in the order listed under Run in the Run Logic Tree.
One strong point for LoadRunner is that you can define what code or tasks is a step. LoadRunner has its own API and so defining a “Transaction” is just a matter of calling the lr_start_transaction at the beginning of process and closing off the transaction when it is finished with the lr_end_tranaction. A Transaction can include a single REST API call, multiple REST calls or as much code as you wish to include for that step. Most of the steps that are included in a Transaction is either the REST or HTML calls or the necessary code for generating or saving values that are part of the responses.
One of the quirks of creating a script in LoadRunner is that if you wish to retrieve a variable from a returned page of HTML you need to ask for it before the call. This may have made the process easier for the original developers but it does take a while to get used to this odd ordering of instructions in the source code.
The source code is standard C. Depending on your language preferences you may find this encouraging or depressing but the good news is that there is an integrated debugger. Just like other modern IDE’s it is possible to set breakpoints and examine variables. The log output is a monologue what was done or called along with the responses. This information is displayed with the line and source file so it is possible to examine the output at your leisure.
Considering the nature of Load testing, it should not be too surprising that LoadRunner has a built in system for defining lists of values that should be available as parameters for your REST or Web calls. LoadRunner provides a dialog to create and maintain such lists of values but the values themselves are written out to a csv file. Once the mapping is performed in loadrunner it may be easier to simply edit these csv files in a standard editor.
LoadRunner also provides a recorder. This will allow you to visit a website (ie HTML Testing) and go through a series of steps. It is quite likely that this recorded script will need to be amended as many websites have logic in place against replay attacks, but recording is a nice way to get at least small jump on the task.
The general development of the script may involve running it and looking at the log output or it may be debugging it line by line. But eventually you will have a working script which can be used for the actual load testing. Once you are at this step you use the second tool from LoadRunner – Controller.
Controller
There is actually a surprising amount of possible configuration possible in the Controller but the main task of this program is to run any test or tests and to capture the results. The most used settings run time and number of virtual users. You decide how many users to use and how long it should take to start that many users. (ie 10 every 2 seconds) Once that number of users has been reached LoadRunner will maintain that number for the duration of the run.
Actually there is another interesting feature. It is possible to have other computers assist in the load generation process. These other computers are called LoadGenerators. When they are used it is to support the main machine in generating the necessary load but without exceeding the number of licensed users. It is usually necessary to have more than one load generator if you are trying to really push your system test. I am not clear if this is because of the raw cpu that is necessary to run thousands of virtual users or if this is due to a operating system limitation (ie number of theads).
As you can see there are quite a number of different graphs that can be used. These graphs are some of the values that can be gathered and displayed immediately but all values will be saved and used by the third LoadRunner program – Analysis.
Analysis
The analysis program tends to be a little anti-climactic. The testing script is create and debugged using the Virtual User Generator program, and it is tested in the Controller. The controller will even show some of the available graphs in real time.
The actual analysis phase is simply picking which graphs you wish to look at and displaying them. It is possible to create a report and save it as a PDF/RTF/Excel/JPG/… The graphs are so helpful but somewhat difficult to read in the PDF report yet there is another way. It is also possible to create a HTML report instead of saving a normal report data as HTML. This will create a sub-directory and store all the necessary data, html and pictures in it. This “report” can then be viewed in your favorite web browser. That entire directory can be zipped up and emailed to colleagues who can then look at the report. This is perhaps a bit clumsy but the graphs are then very legible. Not only that but for some of the graphs produced it is possible to deselect various information being graphed to allow you to focus on one aspect or another of the test.
Yet, there is one more little gem in the analysis tool. It is possible to create Service Level Agreement templates. These templates define what the expected behavior (i.e. response times) should be. This template is color coded and can be copied between test directories. In this way it is possible to open up this SLA agreement summary after a test and quickly and visually see (green or red) if any of the results are not as quick as expected or required.
Summary
The LoadRunner tool from MicroFocus is a comfortable tool to use. It is easy to create, debug, run and graph all output. I understand that MicroFocus’s goal is indeed to receive income from this tool but most certainly for a small shop this tool might be too expensive to consider.
I am not going to go into too much depth about this program because of the pricing structure. If you are an enterprise and have a budget for software, license costs, personnel and so forth than the pricing of LoadRunner may not as important. Unfortunately for the rest of us money may be an issue. LoadRunner has its pricing based on the number of users that you will be using to generate your load. The more users you use the higher the cost. Unfortunately there is no real fixed price sheet that I can use for analysis but there are a few web resources that do give some hints.
https://aws.amazon.com/marketplace/pp/Micro-Focus-Micro-Focus-LoadRunner-1000-Virtual-Us/B01K3PEB7O
https://www.microfocus.com/de-de/products/loadrunner-load-testing/pricing
If these examples are correct, then the cost for 1000 virtual users is 500 USD per hour from Amazon as a service or 3500 USD for 1000 virtual user days from Micro Focus. I seem to remember that there were other restrictions of the permanent license. If memory serves it was limited to a specific piece of hardware in a specific location. Furthermore it could only be used by users physically at the same location.
https://www.jds.net.au/loadrunner-licensing/
https://www.microfocus.com/media/white-paper
loadrunner_to_performance_center_migration_wp.pdf
Don’t lose hope if you really desire to write your tests in C and to use LoadRunner to do it. It is possible to download a free version of LoadRunner that will support 50 virtual users and can be used for all protocols except for a handful. Load testing with 50 users per second it not quite Google or Facebook territory but this might be a good test for an internal site or perhaps a website that supporting s a small business or sports club.
There are certainly other proprietary tools in the market place but the next tool – Gatling – is actually an open source tool.
