LOAD TESTING, BENCHMARKING, AND APPLICATION PERFORMANCE MANAGEMENT FOR THE WEB

Published in the 2002 Computer Measurement Group (CMG) Conference, Reno, NV, Dec. 2002.
LOAD TESTING, BENCHMARKING, AND APPLICATION PERFORMANCE
MANAGEMENT FOR THE WEB

Daniel A. Menascé
Department of Computer Science and E-center of E-Business
George Mason University
Fairfax, VA 22030-4444
menasce@cs.gmu.edu

Web-based applications are becoming mission-critical for many organizations and their
performance has to be closely watched. This paper discusses three important activities in
this context: load testing, benchmarking, and application performance management. Best
practices for each of these activities are identified. The paper also explains how basic
performance results can be used to increase the efficiency of load testing procedures.

infrastructure depends on the traffic it expects to see
1. Introduction
at its site. One needs to spend enough but no more
than is required in the IT infrastructure. Besides,
Web-based applications are becoming mission-
resources need to be spent where they will generate
critical to most private and governmental
the most benefit. For example, one should not
organizations. The ever-increasing number of
upgrade the Web servers if most of the delay is in
computers connected to the Internet and the fast
the database server. So, in order to maximize the
growing number of Web-enabled wireless devices
ROI, one needs to know when and how to upgrade
create incentives for companies to invest in Web-
the IT infrastructure. In other words, not spending at
based infrastructures and the associated personnel
the right time and spending at the wrong place will
to maintain them. By establishing a Web presence, a
reduce the cost-benefit of the investment. In this
business has the potential of reaching out to a very
paper, we provide a framework, based on
large number of customers. Of course, they need to
benchmarking, testing, and application performance
know that a site exists, the site has to earn their
management, which allows companies to maximize
trust, and customers need to have a good
the ROI on their IT expenditures.
experience when they visit the site. Customers
should be able to easily and in a timely manner find
Section two defines in more detail the important QoS
what they want. A Web site should be always there
metrics to be considered. The next section presents
when customers need it.
a multi-layer reference model to analyze Web-based
applications. Section four describes the main
A critical question is what is the Return on
features of benchmarking, testing, and application
Investment (ROI) of Web-based applications? There
performance testing. Section five presents some
are several important factors to take into account in
important requirements of tools used in these three
order to answer this question. First, offering services
activities. Section seven provides some basic
and products over the Web cuts down some costs in
performance concepts that should be taken into
personnel and physical infrastructure that would be
account when carrying out load testing. Finally,
devoted to direct customer assistance, either in
section seven presents some concluding remarks.
person or over the telephone. On the other hand, to
provide online access to products and services, a
company needs an adequately sized IT
2. Web-Based Application QoS
infrastructure that provides the services needed by
Quality of Service (QoS) is a key concept in
its customers with the quality of service (QoS) they
assessing how well Web-based applications deliver
expect.
what customers expect in terms of response time
QoS for Web applications is usually measured in
and availability.
Availability measures the
terms of response time and availability from its
percentage of time a Web-based application is
user’s point of view and in terms of throughput from
available to customers. Typical availability goals
the site management standpoint. Poor QoS
may vary according to the type of application. Some
translates into frustrated customers, which translates
critical applications, such as on line brokerage, may
into loss of business opportunity. The amount of
have more stringent availability requirements from
money a business needs to spend in its IT
the user perspective than other applications, such as

1

---

Published in the 2002 Computer Measurement Group (CMG) Conference, Reno, NV, Dec. 2002.
online travel sites. Availability requirements may also
site, and the delays imposed by third-party services
vary according to the time of day or according to
used by the Web site. Examples of third-party
special events. For example, during times of high
services include Content Delivery Networks (CDNs)
market volatility, it is essential for an online
[Buzen2002] used to provide images and streaming
brokerage site to exhibit an availability as close to
media to customers or ad networks that provide ad
100% as possible. An online ticket sales service has
banners to the pages served by a site. So,
to exhibit very high availability when tickets for a
measuring response time from a single geographical
sporting event or for a concert go on sale.
location and at a specific time window does not
Unfortunately, these are the times, when Web sites
provide the complete picture on a Web site’s
are subject to “flash crowds” and their resources are
response time. End-to-end response time is time-
stressed to their limits, which may cause user
and space-dependent. This means that one needs
requests to be rejected due to lack of enough
to be able to know how users from different
resources, thus decreasing the availability. Even in
locations, with different connectivity, and at different
industries in which availability is not as critical from
times of the day, perceive a Web site. For instance,
the user’s perspective —e.g., e-tailers as opposed to
according to recent measurements by Keynote, the
online brokerage sites—availability is crucial to the
site of the International Olympics Committee
bottom line of the business.
delivered variable response time for users in
different parts of the globe, ranging from over 9.0
The availability of a Web or E-commerce site is not
seconds for users in Asia to 3.97 for users in
the same for all its customers. In fact, the
Europe, on average, from January 21 to February 11
geographical location of a customer may impact the
(www.keynote.com).
availability of a Web application given that different
customers may have their traffic served by different
Let us now zoom in into the end-to-end response
ISPs and different networks. For example, recent
time and break it down into its components
measurements by Keynote of Olympic.org, the site
[Zhi2001]. Suppose you are running a travel site and
of the International Olympics Committee, showed
a customer submits a search for flights for a given
99.33% availability overall internationally, 98.9% for
itinerary. The result of the search is presented to the
users from Asia, and 99.48% for European users
user in 4.03 seconds. What are the main
(www.keynote.com).
components of the 4.03 seconds? Let us divide the
end-to-end response time into the following
Another key factor in determining the quality of
components: i) DNS lookup time (i.e., time to convert
service offered by Web and E-commerce sites is the
your site human-readable name—say
response time. But, what is response time anyway?
www.mytravelsite.com—into an IP address, ii) time
How should it be measured? What are its main
required for the browser to establish an initial TCP
components? These are important questions to
connection with a server at your site, iii) redirection
consider since they affect the choice of
time, if your site, redirects requests to another site,
measurement procedures and tools.
iv) time to download the first byte of the base page
Let us start with the definition of response time. In
requested, iv) time to download the entire base
Web-based environments, we need to measure end-
page, and v) time to download the content of the
to-end response time. In other words, we need to
page with all its embedded objects, such as images
measure response time outside the firewall. This
and ad banners. Some of the components of the
type of measurement indicates
customers’
end-to-end response time are only network related
perception of the time it takes a Web page to
(e.g., TCP connect time), while others involve both
download or the time elapsed for a keyword search
the network and several resources within your site
to return the results. When defining end-to-end
(e.g., firewalls, load balancers, Web serve rs,
response time, one must distinguish between the
application servers, database servers, and internal
time it takes for the base HTML page to download
networks).
and the time for the other components of the page
Table 1 provides an example of a possible
(e.g., images, ad banners) to download.
breakdown of the 4.03 seconds of our example
Customers’ perception of a Web application
search request. Note that each file's download starts
response time varies according to many different
with a DNS request (which may need zero time, if
factors that are outside a Web site’s environment,
that host name was previously requested), then a
and that affect the customers’ perception of
TCP Connection (assuming HTTP/1.0 or the use of
response time. These factors include: the Web site’s
a firewall), then the arrival of the first packet, then
ISP, the customers’ ISPs, the bandwidth of
the arrival of the rest of the data, which may use
customers’ connection to their ISPs, the networks
multiple packets transmitted without waiting for an
involved in routing packets from customers to the
acknowledgement from the receiver.

2

---

Published in the 2002 Computer Measurement Group (CMG) Conference, Reno, NV, Dec. 2002.
Table 1 – End-to-End Response Time Breakdown
Outside the Firewall
Inside the Firewall
Starting
Completion
External
Total
Web
Application Database
Total
Time
Network
Time
Server
Outside
Server
Server
Server
Inside
DNS Lookup
0.01
0.02
0.03
0
0.03
Initial TCP
Connection
0.08
0.08
0
0
0.08
First
First Packet
HTML File Download
0.08
0.08
0.02
0.06
1.4
1.48
1.56
Last Packet
0.4
0.4
0.1
0.1
0.5
TOTAL
0.57
0.02
0.59
0.12
0.06
1.4
1.58
2.17
DNS Lookup
2.2
0
0
0
2.2
Initial TCP
Connection
2.2
0.08
0.08
0
0
2.28
Image 1
First Packet
Download
2.2
0.08
0.08
0.02
0.02
2.3
Last Packet
2.2
0.15
0.15
0.06
0.06
2.41
TOTAL
2.2
0.31
0
0.31
0.08
0
0
0.08
2.59
DNS Lookup
3.59
0
0
0
3.59
Initial TCP
Connection
3.59
0.08
0.08
0
0
3.67
Image n
First Packet
Download
3.59
0.08
0.08
0.02
0.02
3.69
Last Packet
3.59
0.2
0.2
0.06
0.06
3.85
TOTAL
3.59
0.36
0
0.36
0.08
0
0
0.08
4.03
DNS Lookup
2.5
0.01
0.03
0.04
0
2.54
Initial TCP
External
Connection
2.5
0.03
0
0.03
0
2.53
CDN
First Packet
Server
Download
2.5
0.04
0.03
0.07
0
2.57
Last Packet
2.5
0.05
0.03
0.08
0
2.58
TOTAL
2.5
0.13
0.09
0.22
0
0
0
0
2.72

The table shows a breakdown of the five
and outside the firewall, at different load levels, from
components into outside the firewall components
different geographical locations, and at different
and inside the firewall components. The outside the
times.
firewall components include network (transmission,
DNS servers, and network round-trip times), Content
3. A Reference Model for Web and E-
Delivery Network, and Ad Network. The inside the
commerce Applications
firewall components include Web server, application
server, and database server. For example, the time
We use here the reference model for Web and E-
to download the first packet, 1.56 seconds, includes
commerce applications presented by Menascé and
0.08 of network time, 0.02 of Web server time, 0.06
Almeida in [MA00]. This model is used in the
seconds of application server time, and 1.4 seconds
subsequent sections to discuss a framework for
of database server time. Note that when the first
managing the QoS of Web-based applications. We
packet of the response page is sent, the site has
will use the example of a simplified online travel site
already processed the search request, which
to illustrate the concepts in the remaining sections of
includes accessing the database to locate the flights
this paper.
that satisfy the given itinerary. When the remaining
elements of the page are being downloaded, a CDN
is involved to supply images.
3.1 Business Model
The inside the firewall components of the response
The top level of the reference model is composed of
time depend on the load, measured for example in
elements related to the nature of the business, its
requests/sec, submitted by customers to the site. If a
revenue goals, its partnerships with other
site is experiencing a high volume of requests, the
businesses, its position within the industry. In our
level of contention for site resources (e.g.,
example, the business is an online site used to book
processors, storage devices, and LANs) will be
seats on flights operated by a large number of
much higher than when the load is light [MA02].
airlines and to sell cruises to islands in the
Caribbean. Our online travel site has established
This example clearly shows that one must be able to
partnerships with other businesses in order to get
measure the components of response time inside
access to their databases of flight schedules.

3

---

Published in the 2002 Computer Measurement Group (CMG) Conference, Reno, NV, Dec. 2002.
The business model specifies that revenue will come
surge in traffic? Changes in the business model may
from sales commissions on booked flights and cruise
also have an impact on the site’s IT infrastructure. If
packages and from the placement of advertisement
the site decides to offer hotel and car reservations in
from travel-related companies on the site pages.
addition to the previous services, the processing
The cruise section of the site is multimedia-rich and
load on the site will increase. Again, one has to
includes streaming video, delivered by a CDN. Ad
assess if this new type of load can be supported with
banners are delivered by an ad network.
acceptable QoS levels for both the existing and new
load.
The revenue goals for the site are a function of the
number of bookings/week, number of cruise
packages sold per week, and the number of ad
banner impressions per week. Based on the amount
3.2 Functional Model
of money brought to the business by each of these
At this level, we have to look at the various functions
activities, one can compute the revenue per week,
offered by a site as well as the kind of support
defined as revenue throughput [MA00]. Note that the
technologies (e.g., Flash, JavaScript, Active X,
revenue throughput depends on the number of
cookies, SSL) used to implement them. In our travel
completed transactions per unit time—the site
site example, the functions offered could be: Display
throughput—and on the number of displayed ads
the Home Page, Search for Flights, View Flight
per unit time. When a site has poor performance,
Reservations, Reserve a Flight, Book a Flight,
sessions tend to be abandoned by users, and the
Browse through Cruises, Show Cruise Video,
site loses revenue. Another relevant metric at the
Reserve a Cruise Package, and Book a Cruise
business level is the potential lost revenue
Package.
throughput [MA00], measured in dollars per unit time
For example, SSL is used for authentication in the
that are in your customers shopping carts but are
functions Reserve a Flight, Reserve a Cruise
not converted into sales.
Package, Book a Flight, and Book a Cruise
We need to be able to determine, at the business
Package. Streaming media is used in the Show
level, the QoS goals for the site, in terms of
Cruise Video function.
performance and availability, and how the site
should rank in terms of its competitors. There are
industry benchmarks, such as the Keynote indices
3.3 Customer Behavior Model
(e.g., Broker Trading, Business 40, Consumer 40,
The next layer down in the reference model
Government 40, Streaming, and Wireless SMS), that
describes how customers navigate through the site
you can use to establish comparisons with other
and what their characteristics are. Customers of a
sites in your industry. For example, the travel site
Web or E-commerce site interact with it through
may decide that its performance and availability
sessions, which are sequences of requests coming
goals are to be relatively close to Keynote’s
from the same user during the same visit to the site.
Consumer 40 Index, or they may choose to measure
The navigational pattern of a user can be captured
competitors directly, possibly doing transactions that
by a graph, called a Customer Behavior Model
are similar in function to the ones they measure on
Graph (CBMG), as shown in Fig. 1 [MARPFM00,
their own site.
MA00, MAFM99]. The picture shows the possible
There is a direct relationship between the QoS
states in which customers of our travel site can be
requirements establish by an e-business and the
found, along with the possible transitions between
amount of money spent in the IT infrastructure and
states and the probability that such transitions occur.
IT personnel to deliver that kind of service. The goal
We have only shown in Figure 1 the flight booking
should be to get the best possible performance by
aspect of the travel site.
spending as little money as possible to provide the
The states a customer can be found in Fig. 1 are:
service. At the business level, one has to evaluate
viewing the home page, searching for flights,
decisions regarding what to do in-house and what to
retrieving previously completed reservations, making
outsource.
a reservation, and booking a flight. The figure also
At this level level, one has to deal with issues such
shows that a customer that completes a search has
as marketing campaigns and changes in the
a 30% probability of submitting a new search, 6%
business model. Our travel site may decide to
probability of reserving a flight, 4% probability of
launch a marketing campaign to promote the site
booking a flight, and 60% probability of leaving the
close to Thanksgiving when many people travel. If
site after performing the search. Note that the arrows
successful, the campaign will attract a large number
in the graph that do not go to any other state are
of visitors to the site and increase the number of
actually going to the Exit state not shown explicitly.
transactions that have to be processed by the site

per unit time. Will the site be able to handle this

4

---

Published in the 2002 Computer Measurement Group (CMG) Conference, Reno, NV, Dec. 2002.
0.3
0.6
the customer? Well, it is quite likely that the
Search
customer will no longer be there to see the result.
Flight
The customer would have abandoned the session!
0.04
1.0
How long are customers willing to wait for a page to
download? The answer depends on the type of
0.7
Book
customer and on the type of page. An industry de
Flight
0.06
facto standard of 8 seconds has been used as a
1.0
Home
0.3
threshold after which customers become frustrated
Page
0.1
and abandon a Web site. While customers may be
0.1
Reserve
less willing to wait for the result of a search request,
0.2
Flight
they will tolerate longer waits when they are waiting
0.9
for their selected flight to be booked.
View
Reservation
0.2
3.4 IT Resource Level
0.5

At this level one must consider the several IT
Figure 1 – Simplified Customer Behavior
resources, including hardware (e.g., server boxes,
Model Graph (CBMG) for the Travel Site.
storage boxes, LAN segments, routers, and
firewalls), software (e.g., operating systems, web
servers, database management systems, ERP
Not all customers behave the same way when they
systems, transaction processing monitors,
visit a site. So, one must consider groups or clusters
performance monitoring tools, capacity planning
of sessions that represent customers with “similar”
tools), and personnel (e.g., programmers, system
behavior. Table 2 shows some examples of types of
analysts, system and network administrators, DBAs,
sessions. Each type of session can be described by
performance analysts, capacity planners, Web
a CBMG that represents a common behavior of a
content developers, and graphic designers).
group of users.
Table 2 – Examples of Types of Sessions
4. Three Key Activities in QoS Management
Home ⇒ Search ⇒ Exit
Managing the QoS of Web-based applications
Home ⇒ Search ⇒ Reserve Flight ⇒ Exit
requires a combination of three types of activities:
benchmarking, testing, and application performance
Home ⇒ Search ⇒ Reserve Flight ⇒ Book Flight
management (APM). These three activities involve
⇒ Exit
performance measurement. What distinguish them is
Home ⇒ View Reservation ⇒ Book Flight ⇒ Exit
how these measurements are taken, when they are
taken, where they are taken, and how and to whom
they are reported.
If we analyze a CBMG using the methods in [MA00],
we can determine the average number of times per
4.1 Benchmarking
session that each state is visited. For example, for
the CBMG of Fig. 1, we would have the following
Benchmarking is a process used to compare the
numbers: Home Page (1.00), Search Flight (1.0857),
performance of a hardware and/or software system,
View Reservation (0.20), and Book Flight (0.0899).
called the system under test (SUT). An SUT could
So, 8.99% of the visitors to our travel site end up
be a Web-based application. The following are
booking a flight.
important components of any benchmark:
It is also important to know how long a customer
q Workload specification: determines the type
takes between transitions. For example, some
of requests and the frequency by which these
customers may take 15 seconds on average to go
requests are submitted to the SUT. Some
from Search Flight into Reserve Flight while others
benchmarks, such as TPC-W (www.tpc.org)
may take 60 seconds to make the same transition.
specify a controlled environment in which all
This is called the think time. It is important to
the requests submitted to the SUT come
incorporate this feature into the customer behavior
from a specified workload generation process
model because of the impact it has on the site
[Menascé2001, Menascé2002b]. Others,
performance. As customers submit their requests
such as Keynote Performance Benchmarks
faster, i.e., with smaller think time, a site has a
(e.g., Broker Trading, Business 40,
higher arrival rate of requests to process.
Consumer 40, Government 40, Streaming,
What if a customer submits a search request for a
and Wireless SMS) generate requests in a
flight and the travel site takes 45 seconds to reply to
realistic environment, in which other

5

---

Published in the 2002 Computer Measurement Group (CMG) Conference, Reno, NV, Dec. 2002.
sources—real customers—also generate
scripts along with a set of metrics and measurement
requests to the SUT. Consider for example
procedure constitutes a benchmark for the site.
Keynote’s Business 40 (KB40) Internet
Benchmarks need to be run on a regular basis from
Performance Benchmark. The workload of
different Internet locations in order to know what the
this benchmark is composed of requests to
customer experience is.
download the home page of a Web site.
If a Web site provides customer access through
These requests are generated by automated
wireless devi ces such as Palm OS devices, RIM
agents every 15 minutes, 24 hours a day,
pagers, SMS and WAP phones, it is important to
seven days a week, and are reported at
benchmark the
site under the conditions
Keynote’s Web site for the period of Monday
experienced by customers using these wireless
to Friday between 9 am and 3 pm Eastern
devices.
time.
q Specification of metrics: determines what to
4.2 Performance Testing
measure. Consider again the example of
Performance testing, also called load testing, is the
Keynote’s KB40 benchmark. What is
process of running a specified set of scripts that
measured by the benchmark is the average
emulate customer behavior at different load levels
download time of the home pages of 40
and taking measurements of a site’s performance for
important US -based business Web Sites,
each of the load levels.
spread through categories such as
publishing, search engines, business
There are several circumstances that call for load
services, financial services, high-technology,
testing. Suppose a business is expecting a
and communications. This benchmark is
significant increase in the number of visitors due to a
used to measure download speed for well-
marketing campaign. So, instead of the current
connected major business users.
3,000 session starts per hour at the peak hour, the
business expects to see twice as many sessions per
q Specification of the measurement procedure:
hour. Dial-up customers are currently experiencing
determines how the values of the metrics
an average response time of 6.5 seconds on search
should be obtained. Again, in the case of the
requests, your most critical e-business function.
KB40 benchmark, measurements are
What will be their response time when the load on
obtained by averaging the results obtained
the site increases to 6,000 sessions per hour?
by agents attached to key points in the
Suppose that the site is adding new functionality or
Internet backbone in the 25 largest
that the Web pages have been redesigned. What
metropolitan areas of the United States. The
will be the customers’ experience in terms of
exact location, i.e., backbone provider and
response time after the changes? One must be able
geographical location, of each agent has to
to answer these questions before the customers so
be specified as part of the measurement
that potential performance problems are detected
procedure.
and fixed prior to making the Web application
available to customers. You may need to do load
Benchmarking is an important activity at the
testing if you plan to implement changes in your IT
business level of the reference model since one
infrastructure.
needs to know how well a Web-based application is
doing with respect to the competitors. One also
Performance testing is usually carried out with
needs to know how different types of customers
controlled and parameterized workloads. A key
perceive the response time and availability of a site.
concept in load testing is the notion of a virtual user
It could very well be that business users (i.e., those
(VU). A virtual user emulates what a real user does.
with high-bandwidth connectivity to the Internet)
A load test is only valid if the behavior of virtual
perceive a response time that is acceptable while
users has characteristics similar to those of actual
home users (i.e., those with dial-up connections)
users. Remember the Customer Behavior Model of
have a very bad experience at the site. This may
section 3.3? Well, one needs to make sure that
lead to a significant loss in revenue if home users
virtual users navigate through a Web site according
account for a substantial share of the buying
to patterns similar to real users, with realistic think
customers.
times, and that they will react as frustrated users and
abandon the session in case of excessive response
For transactional Web sites, such as e-commerce
times.
sites, one
needs to benchmark complete
transactions and not just Web page downloads.
If a load test does not allow one to mimic what
Since not all e-businesses are the same, one may
happens when real users visit a site, one may get
need to create scripts that reflect critical sessions
results that are totally inconsistent. For example,
containing the site’s critical transactions. This set of
customers who abandon a session end up using

6

---

Published in the 2002 Computer Measurement Group (CMG) Conference, Reno, NV, Dec. 2002.
less site resources than those who complete a
danger of this approach is that while performance
session. So, if the capacity of the infrastructure is
and/or availability problems are not identified,
planned assuming that all started sessions will
diagnosed, and corrected, a Web site will operate in
complete, over provisioning may occur. Also, if one
a less than optimal manner. As discussed before,
does not consider session abandonment, one will
these conditions may pose threats to a business.
not be able to accurately quantify revenue
The other, preferred approach, is one in which
throughput nor potential lost revenue throughput.
companies proactively seek to maintain the QoS of
These are important metrics at the business model
their applications through management processes
level.
that reduce occurrences of poorly performing
applications.
The main important parameters to vary during a load
test are:
APM involves real-time monitoring of the QoS of a
Web site. As we saw in the previous section,
q Workload intensity, typically measured in
response time and availability are influenced by
started sessions/hour.
factors that occur both inside and outside the
firewall. This means that end-to-end measurements
q Workload mix, described by the
from multiple locations as well as detailed monitoring
combination of scripts that determine what
inside the firewall should occur.
the typical sessions are and what
When QoS goals are not met, APM tools should
customers do in each session type.
trigger alarms by sending detailed e-mail and or
pagers to the proper members of the technical staff,
q Customer behavior parameters, such as
and provide fast diagnostic capabilities.
abandonment threshold and think time.
Performance and/or availability problems that take
Typical results of a load test include:
long to be addressed are bad news. Th e tools used
for APM should allow for a clear identification of the
q Number of completed and abandoned
type of problem (e.g., DNS server vs. the load
sessions per hour as a function of the
balancer at the site) so that the right team can be
number of started sessions/hour.
put to work on the problem as fast as possible.
An important element in facilitating fast problem
q Revenue throughput and potential lost
resolution is the capability to have a seamless
revenue throughput as a function of the
integration between the measurements and
number of started sessions/hour.
reporting provided by a company’s Enterprise
Management System (EMS) and the measurements
q Individual page download times and
obtained outside the firewall.
transaction completion times versus the
number of started sessions/hour.
The performance of a Web application depends on
the performance of services provided to the
company by service providers such as ISPs, Content
4.3 Application Performance Management
Delivery Networks, ad networks, payment gateways,
(APM)
and others. A company may also be a provider of
Application Performance Management (APM) is a
services to others. In fact, this model will become
collection of management processes used by
more prevalent as the notion of Web Services (see
organizations to ensure that the QoS of their e-
www.w3.org/TR/wsdl) becomes widely adopted.
business applications meets the business goals. An
Whenever companies provide or use services from
e-business application is implemented by a
others, Service Level Agreements (SLAs) must be
collection of Web pages delivered by Web sites with
put in place. SLAs are documents that specify
the support of all servers, programs, and databases
realistic performance guarantees as well as
that make up the site. As discussed previously, QoS
penalties for non-compliance [Overton2002].
goals include response time and availability. Both
SLA management involves three steps:
goals are very important and have to be considered
simultaneously. Poor response time in a highly
q SLA design: realistic SLAs are developed,
available Web site is as undesirable as a site that
agreed upon by the parties, and specified in
can respond fast but is not available close to 100%
a document that lays out thresholds,
of the time.
reporting and enforcement procedures, as
There are two basic approaches to APM: reactive
well as penalties for non-compliance.
and proactive. In the former case, companies
monitor their site’s QoS and react when problems
q SLA measurement and reporting: frequent
arise. This is called the “fire-fighting” approach. The
(e.g., hourly, daily, monthly) measurement

7

---

Published in the 2002 Computer Measurement Group (CMG) Conference, Reno, NV, Dec. 2002.
and reporting on SLA compliance and
Functional level requirements:
computation of penalties for non-
compliance.
q Need to be able to test functions supported
by many different technologies including
q SLA analysis: longer-term re-evaluation of
Flash, JavaScript, Active X, cookies, and
SLAs, which may be necessary due to
SSL.
changes in market conditions, changes in
Customer behavior level requirements:
technology, and user behavior.
SLA management is therefore a fundamental aspect
q Realistic modeling of customer behavior
of APM.
including user abandonment, tolerance to
high response times for different types of
5. Tool Requirements
pages and e-business functions, customer
tenacity, customer experience with the site,
We examine in this section important requirements
and different think times.
that should be satisfied by tools that support the
three activities discussed in the previous section.
q Need flexible and easy-to-use ways of
recording scripts that represent different
types of customer interactions.
5.1 Benchmarking
Business-level requirements:
q Need to be able to easily change load
testing scripts to adapt to changes in
q Need to be able to measure and compare
customer behavior that occur over time.
your response time and availability with that
IT resource level requirements:
of your competitors using metrics and
measurement procedures that are well
q Need to be able to test Web applications on
accepted and recognized in the industry.
a regular basis in an actual production
environment and not just in a scaled-down
q Need to be able to evaluate third-party
testing environment.
service providers (e.g., CDNs) prior to
making a contracting decision and
q Need to assess the impact of changes in
determine whether or not these service
system architecture, server types and their
providers meet your SLAs on an ongoing
capacity, storages boxes, software, and
basis.
networking bandwidth.
Customer behavior model level:
q Need to be able to carry out load tests on-
q Need to account for different user platforms,
demand and at scheduled times.
geographical location, connection speed,
and type of client device and browser,
including wired and wireless platforms.
5.3 Application Performance Management
Business-level requirements:
q Need to be able to benchmark complete
sessions and not just specific pages.
q Need a fast way to detect the cause of
performance and availability problems so
that the right team can be put to work on the
5.2 Testin g
problem as fast as possible and the
Business-level requirements:
company’s revenue stream is not reduced or
even interrupted.
q Need to be able to track revenue throughput
and potential lost revenue throughput.
q Need a methodology for clearly designing
and negotiating
SLAs between your
q Need to be able carry out load tests under
company and your providers. These SLAs
the most realistic and thorough conditions
must be realistic to avoid stiff penalties for
possible to help avoid over- and under-
non-compliance. Thus, before committing to
provisioning.
SLAs, it is important to validate them in live
environments.
q Need to know the impact of business
decisions (e.g., marketing campaigns or new
q Need to review and renegotiate SLAs to
business models) on the IT infrastructure.
keep them consistent with new conditions.

8

---

Published in the 2002 Computer Measurement Group (CMG) Conference, Reno, NV, Dec. 2002.
q Need reporting mechanisms on financial
n
N : number of concurrent requests being
penalties for non-compliance of SLAs.
C
processed by a Web site.
n
q

Z : average think time, in seconds.

Need to put in place a process to enforce
and report SLA compliance that is
n
R : average response time of a request, in
acceptable by all parties involved. A credible
seconds.
third-party service is usually preferred.
n
X : average throughput, in requests/sec.
0
Customer behavior level requirements:
Using the Response Time Law [DB78, MA02], we
get the following relationship:
q Need to identify which aspects of the
N
customers’ characteristic (e.g., their

R
VU
=
− Z .


(1)
geographic location) are responsible for
X 0
detected performance problems.
Z
IT Resource level requirements:
q
VU

Need to integrate measurements obtained
inside the firewall with measurements taken
Web
outside the firewall.
.
Site
.
q Need to be able to integrate outside the
under
.
firewall measurement with several EMS
Test
frameworks and provide for easy data
NC
correlation.
VU
R
q Need a single and effective visualization
Virtual Users
X
interface with drill-down capabilities that
0
display the components of delays inside and

outside the firewall.
Figure 2 - A Web site receives requests from a
number of Virtual Users (VU).
q Need detailed network level and page
element details (e.g., DNS resolution, TCP
The throughput of a Web site is a function of the
connection, redirection, SSL handshake
number N of concurrent requests in execution—
times, first packet response time, content
C
download time, and content errors).
the load level—and of the service demands of these
requests at each of the resources (e.g., processors,
q
storage devices, networks) of the site. The service

Need measurement and reporting
procedures regarding SLA compliance.
demand, D , of a request at a resource i is defined
i
as the average total time spent by the request
6. Fundamental Concepts in Load Testing
receiving service from the resource [MA02]. This
time does not include any queuing time and is
Load testing can be used to predict the performance
therefore independent of the load level. So, we can
of a Web site at any load level by simply increasing
write that
the number of VUs until the desired load is
achieved. Running load tests for a large number of
X (N ) = f ( D ,..., D , N )

(2)
0
C
1
K
C
values of the number of VUs and workload mixes
may be very time consuming and expensive. Faster,
to indicate that the throughput is a function of the
albeit less accurate results, can be obtained by
service demands at the K resources of a Web site
combining load testing with performance models—
and of the load level. The response time is also a
analytic or simulation [Menascé2002a]. We provide
function of the service demands and of the load
in what follows some basic performance
level. Thus, we can write that
relationships that can be used to speed up
R( N ) = g(D ,..., D , N ) .

(3)
scalability analysis with load testing.
C
1
K
C
Consider Figure 2, which depicts several virtual
So, combining equations 1-3, we get that
users submitting requests to a Web site, and let:
N
= R N + Z × X N .
(4)
VU
[ ( )
C
]
(
)
0
C
n
N
: number of virtual users.
VU
We can now use either an analytic or simulation
model to predict the values of the response time and

9

---

Published in the 2002 Computer Measurement Group (CMG) Conference, Reno, NV, Dec. 2002.
throughput for different values of the load level N
for processing and for I/O are 8 and 9 milliseconds,
C
respectively. Using the Mean Value Analysis method
and use equation 4 to estimate the number of VUs
[MA02] one can compute X (N ) and R( N ) for
needed to generate a given value of N .
0
C
C
C
various values of N . Figure 3 shows the resulting
C
Load testing tools can be quite useful in that regard
since they can be used to generate scripts that are
curves of R( N ) and X (N ) versus N . The
C
0
C
C
submitted for a low number of VUs in order to
figure also shows that for N = 19, the number of
measure the service demands—remember that the
C
service demands are load-independent. The service
VUs, computed using equation 4 and assuming an
demands are then used as input parameters to
average think time of 8 seconds, would be equal to
performance models as explained in [MA02]
897.
We illustrate these concepts through an example.
Consider a Web site in which the service demands
120
0.25
Z
100
0.20
NVU = (0.173+8) x 109.7 = 897
80
0.15
60
0.10
40
Throughput (req/sec)
Response time (seconds)
0.05
20
0
0.00
0
1
2
3
4
5
6
7
8
9
10 11 12 13 14 15 16 17 18 19 20 21 2 2 2 3 2 4 25 26
No. Concurrent Requests
Throughput (sec)
Response time (sec)

Figure 3 – Throughput and Response Time vs. No. of Concurrent Users.
The maximum value of the site throughput can also
events. One must be able to measure the
be estimated from the service demands by using the
components of response time inside and outside the
upper bound equation [MA02]:
firewall, at different load levels, from different
geographical locations, and at different times.

When analyzing Web and E-commerce applications,
X (N ) ≤ 1/ max{ D }.


(5)
0
C
i
it is useful to consider a four-level reference model
composed of a business level, functional level,
In our example, the maximum throughput would then
customer behavior model level, and IT resource
be 1 / max (0.008, 0.009) = 1 / 0.009 = 111.1
level. The business level is concerned with the
requests/sec, which is the horizontal asymptote of
nature of the business, its revenue goals, its
the throughput curve in Figure 3.
partnerships with other businesses, and its position
within the industry. It is important to be able to
assess the revenue throughput and potential lost
7. Concluding Remarks
revenue throughput. Decisions at the business level
The QoS of Web-based applications is measured in
are aimed at obtaining the best possible QoS with
terms of end-to-end response time and availability,
the least possible amount of IT capital and
as perceived from many different geographical
personnel expenditures.
regions. The availability requirements may also vary
There are three key activities required to manage
according to the time of day or according to special
the QoS of your Web-based applications:

10

---