Andy Edmonds, an alumnus of our HF program, will be speaking to us this week. The title of his talk and some suggested readings are given below.

As usual we will meet at 2:30 in the Psychology Conference room, 419 Brackett Hall.

From Andy:

Title: "Experimentation on the Web for Business: Opportunities & Pitfalls"

SUGGESTED READING
Practical Guide to Controlled Experiments on the Web: Listen to Your
Customers not to the HiPPO.

http://exp-platform.com/hippo.aspx
The web provides an unprecedented opportunity to evaluate ideas quickly
using controlled experiments, also called randomized experiments
(single-factor or factorial designs), A/B tests (and their
generalizations), split tests, Control/Treatment tests, and parallel
flights. Controlled experiments embody the best scientific design for
establishing a causal relationship between changes and their influence
on user-observable behavior. We provide a practical guide to conducting
online experiments, where end-users can help guide the development of
features. Our experience indicates that significant learning and
return-on-investment (ROI) are seen when development teams listen to
their customers, not to the Highest Paid Person's Opinion (HiPPO). We
provide several examples of controlled experiments with surprising
results. We review the important ingredients of running controlled
experiments, and discuss their limitations (both technical and
organizational). We focus on several areas that are critical to
experimentation, including statistical power, sample size, and
techniques for variance reduction. We describe common architectures for
experimentation systems and analyze their advantages and disadvantages.
We evaluate randomization and hashing techniques, which we show are not
as simple in practice as is often assumed. Controlled experiments
typically generate large amounts of data, which can be analyzed using
data mining techniques to gain deeper understanding of the factors
influencing the outcome of interest, leading to new hypotheses and
creating a virtuous cycle of improvements. Organizations that embrace
controlled experiments with clear evaluation criteria can evolve their
systems with automated optimizations and real-time analyses. Based on
our extensive practical experience with multiple systems and
organizations, we share key lessons that will help practitioners in
running trustworthy controlled experiments.

OPTIONAL READING

INSTRUMENTING THE DYNAMIC WEB Edmonds, A., White, R. , Morris, D.,
Drucker, S., Journal of Web Engineering, Vol. 6, No. 3, 2007.

One of the most critical driving forces in the evolution of interfaces
on the Internet has been the logging
built into common Web servers and the decade-long deployment of
analytics based upon this data source.
Page-view logging has slowly moved to callback systems using client-side
scripting to capture more
aspects of the user experience. With the rise of JavaScript-based
client-side interactivity and, more
recently, asynchronous Javascript and XML (AJAX), server-side logging is
less able to capture the user
experience of Web sites and applications that are rising in complexity.
We present a new technique for the
in-page logging of interaction events that will help interaction
designers make more informed design
decisions based on how users are interacting with their systems. The
potential benefit of our technique is
demonstrated in a case study with a working system.

http://research.microsoft.com/%7Esdrucker/papers/EdmondsJWE2007.pdf