We are all caught up in this measurement
mania. We are not growing in wisdom right now. We may be just growing
in freneticism.
(Margaret
J. Wheatley) (Brainquotes.com).
Evaluation of IS system impacts implies measurement. However,
measurement is far from unproblematic.
Productivity/efficiency:
Kock, p. 41: "IT 's dirty little
secret:"
productivity paradox.
Bank & insurance industry does not
take advantage of IT.
US economy: stationary productivity
1960-1995.
Productivity losses may be acceptable if
properly compensated with revenue growth (expanding market or market
share) (UPS/Fedex)..
Defects/1000 units; Mean Time
Between Failure (MTBF); % uptime.
Many small setbacks vs. one really nasty one.
Nevertheless; we must measure ROI:
Issues of validity:
Model of change (Walker, p. 6): "specific set of relationships that one
believes connects the intervention
to the achievement of the impact
objectives."
Model of change (causation):
Walker example: introduction of chemical engineering
curriculum.
Introduction of new drug benefit, tax hikes/reductions,
rules, ordinances and laws, medication, etc.
Model is a theory from which hypotheses can be formulated
that can be empirically tested.
Do we measure what we intend to measure?
Do our observables match our theoretical constructs? (construct validity):
MIS undergraduate curriculum change to increase
employability --> measure one-year employment rate or average time
before first employment or starting salary or ...
Core curriculum change to increase students'
professionality --> ??
Theoretical
variables or constructs
--> indicator variables
or observables: (operationalization).
Can we infer causation from our observations? Can we tie
the intervention or treatment to the measured
variables?
Research design:
experiments & quasi experiments (Neff: Step 6).
Basic idea: choose a research design that isolates the
effect of the intervention.
No good way to tie the values of the after-treatment
OA1 to the treatment.
Construct
validity threat.
History
threat.
One group pre and post test:
O1
T O2
Rather weak design.
No good way to tie the values of the after-treatment O2
to the treatment.
Sensitive to trend and other factors (history threat).
Problem:
when evaluating our
IT investments, does it matter that we do not have a control group?
One group interrupted
timeseries:
O1 O2 O3
O4 O5 O6 O7
T O8 O9 O10 O11
O12 O13 O14
Corrects for trend ==> reduces history threat.
Selection threat.
Must collect data prior to the treatment.
Very relevant to IS-ROI.
Cross-sectional (survey) designs:
Select (sample) many more cases.
Look for patterns across cases and/or in subgroups.
Conceptually matches the one- and two-group nonrandomized
designs.
Many threats assumed(!) to cancel each other out.
Delone & McLean (2003) The
DeLone and McLean Model of Information Systems Success: A Ten-Year
Update:
Original paper (1992) Information
Systems Success: the Quest for the Dependent Variable:
Reviewed 100 studies assessing ISs on six dimensions of success:
Systems quality (technical success).
Information quality (semantic success - accuracy,
meaningfulness and timeliness).
Use.
User satisfaction.
Individual impacts (e.g.,
influence on management
decisions).
Organizational impacts.
Note: each of
these are theoretical variables (constructs).
Model of causation (Figure 1, p.12).
2003 paper: Operationalization and empirical studies of the
1993 D&M model (Figure 2, p. 14).
Construct and causal model validity:
Seddon (1997):
"...the inclusion of both variance and
process interpretations in their model leads to so many
potentially confusing meanings that the value of the model is
diminished."
"...when a reader looks
at
D&M's model, his/her efforts to make sense of different parts of
the model will frequently cause slippage from one meaning for a box or
arrow to another. The result is a level of muddled thinking that is
likely to be counter-productive for future IS research."
D&M: Process vs.
Causal Models (p.15): is satisfaction caused by use, does it
merely follow use or is there a reinforcing relationship
Use vs. benefits
of use. Is 'use' an appropriate measure of
success?
D&M: use patterns over time (does use go up or down).
Others: use and non use of different parts of a system.
