We are all caught up in this measurement mania. We are not growing in wisdom right now. We may be just growing in freneticism.
(Margaret J. Wheatley) (Brainquotes.com).
Evaluation of IS system impact implies measurement. However, measurement is far from unproblematic.
Issues of validity:
In general:
Do we have a clear idea of our 'model of change.'
Do we measure what we intend to measure?
Do we have sufficient measurements? Inductive/deductive issues.
Are our measurements sufficiently precise and reliable?
Does what we measure today, obtain tomorrow?
Who does the measuring?
Do we really (!!) want to know?
More specific:
Model of change (Walker, p. 6): "specific set of relationships that one believes connects the intervention to the achievement of the impact objectives."
Model of change (causation):
Walker example: introduction of chemical engineering curriculum.
Problem: Any issues with Walker's model of change?
Introduction of new drug benefit, tax hikes/reductions, rules, ordinances and laws, medication, etc.
Do we measure what we intend to measure?
Do our observables match our theoretical constructs? (construct validity)
MIS undergraduate curriculum change to increase
employability --> measure one-year employment rate or average time
before first employment or starting salary or ...
Core curriculum change to increase students' professionality --> ??
CSR system:
if software is sold as a product --> sales & profits/losses.
if software is used to prevent future CSR incidents: --> CSR incidence rate.
if software is given away as a loss leader: --> ??
Evaluation may imply cost/benefit trade-off:
Must measure both the TCO and the TBO/TVO.
Theoretical variables or constructs --> indicator variables or observables: (operationalization).
Can we infer causation from our observations? Can we tie the intervention or treatment to the measured variables?
Research design: experiments & quasi experiments.
Basic idea: choose a research design that isolates the effect of the intervention.
Look for patterns across cases and/or in subgroups.
Conceptually matches the one- and two-group nonrandomized designs.
Many threats assumed to cancel each other out.
Quantitative vs. Qualitative.
Delone & McLean (2003) The DeLone and McLean Model of Information Systems Success: A Ten-Year Update:
Original paper (1992) Information Systems Success: the Quest for the Dependent Variable:
Review 100 studies assessing ISs on six dimensions of success:
Systems quality (technical success).
Information quality (semantic success - accuracy, meaningfulness and timeliness).
Use.
User satisfaction.
Individual impacts (e.g., influence on management decisions).
Organizational impacts.
Note: each of these are theoretical variables (constructs).
Model of causation (Figure 1, p.12).
2003 paper: Operationalization and empirical studies of the 1993 D&M model (Figure 2, p. 14).
Construct and causal model validity:
Seddon (1997):
"...the inclusion of both variance and process interpretations in their model leads to so many potentially confusing meanings that the value of the model is diminished."
"...when a reader looks at
D&M's model, his/her efforts to make sense of different parts of
the model will frequently cause slippage from one meaning for a box or
arrow to another. The result is a level of muddled thinking that is
likely to be counter-productive for future IS research."
"Since the boxes in a
process model represent discrete have-happened/have-not-happened
events, and the arrows indicate sequence, not causality, it is not
possible to adopt a variance model interpretation of one part... and a
process model interpretation of another part. If one does, there must
be a slippage of meanings somewhere in between."
D&M: Process Versus Causal Models (p.15)??
Use vs. benefits of use. Is 'use' an appropriate measure of success?
D&M: use patterns over time (does use go up or down).
Others: use and non use of different parts of a system.
