BA371 - IS Impact Assessment - What Difference Does it Make?

• Bill Hewlett: "You cannot manage what you cannot measure."
  

• Until you can measure something (and express it in numbers,) you have only the beginning of understanding. Lord Kelvin (1824 - 1907).
  
• A strong conviction that something must be done is the parent of many bad measures. Daniel Webster (1782 - 1852).
• Those who speak most of progress measure it by quantity and not by quality. George Santayana (1863 - 1952).

• We are all caught up in this measurement mania. We are not growing in wisdom right now. We may be just growing in freneticism. (Margaret J. Wheatley) (Brainquotes.com).

• Evaluation of IS system impacts implies measurement. However, measurement is far from unproblematic.
• Productivity/efficiency: Kock, p. 41: "IT 's dirty little secret:" productivity paradox.

• Bank and insurance industry does not take advantage of IT.
• US economy: stationary productivity 1960-1995.
• Productivity losses may be acceptable if properly compensated with revenue growth (expanding market or market share).
• Brynjolfsson & Hitt (1998).

• Quality; e.g., effectiveness, customer satisfaction, reliability, etc.:
• Proper product mix.
• On-time delivery.
• Defects/1000 units; Mean Time Between Failure (MTBF); % uptime.
• Many small setbacks vs. one really nasty one.
  
• Nevertheless; we must measure IS impact and/or ROI: first issue in this year's case study.
  
• A few words on this year's case study.
  

• Issues of validity:

• Model of change (Walker, p. 6): "specific set of relationships that one believes connects the intervention to the achievement of the impact objectives."
• Model of change (causation):
• Walker example: introduction of chemical engineering curriculum.

• Introduction of new drug benefit, tax hikes/reductions, rules, ordinances and laws, medication, etc.

• Model (Walker p. 6) is a theory of causes and effects from which hypotheses can be formulated that can be empirically tested.

• Do we measure what we intend to measure?
• Do our observables match our theoretical constructs? (construct validity):
• Blood pressure reducer --> measure blood pressure.
• MIS undergraduate curriculum change to increase employability --> measure one-year employment rate or average time before first employment or starting salary or ...
  
• COB 2007: Core curriculum change to increase students' professional readiness --> ??

• Theoretical variables or constructs --> indicator variables or observables: (operationalization).

• Can we infer causation from our observations? Can we tie the intervention or treatment to the measured variables?
• Research design: experiments & quasi experiments (Walker: Step 6).
• Core idea: choose a research design that isolates the effect of the intervention.
• Designs:
• Pre & post test with randomized treatment:

  O₁    T    O₂
  ----------------
  O₁           O₂

  • Very strong & robust design.
  • Requires a control group.
  • Randomized treatment (single vs. double blind).
  • Often not realizable (OSU Institutional Review Board (IRB)).

• One-group quasi pre- and post test:

• Very weak design.
• Pre- and post test often do not measure the same thing (construct validity threat).
• Recording effects; e.g., lawn mowing injuries example (measurement error)
  

• Sensitive to trend and other, intervening factors (history threat).
• Problem: when evaluating our IT investments, does it matter that we do not have a control group?

• One group interrupted timeseries:
  

• What does this mean for our class project?
• When you think about how do implement an IS impact / ROI plan for the case study:

• Think carefully about your cause and effect variables.
• Think carefully about your causal model (model of change).

• Consider how to operationalize the theoretical (construct) variables).
  
• Consider your research design.
• Play devil's advocate; work hard to find the validity threats in your own proposals (or ask your peers to do it).
  
• Formulate carefully and clearly in your IS impact analysis plan.