Evaluating CCTV and Recording of Child Interviews and

Evaluating CCTV and Recording of Child Interviews and Testimony M. Christine Kenty, Ph. D Sharon Elstein, MS ABA Center on Children and the Law

KENTY'S CLUES FOR EVALUATION n #1. Base any evaluation on your own goal pathway and logic model. n #2. Organizations are social systems with their own cultures. n #3. Whatever people don't want you to study, that's the really important thing.

And those touchy things are likely to be: n n n Collaboration Quality of Forensic Interviewing Decision-making about arrest, prosecution and child protection Relating to victims and families Children’s experiences before and in the grand jury or courtroom

Kenty’s Clues n #4. It's trouble if only one person is doing all the thinking about evaluation. n #5. You can't keep partners and stakeholders too well informed about the evaluation process.

Kenty’s Clues n #6. Any evaluation (or program report) requires DATA, so decide early what you DATA need to record and then keep it up. n #7. For any evaluation, there are many good designs, but no perfect ones. n #8. Don’t kill the messenger if you don't like the news!

Evaluation… is a way to improve a program by systematically examining and analyzing what the program is doing and what it has accomplished.

The “I’ll know it when I see it” Rule I’ll know that our program is working when I see……….

What evidence can convince us and others that our program is on target? What would tell us that something has happened? How can we count it or track it?

What evaluation can do: n n Help improve the program from the beginning Provide staff and stakeholders with a much-needed sense of accomplishment Guide protocol, policy and law reform Assist in developing future funding

Don’t put the evaluation in the hands of just one individual Whether that is an internal or external evaluator

A healthy organization needs to know the program mission, plan the work, develop enthusiasm, and bring things to fruition.

An organization also needs to look at what it’s doing, keep what's good and try to jettison what isn't working.

Evaluation isn't a separate topic…. it's just one more piece of the work

Forces will try to marginalize and minimize an evaluation Don’t let that happen – establish a strong committee!

Base the evaluation on your agency pathway. “Logic model” and “pathway map” are popular phrases with funders.

Logic Model or Pathway Map Each part should logically follow from the last Context Strategy Activities Outcomes (What we live in) (Where we focus) (What we do) (Results)

We all work on underlying assumptions, which might be --§ § § Better technology will improve dispositions Technology will make things easier for kids – fewer interviews, less testifying… Our forensic interviewing will stand up to scrutiny The defense bar will not limit the potential of recorded testimony We will know if this is working well TALK ABOUT AND CLARIFY THESE!

Logic Model or Pathway Map Context • Needs • Resources • Laws/man- dates • Stakeholder buy-in Strategy Activities Outcomes

You may do a needs assessment to describe your context A needs assessment is a systematic way to discover: n what you “need” in order to accomplish a goal, and then nmake decisions based on that assessment

Typical needs assessments 1. Estimate how many clients/professionals will participate in a new program 2. Determine what resources are already in place and what has to be put into place 3. Decide what an agency or community needs to provide to get a particular result 4. Envision how technology and products will be used so that the equipment will be right 5. Decide what training people need

You may have already done kind of needs assessment but you may still want to do another piece as you begin to implement your program

1. Estimate how many clients/professionals will be involved § Count # of allegations, investigations, interviews, arrests, prosecutions, hearings, dispositions in the last year § List all the professionals who will need to be trained or familiarized

2. Determine what resources are in place and needed § § § All professionals and what they do Adequacy of infrastructure: rooms, technology, wiring, lighting Relevant state statutes re victims and CCTV and/or recorded testimony Policies and procedures for interagency work Other available funding

3. Decide what is needed for specific result (These are suggestions, not requirements) § What equipment, facilities, personnel, time, training, systems, policies, statutes do we need, for example: § § To videotape all child interviews for children aged 3 -13? To decrease the number of child victims who testify in person? To improve successful disposition rates? To make the system more child-friendly?

4. Envision how equipment/product will be used § § § § Stationary or portable Professional technician or multiple users Video all children or selected cases or ages CCTV according to statutes Show videos in what rooms to what audiences Who needs a cut-off switch Maintenance, upkeep costs, back-ups

5. Determine what training people will need § § § Technical skill in equipment use and maintenance Scheduling and informing children/caregivers Interviewing skills Judiciary, Prosecutors and Bar § Permissibility/use of recording and CCTV according to all statutes § Awareness of capacity § Forensic use of recordings at multiple points Quality assessment and record-keeping for recorders and prosecutors

Design method of data collection Type of assessment question Possible Methods 1. # of clients and professionals 2. Resources have/need Document review: case records, annual reports, policies, inter-agency agreements Interviews/surveys Document review: case records, annual reports, policies, inter-agency agreements Surveys/focus groups Observation

Design methods of data collection Type of assessment question 3. What will achieve a specific result 4. Likely use of equipment Possible Methods Focus Group Interviews Documents/Information Search: literature, state-of-the-art Document Review – statutes, case records Focus Group/Interview

Design methods of data collection Type of assessment question 5. Training needs Possible Methods Review of vendor materials Surveys/interviews Observation

Logic Model or Pathway Map Context Strategy • Needs • Technology • Resources • Staff • Laws/man- • Training dates • Stakeholder buy-in • Information Activities Outcomes

Logic Model or Pathway Map Context Strategy Activities Outcomes • Needs • Technology • Acquire, install equipment • Laws/man- • Training • Train dates • Information • Inform • Stakeholder buy-in • Partner • Use • Resources • Staff

Logic Model or Pathway Map Context Strategy • Needs • Technology • Acquire, • Resources • Laws/man- dates • Stakeholder buy-in Activities Outcomes • Short term install • Staff • Medium equipment term • Training • Train • Long term • Information • Inform • Partner • Use

Logic Model or Pathway Map Context Strategy Activities Outcomes Short term • Technology • Attitudes • Skills • Opinions

Ideas for short term outcomes n n n Plans are complete; we know who contributed and what went into planning Stakeholders are aware and committed Equipment is acquired and installed

Logic Model or Pathway Map Context Strategy Activities Outcomes Medium term • Action • Practice • Policies

Ideas for medium term results n n n Equipment being used regularly People skilled and knowledgeable Stakeholders have assessed use Data being collected on interviewing, use of recordings and CCTV, and can be summed Data establishing a baseline for later comparison

Logic Model or Pathway Map Context Strategy Activities Outcomes Long term • Impact on the overall context

Ideas for long term results (impact) n n n Quality of forensic interviewing and recording is consistently high Prosecutors use recording and CCTV regularly and effectively in known ways Prosecution rates and/or case dispositions have been improved Stakeholders, clients and families are satisfied with the process and use of recordings Laws and policies have been changed

TYPES OF EVALUATIONS NEEDS ASSESSMENT A systematic way to discover what you need to accomplish a goal Helps make informed planning decisions

TYPES OF EVALUATIONS FORMATIVE EVALUATION Short-term initial feedback on how the program is working Helps quickly readjust planned activities to be more effective

TYPES OF EVALUATIONS PROCESS EVALUATION Describes how something happened rather than outcomes To understand the internal dynamics of organizations and relationships, and capture what activities are actually happening.

TYPES OF EVALUATIONS IMPACT or OUTCOME EVAL determines whether a program produced desired results. Requires articulated outcomes and targets and they must be measurable.

RESEARCH METHODS n QUANTITATIVE METHODS n QUALITATIVE METHODS

RESEARCH METHODS QUALITATIVE QUANTITATIVE METHODS: n numerical research by n naturalistic research by studying collecting data about participants' pre-selected perceptions and variables, and experiences in studying cause and context and the way effect they make sense of them

QUANTITATIVE METHODS n n n surveys with pre-determined categories and rating scales document review – e. g. counting up numbers of arrests or prosecutions, and comparing them to other groups or time periods evaluator attempts to keep at an objective distance from the people

QUALITATIVE METHODS n n n n surveys with open-ended questions interviews – semi-structured observation document review – e. g. process, attitudes case studies focus groups evaluator gets close to the people to capture what is actually happening

Collect both quantitative and qualitative data - they are equally valid Qualitative data help capture changes in processes and relationships, and some things just aren’t countable.

Sampling: how many records or which people to ask More important to have a representative sample than a large sample

But, every professional might need to be heard so that no one feels left out and there is no suspicion of bias So sampling may not be acceptable

Shaping evaluation questions Outcome Questions Short term What are the plans Document review and who did what? Interviews Are the plans comprehensive? Plans are complete Method

Shaping evaluation questions 2 Outcome Short term Stakeholders aware and committed Questions Are key players knowledgeable and ready to run? What are their reservations and perceived barriers? Method Observation Interviews Focus Group

Shaping evaluation questions 3 Outcome Questions Short term Is it up and running? Equipment Can technician acquired/in produce good quality -stalled video or transmission? Should quality be improved? Methods Observation Document review Interviews

Shaping evaluation questions 4 Outcome Questions Methods Medium Term What interviews or Observation Equipment testimony are being Document review used recorded or regularly transmitted via CCTV? How are recordings stored and accessed?

Shaping evaluation questions 5 Outcome Medium term Questions Methods Who has been trained? Observation Key players Do key players feel Interviews skilled and/or prepared, confidant? knowledgeable What else is needed?

Shaping evaluation questions 6 Outcome Medium term Questions Does everyone like the Stakeholders program? have What doesn’t work for assessed use whom? Methods Surveys Interviews Focus group

Shaping evaluation questions 7 Outcome Questions Medium term Are logs and Document instruments completed review consistently? Interviews Do they have the right indicators? Data being collected on interviewing, use of recordings, etc. Methods

Shaping evaluation questions 8 Outcome Questions Methods Medium term Have we assembled data on quality of interviewing, arrest, prosecution, disposition before video/CCTV? Document review Data establishing baseline

Shaping evaluation questions 9 Outcomes Questions Methods Long Term How do they rate on adherence to protocol, state of the art, forensic value? Do we peer review? Observation Review of selected recordings Interviews Quality of forensic interviewing is high

Shaping evaluation questions 10 Outcomes Long term Questions How do they use? Prosecutors, Are they satisfied? child What changes can protection use increase use? technology regularly, effectively Methods Observation Document review Interview Survey

Shaping evaluation questions 11 Outcomes Questions Long term What are the rates Document and dispositions review before and since video/CCTV Prosecution rates and/or case dispositions have improved Methods

Shaping evaluation questions 12 Outcomes Questions Long term What works/does Survey not work for each Interviews profession, group, agency? Stakeholders, clients and families are satisfied Methods

A monitoring system is part of the implementation of the program, and is also a key element of any evaluation.

The monitoring system insures that. . n Each part of the system can get the information it needs to proceed, i. e. arrest and prosecution, child protection, treatment, victim advocacy. n You can assess the effectiveness of your program.

If you know what is happening to your children and defendants from start to finish. . . Then you have an integrated data system

If you don’t. . . . Then policies, technology or turf issues are preventing it, and your evaluation will be affected.

Data collection for recorders § § Intake log of alleged victims: demo, perps Log of recorded interviews § Date, Number, Name, Age, Gender, Length § Forensic Interviewer § Observers – keep copy in central location, not just in evidence § Referred for prosecution – capture prosecutor # and date § Quick assessment of quality of interview by effectiveness, protocol and forensic value – highlight really good or not so good interviews for training and peer review library

Data collection instrument for assistant prosecutors Keep copies in a central location, not separately in each file or jacket!! #____ Defendant _______Victim____AP____ q Observed the forensic interview q Reviewed the recording of the interview q Forensic value of the recorded interview = Poor Good Very Good Explain: q Used the recording to encourage family cooperation w/ prosecution Explain: q Used the recording in diverting the case Explain: q Used the recording to obtain a pre-indictment plea Explain:

Data collection instrument for assistant prosecutors 2 q Used the recording or CCTV to obtain an indictment (GJ, deposition, other) Explain: q Used the recording to obtain a post-indictment plea Explain: q Used the recording or CCTV at pre-trial hearing or trial Explain: q The child testified before the grand jury or at preliminary hearing Explain: q The child testified at trial Explain: q Verdict and sentence:

Other possible evaluation data n n n Answers to items on a survey Answers to questions in an interview format Responses in a focus group Observations of group interactions in a planning/protocol meeting Observations of forensic interviews or children’s testimony

You can’t keep everybody too well informed about the evaluation. Too much is never enough; despite your best efforts, somebody may still feel blindsided.

Don’t kill the messenger if you don’t like the news! Don’t allow the evaluation to be deep-sixed if there are negative findings.

And don’t ever let the evaluator or committee twist in the wind! If the committee has done it right, everybody will understand the evaluation process and what the findings are before a final report comes out.