Presentation for the 4 th Annual Center for

Presentation for the 4 th Annual Center for AIDS Prevention Studies Conference San Francisco, April 16, 2004 Client Level Data Collection: The California Experience Jeffrey Bernstein, MS Center for AIDS Prevention Studies, California Technical Assistance Program Shanna Livermore, BS Steven R. Truax, Ph. D California Department of Health Services, Office of AIDS, HIV Education and Prevention Branch, Sacramento 1

Today’s Presentation • Overview of ELI • What challenges have been experienced at the state and local levels? • How does ELI work successfully in California at the state and local levels? • How has this technology been used to inform public health decisions? 2

ELI Background Overview Development 3

What is ELI? • Evaluating Local Interventions (ELI) is a webbased evaluation system that enables California’s prevention providers to collect and access information critical for the planning and implementation of effective HIV prevention programs. 4

Background • Evaluation is important for planning and enhancing programs and for securing resources. • Office of AIDS (OA) has long recognized the need for evaluation and technical resources required to support it. 5

Background • CDC Guidance on evaluation released in 1999 added additional motivation. • CDC’s and OA’s shift in new directions 6

Shift happens!! 7

Benefits • Systematic data collection – Gives the client a voice • Consistent reporting requirements 8

Benefits • There is a strong relationship between data collection and program planning • Helps in grantwriting • Helps structure your services/conversations 9

Benefits • Immediate feedback to providers – Over 50 “cannned reports” – Raw data available 10

Prevention Activities Required to be Evaluated • Individual-level interventions • Group-level interventions • Outreach interventions • Prevention case management • Health Communication 11

ELI Development Partners • Coordination & Development – California State Office of AIDS – Universitywide AIDS Research Program/UCOP • Program Evaluation Technical Assistance – Center for AIDS Prevention Studies, UCSF • Guidance & Input – Local Health Departments – CDC 12

Strategic Planning Process for Development of ELI • Sought input from stakeholders • Assessed – existing evaluation efforts – state of the art measures, methods – local feasibility and utility 13

Pilot Phase Fall 2001 14

Pilot Process for ELI • ELI piloted in 5 counties • Feedback collected on – – data collection forms & protocols web system resource issues technical assistance needs • Trainings and TA developed • ELI revised 15

Pilot Feedback: Key Concerns • Use of individual-level forms in outreach and group settings • Need for additional fields, data elements, & reports • Ability to download data • Integration of existing data collection efforts into ELI system • Need for training and technical assistance on data collection, system use, and use of data • Resources 16

OA Response to Pilot Feedback • Revised protocol for data collection in outreach and group settings • Created additional data fields, optional fields & reports • Developed downloading capability • Designed & implemented statewide trainings • Developed partnership with CAPS to provide evaluation TA to support use of forms & data • OA is available for – system technical assistance – consultation for integration of ELI forms into existing data collection efforts 17

Explanation of forms 18

Prevention Interventions to be Evaluated Using ELI • Intervention: A specific activity intended to bring about HIV risk reduction – – – 1 Individual-Level Interventions (ILI) 2 Group-Level Interventions (GLI) 3 Outreach 1 Health Communication 1 Prevention Case Management (PCM) 19

Individual Level Interventions 20

Group Check Sheet (front) 21

Group Check Sheet (back) 22

Overview of ELI Elements • Agency Information • Intervention Info. –Intervention setting –Intervention type –Referrals –Risk reduction materials • Behavioral Info. – Sexual Behavior – Hepatitis & STD History – Substance Use History • Supplemental Info. • Demographics –Gender –Race/ethnicity –Sexual orientation –Matching criteria 23

What are the Matching Criteria & Why are They Important? • • • Date of Birth First Letter of Last Name Gender Race/Ethnicity Zip code 24

Selecting Forms • Discuss form selection • Forms should be decided on by the time intervention is set-up • Some degree of form flexibility upon Office of AIDS approval 25

ELI Data Collection Forms • Not a script! – Provide a framework for a conversation with a client – Optional word use and question order – Use terms familiar to and appropriate for the client 26

Explanation of Reports • Over 50 “canned” ELI reports available – Age by Race by Gender – Missing Fields – Intervention Effectiveness – Referrals by Risk Raw data available 27

Reports 28

Reports • • • Budget Report Referrals by Risk Gender by Race by Age Group Sexual Orientation by Race by Age Group STD by Race by Gender STD Type by Race by Gender Drugs by Race by Age Group Materials Distributed by Risk Missing Fields Intervention List 29

Reports 30

Reports • Garbage in – garbage out • Reports are as rich as the data you enter 31

Challenges 32

Challenges • • Time Staffing Changing CDC requirements Funding – Program cuts 33

Challenges • Individual client data requirements in certain settings • Federal vs. State vs. Local requirements • Information Technology is not always a super highway • Different levels of knowledge • Requires new scope of work format • Intervention set-up: LHJ vs. CBO 34

Challenges • Data Collection – New forms/form integration – Matching Criteria • BIG challenge with some populations (e. g. , clients with immigration issues) 35

Successes 36

Successes • • Increased communication Increased collaboration Increased technical assistance Increased training opportunities 37

Successes • Accurate timely information about HIV prevention interventions • Efficiency increase scope of work review • Intervention Working Group (IWG) 38

Increased focus on quality interventions! NOT quantity. 39

Characteristics of Successful Interventions • • Science based Highly targeted Clearly defined goals and objectives Realistic goals, objectives and implementation timeline • Focused on reducing specific risk behaviors • Adequately trained staff 40

Characteristics of Successful Interventions • • • Sensitive to target population Sustainable resources and support Flexible Part of community plan Culturally, developmentally competent Gender specific 41

Lessons Learned • Secure funding ahead of time • Lengthen and expand the pilot phase • Be prepared for major delays 42

Lessons Learned • Need forms in Spanish • Be prepared for varied knowledge levels – Including data issues • Promote data utilization 43

Lessons Learned: Agency level data use • Clearly identify the intended users of the data • Develop buy-in among data stakeholders • Identify evaluation questions meaningful to the intended users 44

Lessons Learned: TA and Training Issues • Regional trainings require out of county travel • Training design must support new and experienced users of ELI • SOW vs ELI vs Real Life • Site visits often perceived as punitive 45

Next Steps • Outcome evaluation • Continue on site agency level TA on intervention set up • Data Use – More training required – Report manual 46

Next Steps • Suggestions for next steps – What should be the focus of future ELI trainings? – How can we make ELI more agency friendly? 47

Contact info • Shanna Livermore, Office of AIDS slivermo@dhs. ca. gov 916 -449 -5823 • Jeffrey Bernstein, CAPS/CTAP jbernstein@psg. ucsf. edu 1 -800 -983 -5550 48