Kensington Research Institute

Services -- Part 2

Implementation

When you are ready to launch your new project, or if your project has already begun, KRI can provide a wide array of services to maximize the quality of the research and evaluation work that you do.  Among the many project implementation-related services we offer, KRI staff has had considerable experience with questionnaire design, reviewing questionnaires for thoroughness and selecting or constructing new scale measures, developing coding schema and codebooks, conducting interviewer training sessions, implementing workshops focusing on such topics as enhancing follow-up rates, and performing a variety of data quality assurance tasks.  For more information on these topics, please see below.

Questionnaire Design

            Relying upon our many years of experience in questionnaire design, 

              at KRI, we can help you develop the instrumentation you need in order 

               to conduct your research and evaluation studies.  This might entail 

              helping you to develop a questionnaire "from scratch" (or just developing 

              it in its entirety for you).  We can also help you to identify specific 

              modules to add to an existing questionnaire to make it more

              comprehensive.  For example, perhaps you have a data collection 

              instrument that you already use, but want to add to it so that you can 

              capture information on additional topics.  We could identify for you any 

              existing standardized scales that have been created and tested by 

              others to examine this subject, or we could develop new questionnaire 

              modules and/or scale measures for you, specific to your study and 

              research needs.

               If, upon completion of the development of this instrument, you decided 

              that you wanted to pre-test and/or pilot test the newly-designed

              questionnaire, KRI could also assist you in designing and implementing 

              the pre-testing or pilot testing procedures.  This could include training 

              your research team and interviewers to administer the new 

              questionnaire, designing the pre-test or pilot test, actually conducting the

              pre-test or pilot test with whatever persons you wanted to involve,

              evaluating the results of the pre-test or pilot test, making 

              recommendations (as appropriate) for the improvement and/or

              modification of the questionnaire, and modifying the data collection

              instrument to develop its final form for use "in the field."

 

Questionnaire / Instrument Review

            Perhaps you have already developed the questionnaire that you plan

              to use.  Before you begin implementing it "in the field," it is often a good

              idea to have an outsider who is an expert in the field of questionnaire

              construction review your instrument.  This can be an invaluable process 

              that can identify shortcomings related to the clarity of the questions you 

              are asking, assess the level of easiness-of-understanding for your 

              instruments' questions, and discover hidden or unanticipated meanings 

              that might be inferred by study participants who are asked questions 

              worded in a certain way.  Asking questions that are too difficult or 

              complex can lead to problems of comprehension, which can only result 

              in bad or unusable data.  Phrasing questions in a way that could lead 

              them to be interpreted in different ways by different persons is also a 

              hidden danger in conducting survey-type research.  At KRI, our 

              knowledgeable staff can review your questionnaires to make sure that 

              these potential pitfalls are identified and remedied before they cause 

              data-related problems in your study.

              Having an outside expert review your questionnaire before you begin 

              your work "in the field" can also be a good way to identify analysis-related

              limitations in the way that study participants are permitted to respond to 

              various questions. For example, very different things can--and cannot--be 

              done analytically if people are permitted to respond "never--rarely--

              sometimes--usually--always" to a question, versus being asked to respond 

              to that same question "less than once a year--less than once a month--

              less than once a week--less than once a day--daily."  Therefore, it is 

              important to choose a way of responding to questions that best suits 

              your research needs.  Using our broad basis of experience and 

              knowledge in questionnaire design, we at KRI can help you to make the 

              decision that is right for you, based on what you are trying to accomplish 

              with your particular research or evaluation project.  

              Pre-implementation questionnaire review can also be a valuable process 

              that can help you make sure that key "pieces" of information or key 

              topics of interest and importance to your project have not been omitted 

              from your questionnaire inadvertently.  The skilled research staff at KRI can 

              help with this task, to make it possible for you to get your project off to

              the best possible start.

 

Developing a Coding Schema and a Codebook

            How you decide to allow people to answer the questions you pose to them--that

             is, how you decide to code the data that you collect--can have a profound impact

             upon the types of research and evaluation questions that you can address and

             the types of answers that you can derive from your data.  Giving careful

             consideration to how you want your questions answered is just as important as

             giving careful consideration to how you want those very same questions asked. 

             KRI can be of great assistance in this process.

             In addition, it is usually very helpful to develop a codebook (or a series of

             codebooks if multiple data collection instruments are used in a particular project)

             anytime you are planning to capture survey-type data.  A good codebook not only

             lists the questions as they appear on the questionnaire used in the field, but also

             such things as previous question wordings (appropriate if more than one version

             of a questionnaire has been used on a particular project), response options,

             numeric or alpha-numeric codes used for the various response options, special

             instructions given to interviewers, special instructions that data entry personnel

             should follow, variable names, variable labels, and sometimes other information

             as well.  Ideally, a completed codebook can be a useful not only to the

             interviewers, but also to the persons who conduct the data entry, statistical

             analysis, report writing, among others.  At KRI, we have had extensive experience

             developing and using research codebooks and would be happy to help you

             create those needed to complement the work you are doing.

      

Interviewer Training

            An evaluation/research study can only be as good as the data collected.  Part of

             the process of collecting good data requires the use of good measures, reliable

             and properly-worded questions, and a methodologically-sound questionnaire.  

             Another key part of the process necessitates the use of properly-trained

             interviewers.  One thing that we always point out to our clients is that, just

             because someone has conducted interviews before does not necessarily mean

             that that individual is automatically well-trained to conduct interviews for your

             project.  Every evaluation and research study is different, with its own

             project-specific challenges that must be faced and handled during the interviewing

             process.  Sometimes, these challenges are the result of the study population

             being targeted in your work.  Other times, these challenges are the result of the

             types of questions you are asking.  At still other times, these challenges may be

             the result of where your interviews are being conducted (project office versus "in

             the field," for example), when your interviews are being held (mornings when

             some people may be sleepy versus evenings when people may be distracted by

             their day's events), or certain characteristics of the person conducting your

             interviews (some people respond differently to men versus women, others may

             feel differently about answering questions posed by a younger versus an older

             person).  

             To overcome potential obstacles--ideally, to prevent them from becoming

             problems in the first place--it is always wise to train your interviewers before they

             begin collecting data for you.  KRI can help you with this task.  Interviewer training

             can take many forms.  At its most basic form, there are certain ways that

             questions should and should not be asked during an interview.  Instructing

             interviewers about proper interviewing procedures and providing them with tips

             for asking questions effectively is one basic but very element of many interviewer

             training sessions.

             Oftentimes, it is wise to give interviewers a significant amount of rehearsal time

             with their questionnaire(s), so that they can become intimately familiar with the

             instrument(s) that they are expected to use.  On a project involving the

             implementation of a lengthy questionnaire, this may take one or several weeks.  

             During this rehearsal period, it is important that your interviewers learn how to

             answer study participants' queries about the questions being asked of them, that

             they can learn how to probe for answers whenever specific responses are not

             forthcoming, and so forth.  

            A good interviewer training process often includes modules focusing on how to

            obtain accurate information from people who may be reluctant initially to

            disclosing sensitive information, how to look for inconsistent answers given during

            an interview (and when logically-inconsistent answers are given, how to "fix" them

            so that the inconsistencies are removed), and how to keep study participants

            interested in the interview, so that they willingly give thoughtful and accurate

            information throughout the entire duration of the interview.

            KRI can help you with all of these tasks.  If you like, we can conduct full

            interviewer trainings for you, taking your staff members from their very first contact

            with a particular data collection instrument to their last day of rehearsal and

            training.  If you prefer, we could also come to your office or project locale and give

            your staff a one-day or a two-day workshop about the proper implementation of a

            questionnaire, including coverage of the "DOs and DON'Ts" of good interviewing.  

             One other interviewer training-related service that KRI has provided on several

            occasions--and one that we strongly recommend for longer-term research /

            evaluation projects--is the booster training.  Oftentimes, even though staff

            members have undergone extensive and effective interviewer training, over time,

            they begin to experience what is known in scientific circles as "interviewer drift"--a

            process by which interviewers develop their own individualized interviewing styles,

            that, as time passes, take them farther and farther away from optimal interviewing

            practices.  One effective way of identifying and rectifying this is to provide periodic

            booster training sessions.  These sessions typically take the form of half-day,

            one-day, or two-day trainings, and they can be done one-on-one with individual

            interviewers or in group settings with your entire interviewing team in attendance. 

            During these booster trainings, the rules of good interviewing are reviewed, mock

            interviews are undertaken and critiqued, and interview rehearsals are conducted

            so that appropriate feedback can be given to the interviewers.  Such booster

            training sessions are a simple and effective way of minimizing interviewer drift, and

            they are an excellent way of enhancing the quality of your project's data

            collection.

            Another way in which KRI can help you with your interviewing needs is to devise

            and implement an interviewer certification process.  This is often done when the

            person heading up a research/evaluation study wants to make sure that each

            interviewer collecting his/her project's data meets a specified minimum standard

            of interviewing proficiency.  To earn interviewer certification status, after the

            completion of the required training regimen, an interviewer ordinarily would have

            an interview observed first-hand (ideally) or reviewed on videotape or audio-tape. 

            The purpose of the observation is for the observer (who would be one of the

            project leaders or a KRI staff member assigned to conducting the certification

            process) to witness an interview, note the interviewer's strengths and weaknesses,

            check the completed data collection form(s) for thoroughness and accuracy, and

            make sure that the interview meets the pre-determined minimum proficiency

            criteria.  Written comments are provided to interviewers after their interviews are

            monitored and evaluated, giving them specific feedback about their question

            reading skills, clarity, fielding of queries, handling of inconsistencies, and overall

            quality of the submitted written document.  The written critiques end by indicating

            whether the person has earned certification status (indicating that he/she has met

            the standards of proficiency established for that study) or whether the interviewer

            should submit another interview for review and critique (indicating that the

            minimum proficiency standards for certification have not yet been met).

Workshops on Improving Follow-up Rates

            Many research, intervention, and evaluation projects collect data at the time that

             people begin participating and then afterwards, at some pre-determined follow-up

             point, so that changes occurring in beliefs, attitudes, and/or behaviors between

             Time 1 and Time 2 can be measured.  It is very important to have high follow-up

             rates whenever such Time-1-to-Time-2 changes are to be evaluated.  Recent

             published evidence by Michael Dennis and colleagues, for example, has shown

             that it is difficult to draw conclusions about changes over time if follow-up rates

             are below 70%, and that it is far safer (from a scientific standpoint) to do so if

             follow-up rates are 80% or greater.  Finding people for those follow-up interviews

             can be difficult, though, if proper procedures are not put into place when the

             study begins, and if interviewers/trackers do not know a variety of ways (some

             might call them "tricks of the trade") to locate and keep track of the people they

             wish to reinterview.  

             If you have a plan in place for obtaining your follow-up interviews, KRI can review

             your procedures and make recommendations regarding things you might

             consider doing to enhance your follow-up rates.  If you do not have a plan in

             place, we can work with you to develop a strategy that can be used to track your

             study participants and maximize your chances of finding them when you need to

             reinterview them.  We can also conduct a training with your key staff members

             regarding this process, including interactive sessions designed to bolster their

             effectiveness and increase your follow-up rates. 

 

Data Quality Assurances

            Even when all of the proper training is provided to project personnel, errors

             routinely happen in the process of collecting and entering data.  Before any

             statistical analysis is undertaken, it is always prudent to have someone who is

             detail-oriented review the data in search of errors that have occurred. 

             Sometimes, these errors take the form of a transcription mistake, in which the

             person who entered the data from the questionnaire into the computer or data

             entry program typed in an incorrect response or number.  In other instances,

             errors may entail omitting a question or a series of questions, or having a study

             participant answer a question or a series of questions that were supposed to be

             skipped based on his/her previous responses.  Another common data-related

             problem is having too many "don't know" or "refused" responses, which more

             often than not indicate an insufficiently-motivated interviewer or interviewee rather

             than "real" responses indicating that the person truly does not know the answer or

             refused to respond to the interviewer's question.   Still another data-related

             problem occurs when an interviewer inadvertently accepts two logically-

             incompatible answers without probing for clarification so that the inconsistency

             could be identified.  

             All of these data-related problems occur easily and, therefore, are commonplace

             when quantitative data collection is involved in a research, intervention, or

             evaluation study.  This is true even when staff members are highly trained and

             trying their best to be careful during the data collection process.  KRI can help you

             with these matters in a number of ways.  First, we can conduct spot-checking of

             your questionnaires and data files to look for transcription errors.  An extra pair

             of eyes never hurts!  Second, we can review your questionnaires and/or data files

             to identify any items that have missing data.  Similarly, we can also review your

             questionnaires and data files to identify any questions that have too many

             unusable answers like "don't know" or "refused," or questionnaire items for which

             implausible answers are being given.  Not only can KRI bring these types of

             problems to your attention, but we could also work with you to conduct briefing

             sessions and/or retraining sessions to alert your staff members to the types of

             data-related problems that are occurring.  Along similar lines, KRI could also

             conduct a systematic logical consistency check of your data, to make sure that all

             responses given by an individual appear reasonable and, at a minimum, are

             logically consistent with one another.