The wording of a question is extremely important. Researchers strive for objectivity in surveys and, therefore, must be careful not to lead the respondent into giving a desired answer. Unfortunately, the effects of question wording are one of the least understood areas of questionnaire research.
Many investigators have confirmed that slight changes in the way questions are worded can have a significant impact on how people respond. Several authors have reported that minor changes in question wording can produce more than a 25 percent difference in people's opinions.
Several investigators have looked at the effects of modifying adjectives and adverbs. Words like usually, often, sometimes, occasionally, seldom, and rarely are "commonly" used in questionnaires, although it is clear that they do not mean the same thing to all people. Some adjectives have high variability and others have low variability. The following adjectives have highly variable meanings and should be avoided in surveys: a clear mandate, most, numerous, a substantial majority, a minority of, a large proportion of, a significant number of, many, a considerable number of, and several. Other adjectives produce less variability and generally have more shared meaning. These are: lots, almost all, virtually all, nearly all, a majority of, a consensus of, a small number of, not very many of, almost none, hardly any, a couple, and a few.
The Length of a Questionnaire
As a general rule, long questionnaires get less response than short questionnaires. However, some studies have shown that the length of a questionnaire does not necessarily affect response. More important than length is question content. A subject is more likely to respond if they are involved and interested in the research topic. Questions should be meaningful and interesting to the respondent.
Anonymity and Confidentiality
An anonymous study is one in which nobody (not even the researcher) can identify who provided data. It is difficult to conduct an anonymous questionnaire through the mail because of the need to follow-up on non responders. The only way to do a follow-up is to mail another survey or reminder postcard to the entire sample. However, it is possible to guarantee confidentiality, where those conducting the study promise not to reveal the information to anyone. For the purpose of follow-up, identifying numbers on questionnaires are generally preferred to using respondents' names. It is important, however, to explain why the number is there and what it will be used for.
Some studies have shown that response rate is affected by the anonymity/confidentiality policy of a study. Others have reported that responses became more distorted when subjects felt threatened that their identities would become known. Others have found that anonymity and confidentiality issues do not affect response rates or responses.
Qualities of a Good Question
There are good and bad questions. The qualities of a good question are as follows:
•Evokes the truth. Questions must be non-threatening. When a respondent is concerned about the consequences of answering a question in a particular manner, there is a good possibility that the answer will not be truthful. Anonymous questionnaires that contain no identifying information are more likely to produce honest responses than those identifying the respondent. If your questionnaire does contain sensitive items, be sure to clearly state your policy on confidentiality
•Asks for an answer on only one dimension. The purpose of a survey is to find out information. A question that asks for a response on more than one dimension will not provide the information you are seeking. For example, a researcher investigating a new food snack asks "Do you like the texture and flavor of the snack?" If a respondent answers "no", then the researcher will not know if the respondent dislikes the texture or the flavor, or both. Another questionnaire asks, "Were you satisfied with the quality of our food and service?" Again, if the respondent answers "no", there is no way to know whether the quality of the food, service, or both were unsatisfactory. A good question asks for only one "bit" of information.
•Can accommodate all possible answers. Multiple choice items are the most popular type of survey questions because they are generally the easiest for a respondent to answer and the easiest to analyze. Asking a question that does not accommodate all possible responses can confuse and frustrate the respondent. For example, consider the question:
What brand of computer do you own? __
A. IBM PC
B. Apple
Clearly, there are many problems with this question. What if the respondent doesn't own a microcomputer? What if he owns a different brand of computer? What if he owns both an IBM PC and an Apple? There are two ways to correct this kind of problem.
The first way is to make each response a separate dichotomous item on the questionnaire. For example:
Do you own an IBM PC? (circle: Yes or No)
Do you own an Apple computer? (circle: Yes or No)
Another way to correct the problem is to add the necessary response categories and allow multiple responses. This is the preferable method because it provides more information than the previous method.
What brand of computer do you own?
(Check all that apply)
Do not own a computer
IBM PC
Apple
Other
•Has mutually exclusive options. A good question leaves no ambiguity in the mind of the respondent. There should be only one correct or appropriate choice for the respondent to make. An obvious example is:
Where did you grow up? __
A. country
B. farm
C. city
A person who grew up on a farm in the country would not know whether to select choice A or B. This question would not provide meaningful information. Worse than that, it could frustrate the respondent and the questionnaire might find its way to the trash
•Produces variability of responses. When a question produces no variability in responses, we are left with considerable uncertainty about why we asked the question and what we learned from the information. If a question does not produce variability in responses, it will not be possible to perform any statistical analyses on the item. For example
What do you think about this report? __
A. It's the worst report I've read
B. It's somewhere between the worst and best
C. It's the best report I've read
Since almost all responses would be choice B, very little information is learned. Design your questions so they are sensitive to differences between respondents. As another example:
Are you against drug abuse? (circle: Yes or No)
Again, there would be very little variability in responses and we'd be left wondering why we asked the question in the first place.
•Follows comfortably from the previous question. Writing a questionnaire is similar to writing anything else. Transitions between questions should be smooth. Grouping questions that are similar will make the questionnaire easier to complete, and the respondent will feel more comfortable. Questionnaires that jump from one unrelated topic to another feel disjointed and are not likely to produce high response rates.
•Does not presuppose a certain state of affairs. Among the most subtle mistakes in questionnaire design are questions that make an unwarranted assumption. An example of this type of mistake is:
Are you satisfied with your current auto insurance? (Yes or No)
This question will present a problem for someone who does not currently have auto insurance. Write your questions so they apply to everyone. This often means simply adding an additional response category.
Are you satisfied with your current auto insurance?
___ Yes
___No
__ Don't have auto insurance
One of the most common mistaken assumptions is that the respondent knows the correct answer to the question. Industry surveys often contain very specific questions that the respondent may not know the answer to. For example:
What percent of your budget do you spend on direct mail advertising?
Very few people would know the answer to this question without looking it up, and very few respondents will take the time and effort to look it up. If you ask a question similar to this, it is important to understand that the responses are rough estimates and there is a strong likelihood of error.
It is important to look at each question and decide if all respondents will be able to answer it. Be careful not to assume anything. For example, the following question assumes the respondent knows what Proposition 13 is about.
Are you in favor of Proposition 13?
___ Yes
___ No
___ Undecided
If there is any possibility that the respondent may not know the answer to your question, include a "don't know" response category
•Does not imply a desired answer. The wording of a question is extremely important. We are striving for objectivity in our surveys and, therefore, must be careful not to lead the respondent into giving the answer we would like to receive. Leading questions are usually easily spotted because they use negative phraseology. As examples:
Wouldn't you like to receive our free brochure?
Don't you think the Congress is spending too much money?
•Does not use emotionally loaded or vaguely defined words. This is one of the areas overlooked by both beginners and experienced researchers. Quantifying adjectives (e.g., most, least, majority) are frequently used in questions. It is important to understand that these adjectives mean different things to different people.
•Does not use unfamiliar words or abbreviations. Remember who your audience is and write your questionnaire for them. Do not use uncommon words or compound sentences. Write short sentences. Abbreviations are okay if you are absolutely certain that every single respondent will understand their meanings. If there is any doubt at all, do not use the abbreviation. The following question might be okay if all the respondents are accountants, but it would not be a good question for the general public.
What was your AGI last year? ______
.Is not dependent on responses to previous questions. Branching in written questionnaires should be avoided. While branching can be used as an effective probing technique in telephone and face-to-face interviews, it should not be used in written questionnaires because it sometimes confuses respondents. An example of branching is:
1. Do you currently have a life insurance policy? (Yes or No) If no, go to question 3
2. How much is your annual life insurance premium? _________
These questions could easily be rewritten as one question that applies to everyone:
1. How much did you spend last year for life insurance? ______
•Does not ask the respondent to order or rank a series of more than five items. Questions asking respondents to rank items by importance should be avoided. This becomes increasingly difficult as the number of items increases, and the answers become less reliable. This becomes especially problematic when asking respondents to assign a percentage to a series of items. In order to successfully complete this task, the respondent must mentally continue to re-adjust his answers until they total one hundred percent. Limiting the number of items to five will make it easier for the respondent to answer.
Sunday, October 18, 2009
Statistical Surveys
What is Statistical Surveys?
Statistical surveys are used to collect quantitative information in the fields of marketing, political polling, and social science research. A survey may focus on opinions or factual information depending on its purpose, but all surveys involve administering questions to individuals. When the questions are administered by a researcher, the survey is called a structured interview or a researcher administered survey. When the questions are administered by the respondent, the survey is referred to as a questionnaire or a self-administered survey
Tactics used to Increase Response Rates
•Brevity - single page if possible
•Financial incentives
•Prepaid in advance
•Paid at completion
•Non-monetary incentives
•Commodity giveaways (pens, notepads)
•Entry into a lottery, draw or contest
•Discount coupons
•Promise of contribution to charity
•Preliminary notification
•Foot-in-the-door techniques - start with a small inconsequential request
•Personalization of the request - address specific individuals
•Follow-up requests - multiple requests
•Claimed affiliation with universities, research institutions, or charities
•Emotional appeals
•Bids for sympathy
Advantages and Disadvantages of Surveys
Advantages
•It is an efficient way of collecting information from a large number of respondents. Very large samples are possible. Statistical techniques can be used to determine validity, reliability, and statistical significance.
•Surveys are flexible in the sense that a wide range of information can be collected. They can be used to study attitudes, values, beliefs, and past behaviors.
•Because they are standardized, they are relatively free from several types of errors.
•They are relatively easy to administer.
•There is an economy in data collection due to the focus provided by standardized questions. Only questions of interest to the researcher are asked, recorded, codified, and analyzed. Time and money is not spent on tangential questions.
•Cheaper to run.
Disadvantages
•They depend on subjects’ motivation, honesty, memory, and ability to respond. Subjects may not be aware of their reasons for any given action. They may have forgotten their reasons. They may not be motivated to give accurate answers, in fact, they may be motivated to give answers that present themselves in a favorable light.
•Structured surveys, particularly those with closed ended questions, may have low validity when researching affective variables.
•Although the chosen survey individuals are often a random sample, errors due to non response may exist. That is, people who choose to respond on the survey may be different from those who do not respond, thus biasing the estimates.
•Survey question answer-choices could lead to vague data sets because at times they are relative only to a personal abstract notion concerning "strength of choice". For instance the choice "moderately agree" may mean different things to different subjects, and to anyone interpreting the data for correlation. Even yes or no answers are problematic because subjects may for instance put "no" if the choice "only once" is not available.
Characteristics of Researcher-administered Surveys
•Fewer misunderstood questions and inappropriate responses.
•Fewer incomplete responses.
•Generally higher response rates and better information on no response, but...
•Respondents may be unwilling to discuss sensitive topics with a stranger.
•Greater control over the environment that the survey is administered in.
•Additional information can be collected from respondent.
•Subject to interviewer bias (e.g. answers influenced by desire to impress interviewer).
•Generally expensive/time-consuming to run.
Characteristics of Self-administered Surveys
•Respondents are more likely to stop participating mid-way through the survey (drop-offs).
•Respondents cannot ask for clarification.
•Low response rate in some modes.
•Often respondents returning survey represent extremes of the population - skewed responses (consequence of low response rates).
•Allows shy respondents to answer sensitive questions in private.
•No interviewer intervention available for probing or explanation.
•Respondents can read the whole questionnaire before answering any questions.
•Free of interviewer bias.
Modes of Data Collection in Survey
There are six common ways to get information. These are: literature searches, talking with people, focus groups, personal interviews, telephone surveys, and mail surveys.
•A literature search involves reviewing all readily available materials. These materials can include internal company information, relevant trade publications, newspapers, magazines, annual reports, company literature, on-line data bases, and any other published materials. It is a very inexpensive method of gathering information, although it generally does not yield timely information. Literature searches take between one and eight weeks.
•Talking with people is a good way to get information during the initial stages of a research project. It can be used to gather information that is not publicly available, or that is too new to be found in the literature. Examples might include meetings with prospects, customers, suppliers, and other types of business conversations at trade shows, seminars, and association meetings. Although often valuable, the information has questionable validity because it is highly subjective and might not be representative of the population.
•A focus group is used as a preliminary research technique to explore people’s ideas and attitudes. It is often used to test new approaches (such as products or advertising), and to discover customer concerns. A group of 6 to 20 people meet in a conference-room-like setting with a trained moderator. The room usually contains a one-way mirror for viewing, including audio and video capabilities. The moderator leads the group's discussion and keeps the focus on the areas you want to explore. Focus groups can be conducted within a couple of weeks and cost between two and three thousand dollars. Their disadvantage is that the sample is small and may not be representative of the population in general.
•Personal interviews are a way to get in-depth and comprehensive information. They involve one person interviewing another person for personal or detailed information. Personal interviews are very expensive because of the one-to-one nature of the interview ($50+ per interview). Typically, an interviewer will ask questions from a written questionnaire and record the answers verbatim.
Sometimes, the questionnaire is simply a list of topics that the research wants to discuss with an industry expert. Personal interviews (because of their expense) are generally used only when subjects are not likely to respond to other survey methods.
•Telephone surveys are the fastest method of gathering information from a relatively large sample (100-400 respondents). The interviewer follows a prepared script that is essentially the same as a written questionnaire. However, unlike a mail survey, the telephone survey allows the opportunity for some opinion probing. Telephone surveys generally last less than ten minutes. Typical costs are between four and six thousand dollars and they can be completed in two to four weeks.
•Mail surveys are a cost effective method of gathering information. They are ideal for large sample sizes, or when the sample comes from a wide geographic area. They cost a little less than telephone interviews, however, they take over twice as long to complete (eight to twelve weeks). Because there is no interviewer, there is no possibility of interviewer bias. The main disadvantage is the inability to probe respondents for more detailed information.
•E-mail and internet surveys are relatively new and little is known about the effect of sampling bias in internet surveys. While it is clearly the most cost effective and fastest method of distributing a survey, the demographic profile of the internet user does not represent the general population, although this is changing. Before doing an e-mail or internet survey, carefully consider the effect that this bias might have on the results.
Questionnaire Design - General Considerations
Most problems with questionnaire analysis can be traced back to the design phase of the project. Well-defined goals are the best way to assure a good questionnaire design. When the goals of a study can be expressed in a few clear and concise sentences, the design of the questionnaire becomes considerably easier. The questionnaire is developed to directly address the goals of the study.
One of the best ways to clarify your study goals is to decide how you intend to use the information. Do this before you begin designing the study. This sounds obvious, but many researchers neglect this task. Why do research if the results will not be used?
Be sure to commit the study goals to writing. Whenever you are unsure of a question, refer to the study goals and a solution will become clear. Ask only questions that directly address the study goals. Avoid the temptation to ask questions because it would be "interesting to know".
As a general rule, with only a few exceptions, long questionnaires get less response than short questionnaires. Keep your questionnaire short. In fact, the shorter the better. Response rate is the single most important indicator of how much confidence you can place in the results. A low response rate can be devastating to a study. Therefore, you must do everything possible to maximize the response rate. One of the most effective methods of maximizing response is to shorten the questionnaire.
If your survey is over a few pages, try to eliminate questions. Many people have difficulty knowing which questions could be eliminated. For the elimination round, read each question and ask, "How am I going to use this information?" If the information will be used in a decision-making process, then keep the question... it's important. If not, throw it out.
One important way to assure a successful survey is to include other experts and relevant decision-makers in the questionnaire design process. Their suggestions will improve the questionnaire and they will subsequently have more confidence in the results.
Formulate a plan for doing the statistical analysis during the design stage of the project. Know how every question will be analyzed and be prepared to handle missing data. If you cannot specify how you intend to analyze a question or use the information, do not use it in the survey.
Make the envelope unique. We all know how important first impressions are. The same holds true for questionnaires. The respondent's first impression of the study usually comes from the envelope containing the survey. The best envelopes (i.e., the ones that make you want to see what's inside) are colored, hand-addressed and use a commemorative postage stamp. Envelopes with bulk mail permits or gummed labels are perceived as unimportant. This will generally be reflected in a lower response rate.
Provide a well-written cover letter. The respondent's next impression comes from the cover letter. The importance of the cover letter should not be underestimated. It provides your best chance to persuade the respondent to complete the survey.
Give your questionnaire a title that is short and meaningful to the respondent. A questionnaire with a title is generally perceived to be more credible than one without.
Include clear and concise instructions on how to complete the questionnaire. These must be very easy to understand, so use short sentences and basic vocabulary. Be sure to print the return address on the questionnaire itself (since questionnaires often get separated from the reply envelopes).
Begin with a few non-threatening and interesting items. If the first items are too threatening or "boring", there is little chance that the person will complete the questionnaire. People generally look at the first few questions before deciding whether or not to complete the questionnaire. Make them want to continue by putting interesting questions first.
Use simple and direct language. The questions must be clearly understood by the respondent. The wording of a question should be simple and to the point. Do not use uncommon words or long sentences. Make items as brief as possible. This will reduce misunderstandings and make the questionnaire appear easier to complete. One way to eliminate misunderstandings is to emphasize crucial words in each item by using bold, italics or underlining.
Leave adequate space for respondents to make comments. One criticism of questionnaires is their inability to retain the "flavor" of a response. Leaving space for comments will provide valuable information not captured by the response categories. Leaving white space also makes the questionnaire look easier and this increases response.
Place the most important items in the first half of the questionnaire. Respondents often send back partially completed questionnaires. By putting the most important items near the beginning, the partially completed questionnaires will still contain important information.
Hold the respondent's interest. We want the respondent to complete our questionnaire. One way to keep a questionnaire interesting is to provide variety in the type of items used. Varying the questioning format will also prevent respondents from falling into "response sets". At the same time, it is important to group items into coherent categories. All items should flow smoothly from one to the next.
If a questionnaire is more than a few pages and is held together by a staple, include some identifying data on each page (such as a respondent ID number). Pages often accidentally separate.
Provide incentives as a motivation for a properly completed questionnaire. What does the respondent get for completing your questionnaire? Altruism is rarely an effective motivator. Attaching a dollar bill to the questionnaire works well. If the information you are collecting is of interest to the respondent, offering a free summary report is also an excellent motivator. Whatever you choose, it must make the respondent want to complete the questionnaire.
Use professional production methods for the questionnaire--either desktop publishing or typesetting and key lining. Be creative. Try different colored inks and paper. The object is to make your questionnaire stand out from all the others the respondent receives.
Make it convenient. The easier it is for the respondent to complete the questionnaire the better. Always include a self-addressed postage-paid envelope. Envelopes with postage stamps get better response than business reply envelopes (although they are more expensive since you also pay for the non-respondents).
The final test of a questionnaire is to try it on representatives of the target audience. If there are problems with the questionnaire, they almost always show up here. If possible, be present while a respondent is completing the questionnaire and tell her that it is okay to ask you for clarification of any item. The questions she asks are indicative of problems in the questionnaire (i.e., the questions on the questionnaire must be without any ambiguity because there will be no chance to clarify a question when the survey is mailed).
Statistical surveys are used to collect quantitative information in the fields of marketing, political polling, and social science research. A survey may focus on opinions or factual information depending on its purpose, but all surveys involve administering questions to individuals. When the questions are administered by a researcher, the survey is called a structured interview or a researcher administered survey. When the questions are administered by the respondent, the survey is referred to as a questionnaire or a self-administered survey
Tactics used to Increase Response Rates
•Brevity - single page if possible
•Financial incentives
•Prepaid in advance
•Paid at completion
•Non-monetary incentives
•Commodity giveaways (pens, notepads)
•Entry into a lottery, draw or contest
•Discount coupons
•Promise of contribution to charity
•Preliminary notification
•Foot-in-the-door techniques - start with a small inconsequential request
•Personalization of the request - address specific individuals
•Follow-up requests - multiple requests
•Claimed affiliation with universities, research institutions, or charities
•Emotional appeals
•Bids for sympathy
Advantages and Disadvantages of Surveys
Advantages
•It is an efficient way of collecting information from a large number of respondents. Very large samples are possible. Statistical techniques can be used to determine validity, reliability, and statistical significance.
•Surveys are flexible in the sense that a wide range of information can be collected. They can be used to study attitudes, values, beliefs, and past behaviors.
•Because they are standardized, they are relatively free from several types of errors.
•They are relatively easy to administer.
•There is an economy in data collection due to the focus provided by standardized questions. Only questions of interest to the researcher are asked, recorded, codified, and analyzed. Time and money is not spent on tangential questions.
•Cheaper to run.
Disadvantages
•They depend on subjects’ motivation, honesty, memory, and ability to respond. Subjects may not be aware of their reasons for any given action. They may have forgotten their reasons. They may not be motivated to give accurate answers, in fact, they may be motivated to give answers that present themselves in a favorable light.
•Structured surveys, particularly those with closed ended questions, may have low validity when researching affective variables.
•Although the chosen survey individuals are often a random sample, errors due to non response may exist. That is, people who choose to respond on the survey may be different from those who do not respond, thus biasing the estimates.
•Survey question answer-choices could lead to vague data sets because at times they are relative only to a personal abstract notion concerning "strength of choice". For instance the choice "moderately agree" may mean different things to different subjects, and to anyone interpreting the data for correlation. Even yes or no answers are problematic because subjects may for instance put "no" if the choice "only once" is not available.
Characteristics of Researcher-administered Surveys
•Fewer misunderstood questions and inappropriate responses.
•Fewer incomplete responses.
•Generally higher response rates and better information on no response, but...
•Respondents may be unwilling to discuss sensitive topics with a stranger.
•Greater control over the environment that the survey is administered in.
•Additional information can be collected from respondent.
•Subject to interviewer bias (e.g. answers influenced by desire to impress interviewer).
•Generally expensive/time-consuming to run.
Characteristics of Self-administered Surveys
•Respondents are more likely to stop participating mid-way through the survey (drop-offs).
•Respondents cannot ask for clarification.
•Low response rate in some modes.
•Often respondents returning survey represent extremes of the population - skewed responses (consequence of low response rates).
•Allows shy respondents to answer sensitive questions in private.
•No interviewer intervention available for probing or explanation.
•Respondents can read the whole questionnaire before answering any questions.
•Free of interviewer bias.
Modes of Data Collection in Survey
There are six common ways to get information. These are: literature searches, talking with people, focus groups, personal interviews, telephone surveys, and mail surveys.
•A literature search involves reviewing all readily available materials. These materials can include internal company information, relevant trade publications, newspapers, magazines, annual reports, company literature, on-line data bases, and any other published materials. It is a very inexpensive method of gathering information, although it generally does not yield timely information. Literature searches take between one and eight weeks.
•Talking with people is a good way to get information during the initial stages of a research project. It can be used to gather information that is not publicly available, or that is too new to be found in the literature. Examples might include meetings with prospects, customers, suppliers, and other types of business conversations at trade shows, seminars, and association meetings. Although often valuable, the information has questionable validity because it is highly subjective and might not be representative of the population.
•A focus group is used as a preliminary research technique to explore people’s ideas and attitudes. It is often used to test new approaches (such as products or advertising), and to discover customer concerns. A group of 6 to 20 people meet in a conference-room-like setting with a trained moderator. The room usually contains a one-way mirror for viewing, including audio and video capabilities. The moderator leads the group's discussion and keeps the focus on the areas you want to explore. Focus groups can be conducted within a couple of weeks and cost between two and three thousand dollars. Their disadvantage is that the sample is small and may not be representative of the population in general.
•Personal interviews are a way to get in-depth and comprehensive information. They involve one person interviewing another person for personal or detailed information. Personal interviews are very expensive because of the one-to-one nature of the interview ($50+ per interview). Typically, an interviewer will ask questions from a written questionnaire and record the answers verbatim.
Sometimes, the questionnaire is simply a list of topics that the research wants to discuss with an industry expert. Personal interviews (because of their expense) are generally used only when subjects are not likely to respond to other survey methods.
•Telephone surveys are the fastest method of gathering information from a relatively large sample (100-400 respondents). The interviewer follows a prepared script that is essentially the same as a written questionnaire. However, unlike a mail survey, the telephone survey allows the opportunity for some opinion probing. Telephone surveys generally last less than ten minutes. Typical costs are between four and six thousand dollars and they can be completed in two to four weeks.
•Mail surveys are a cost effective method of gathering information. They are ideal for large sample sizes, or when the sample comes from a wide geographic area. They cost a little less than telephone interviews, however, they take over twice as long to complete (eight to twelve weeks). Because there is no interviewer, there is no possibility of interviewer bias. The main disadvantage is the inability to probe respondents for more detailed information.
•E-mail and internet surveys are relatively new and little is known about the effect of sampling bias in internet surveys. While it is clearly the most cost effective and fastest method of distributing a survey, the demographic profile of the internet user does not represent the general population, although this is changing. Before doing an e-mail or internet survey, carefully consider the effect that this bias might have on the results.
Questionnaire Design - General Considerations
Most problems with questionnaire analysis can be traced back to the design phase of the project. Well-defined goals are the best way to assure a good questionnaire design. When the goals of a study can be expressed in a few clear and concise sentences, the design of the questionnaire becomes considerably easier. The questionnaire is developed to directly address the goals of the study.
One of the best ways to clarify your study goals is to decide how you intend to use the information. Do this before you begin designing the study. This sounds obvious, but many researchers neglect this task. Why do research if the results will not be used?
Be sure to commit the study goals to writing. Whenever you are unsure of a question, refer to the study goals and a solution will become clear. Ask only questions that directly address the study goals. Avoid the temptation to ask questions because it would be "interesting to know".
As a general rule, with only a few exceptions, long questionnaires get less response than short questionnaires. Keep your questionnaire short. In fact, the shorter the better. Response rate is the single most important indicator of how much confidence you can place in the results. A low response rate can be devastating to a study. Therefore, you must do everything possible to maximize the response rate. One of the most effective methods of maximizing response is to shorten the questionnaire.
If your survey is over a few pages, try to eliminate questions. Many people have difficulty knowing which questions could be eliminated. For the elimination round, read each question and ask, "How am I going to use this information?" If the information will be used in a decision-making process, then keep the question... it's important. If not, throw it out.
One important way to assure a successful survey is to include other experts and relevant decision-makers in the questionnaire design process. Their suggestions will improve the questionnaire and they will subsequently have more confidence in the results.
Formulate a plan for doing the statistical analysis during the design stage of the project. Know how every question will be analyzed and be prepared to handle missing data. If you cannot specify how you intend to analyze a question or use the information, do not use it in the survey.
Make the envelope unique. We all know how important first impressions are. The same holds true for questionnaires. The respondent's first impression of the study usually comes from the envelope containing the survey. The best envelopes (i.e., the ones that make you want to see what's inside) are colored, hand-addressed and use a commemorative postage stamp. Envelopes with bulk mail permits or gummed labels are perceived as unimportant. This will generally be reflected in a lower response rate.
Provide a well-written cover letter. The respondent's next impression comes from the cover letter. The importance of the cover letter should not be underestimated. It provides your best chance to persuade the respondent to complete the survey.
Give your questionnaire a title that is short and meaningful to the respondent. A questionnaire with a title is generally perceived to be more credible than one without.
Include clear and concise instructions on how to complete the questionnaire. These must be very easy to understand, so use short sentences and basic vocabulary. Be sure to print the return address on the questionnaire itself (since questionnaires often get separated from the reply envelopes).
Begin with a few non-threatening and interesting items. If the first items are too threatening or "boring", there is little chance that the person will complete the questionnaire. People generally look at the first few questions before deciding whether or not to complete the questionnaire. Make them want to continue by putting interesting questions first.
Use simple and direct language. The questions must be clearly understood by the respondent. The wording of a question should be simple and to the point. Do not use uncommon words or long sentences. Make items as brief as possible. This will reduce misunderstandings and make the questionnaire appear easier to complete. One way to eliminate misunderstandings is to emphasize crucial words in each item by using bold, italics or underlining.
Leave adequate space for respondents to make comments. One criticism of questionnaires is their inability to retain the "flavor" of a response. Leaving space for comments will provide valuable information not captured by the response categories. Leaving white space also makes the questionnaire look easier and this increases response.
Place the most important items in the first half of the questionnaire. Respondents often send back partially completed questionnaires. By putting the most important items near the beginning, the partially completed questionnaires will still contain important information.
Hold the respondent's interest. We want the respondent to complete our questionnaire. One way to keep a questionnaire interesting is to provide variety in the type of items used. Varying the questioning format will also prevent respondents from falling into "response sets". At the same time, it is important to group items into coherent categories. All items should flow smoothly from one to the next.
If a questionnaire is more than a few pages and is held together by a staple, include some identifying data on each page (such as a respondent ID number). Pages often accidentally separate.
Provide incentives as a motivation for a properly completed questionnaire. What does the respondent get for completing your questionnaire? Altruism is rarely an effective motivator. Attaching a dollar bill to the questionnaire works well. If the information you are collecting is of interest to the respondent, offering a free summary report is also an excellent motivator. Whatever you choose, it must make the respondent want to complete the questionnaire.
Use professional production methods for the questionnaire--either desktop publishing or typesetting and key lining. Be creative. Try different colored inks and paper. The object is to make your questionnaire stand out from all the others the respondent receives.
Make it convenient. The easier it is for the respondent to complete the questionnaire the better. Always include a self-addressed postage-paid envelope. Envelopes with postage stamps get better response than business reply envelopes (although they are more expensive since you also pay for the non-respondents).
The final test of a questionnaire is to try it on representatives of the target audience. If there are problems with the questionnaire, they almost always show up here. If possible, be present while a respondent is completing the questionnaire and tell her that it is okay to ask you for clarification of any item. The questions she asks are indicative of problems in the questionnaire (i.e., the questions on the questionnaire must be without any ambiguity because there will be no chance to clarify a question when the survey is mailed).
Subscribe to:
Posts (Atom)