So you want to write a questionnaire (July 12, 2002)

Dear Professor Mean, I need to write a questionnaire for a research study I am conducting. Can you help me write it? -- Cautious Carmen

Dear Cautious,

If I wrote the questionnaire, it would include a bad joke on every page.

Short answer

You need to think about several issues while writing a questionnaire:

Also make sure that you do some pilot testing of your questionnaire.

I'll presume, for the most part, that you are giving a questionnaires to individual patients, but you can also send them out to groups and organizations as well.

What is the purpose of your questionnaire?

You need to identify the purpose of your survey. Are you trying to identify

or some combination of the above.

Who are you sampling from and who do you want to generalize these results to. Will you make any extrapolations from this survey?

Will you collect the data

Have you considered a focus group as an alternative method for data collection?

What level of anonymity can you provide?

You should provide the greatest degree of anonymity possible and you should inform your patients what level of anonymity you can provide.

For may questionnaires, you will publish only aggregate results; individual responses will not be reported. If you have to link the questionnaire data to a medical record or other source of information, then you need to inform your patients.

Sometimes you need to track the survey in order to find out who to send a follow up reminder notice to. This could be done with the use of a code number on the survey, but if you can, you should assure the patient that this code will not be used beyond the use of reminder notices.

I am starting to write up a web page about privacy concerns in research.

You need to identify any sensitive questions. Some examples include

What is sensitive may depend on what group you are asking. Questions about smoking and alcohol consumption might be more threatening, if your population is a group of pregnant women.

Ask yourself if the disclosure of this information might embarrass or harm the respondent. If it can, then you need to take special precautions.

How will you minimize non response?

Some of the people you send your questionnaire to will not receive it. Some of them will not return the questionnaire. And those who do respond, may not respond to all the questions. All of this can cause a serious bias in your data analysis.

Think first about motivation. Why would anyone take the time to fill out and return your questionnaire? You need to give them some incentive.

Also be sure to avoid common demotivators.

If possible, use follow up reminders by phone, email, or postal mail to those who do not respond by the deadline. These reminders can sometimes raise concerns about anonymity, so be careful about this. Use coded numbers on the surveys to track who has responded and let them know that the link between the codes and any personal identifiers will be destroyed.

The best way to minimize the number of non respondents is to make the survey clean, simple, and easy to respond to.

Finally, be sure choose an appropriate language level. For many questionnaires, you should write at a fifth grade reading level.

In spite of all this some people will not respond. If you can get some abbreviated information from them, such as demographics or their reasons for not participating, that might help. It might determine a good statistical adjustment for your data. Even if you can't adjust for it, this information might help you determine the direction and severity of any bias caused by non response.

Are you asking the right questions?

Use standardized questions and scales whenever you can. These standards were developed and tested over a long period of time, so you know how they will behave. By using standardized questions, you also make it easier for anyone who might incorporate your research into a systematic review or meta-analysis.

A good example of a standardized scale is the Burns Anxiety Inventory. This is a series of 33 questions about anxious feelings, anxious thoughts, and physical symptoms. Here are six of the items:

By asking a wide range of questions about anxiety, you are helping to get an accurate assessment of anxiety, especially for those patients who might show anxiety in some ways but not in others.

Lack of standards can cause problems. Jadad and Gagliardi (1998) criticize scales used to rate web sites providing health information. There were too many of them, most of them did not present any justification for their

When you are using categories, use the same categories that others use. Birthweights, for example, as classified as low (LBW) if less than 2500 grams, very low (VLBW) if less than 1500 grams, and extremely low (ELBW) if less than 1000 grams. 

Running a pilot study of the questionnaire

Pilot test your questionnaire.

-- Dillman, page 143.

Here are some other issues to examine during a pilot.

Try to estimate the resources you need to conduct this questionnaire.

Summary

[This section is not yet available.]

Thank you for filling out this survey. We don't have enough money to include a pre-printed envelope. Be sure to use extra postage, since the survey weighs more than one ounce. We're not sure how we will use this data and maybe we'll disclose this information to other researchers. This survey hasn't had any pilot testing, so if we goof up badly, you might have to fill out a better one later. No one has used the survey form before, so we're not sure if we'll find out anything interesting.

Further reading

  1. Survey Research Methods Second Edition. Earl Babbie (1990) Belmont, California: Wadsworth Publishing Company.
  2. Examination of a survey methodology. Dillman's Total Design Method. FE Crosby, MR Ventura, MJ Feldman. Nurs Res 1989: 3856-58.
  3. Mail and Internet Surveys: The Tailored Design Method. Don A. Dillman (2000) Canada: John Wiley & Sons, Inc.
  4. Mail and telephone surveys: the total design method. DA Dillman. New York et al.: Jon Wiley & Sons 1978:
  5. How Surveys Answer A Key Question: Are Consumers Satisfied With Managed Care?. Karen Donelan. Accessed on 2003-10-20. www.managedcaremag.com/archiveMC/9602/MC9602.survey.shtml
  6. Survey Research Methods Second Edition. Floyd J. Jr. Fowler (1993) Newbury Park, CA: Sage Publications, Inc.
  7. Improving Survey Questions: Design and Evaluation. Floyd J. Jr. Fowler (1995) Thousand Oaks, CA: Sage Publications, Inc.
  8. A Brief Guide to Questionnaire Development. Robert Frary. Accessed on 2001-01-04. www.testscoring.vt.edu/fraryquest.html
  9. Survey research. JA Krosnick. Annu Rev Psychol 1999: 50537-67. [Abstract]
  10. Research Resources. Laurier Institute for the Study of Public Opinion and Policy, Laurier Institute for the Study of Public Opinion and Policy. Accessed on 2003-10-20. www.wlu.ca/lispop/lispop.htm
  11. How to Measure Survey Reliability and Validity. Mark S. Litwin (1995) Thousands Oaks, CA: Sage Publications.
  12. How to Conduct your Own Survey. Priscilla Salant, Don A. Dillman (1994) Toronto: John Wiley & Sons, Inc.
  13. Questions and Answers in Attitude Surveys: Experiments on Question Form, Wording, and Context. Howard Schuman, Stanley Presser (1996) Thousand Oaks, CA: Sage Publications.
  14. Health Measurement Scales A Practical Guide to Their Development and Use. David L. Streiner, Geoffrey R. Norman (1989) New York: Oxford University Press, Inc.
  15. Standard Definitions: Final Dispostions of Case Codes and Outcomes Rates for Surveys. Mischael W. Traugott, Murray Edelman, Warren J. Mitofsky, The American Association for Public Opinion Research. Accessed on 2000-www.aapor.org/default.asp?page=survey_methods/standards_and_best_practices/standard_definitions
  16. The Survey Research Handbook Second Edition Guidelines and Strategies for Conducting a Survey. Alreck PL, Settle, Robert B. (1995) Chicago, IL: Irwin Professional Publishing.
  17. PEDAKSI: methodology for collecting data about survey non-respondents [pdf]. Lynn PJ, Institute for Social & Economic Research. Accessed on 2005-04-06. www.iser.essex.ac.uk/pubs/workpaps/pdf/2002-05.pdf
  18. Separating Refusal Bias and Non-Contact Bias: Evidence from UK National Surveys [pdf]. Lynn PJ, Clarke P, Institute for Social & Economic Research, Working Paper 2001-24 (November 2001). Accessed on 2005-04-06. www.iser.essex.ac.uk/pubs/workpaps/pdf/2001-24.pdf
  19. Brochures about Survey Research. ASA Series: What is a Survey? American Statistical Association Survey Research Methods Section. http://www.amstat.org/sections/SRMS/whatsurvey.html
  20. Further reading -- Empirical evidence of response bias
  21. Effect of UK national guidelines on services to treat patients with acute low back pain: follow up questionnaire survey. A. G. Barnett, M. R. Underwood, M. R. Vickers. British Medical Journal 1999: 318(7188); 919-20. [Medline] [Full text] [PDF]
  22. Mortality and cancer rates in nonrespondents to a prospective study of older women: 5-year follow-up. K. M. Bisgard, A. R. Folsom, C. P. Hong, T. A. Sellers. American Journal of Epidemiology 1994: 139(10); 990-1000. [Medline]
  23. Characteristics of non-responders and the impact of non-response on prevalence estimates of dementia. F. Boersma, J. A. Eefsting, W. van den Brink, W. van Tilburg. International Journal of Epidemiology 1997: 26(5); 1055-62. [Medline]
  24. Non-response bias in a lifestyle survey. A. Hill, J. Roberts, P. Ewings, D. Gunnell. J Public Health Med 1997: 19(2); 203-7.
  25. The Tromso Heart Study: responders and non-responders to a health questionnaire, do they differ? B. K. Jacobsen, D. S. Thelle. Scand J Soc Med 1988: 16(2); 101-4.
  26. Do safety practices differ between responders and non-responders to a safety questionnaire? D. Kendrick, R. Hapgood, P. Marsh. Injury Prevention 2001: 7(2); 100-3. [Medline] [Abstract] [Full text] [PDF]
  27. Nonresponse bias in a national study of dentists' infection control practices and attitudes related to HIV. G. M. McCarthy, J. K. MacDonald. Community Dent Oral Epidemiol 1997: 25(4); 319-23.
  28. Comparison of early and late respondents to a postal health survey questionnaire. A. Paganini-Hill, G. Hsu, A. Chao, R. K. Ross. Epidemiology 1993: 4(4); 375-9.
  29. Quality of response in different population groups in mail and telephone surveys. J. Siemiatycki, S. Campbell, L. Richardson, D. Aubert. Am J Epidemiol 1984: 120(2); 302-14.
  30. Representativeness and response rates from the Domestic/International Gastroenterology Surveillance Study (DIGEST). J. G. Tijssen. Scand J Gastroenterol Suppl 1999: 23115-9. [Medline]
  31. What are the characteristics of general practitioners who routinely do not return postal questionnaires: a cross sectional study. N. Stocks, D. Gunnell. J Epidemiol Community Health 2000: 54(12); 940-1. [Medline]
  32. Further reading -- Ambiguous Questions
  33. Would you say you "had sex" if...? S. A. Sanders, J. M. Reinisch. Jama 1999: 281(3); 275-7.
  34. Collection of Race and Ethnicity Data in Clinical Trials. U.S. Food and Drug Administration. Accessed on 2003-02-25. www.fda.gov/cber/gdlns/racethclin.htm
  35. Further reading -- Format of Your Survey
  36. Different response rates in a trial of two envelope styles in mail survey research. D. A. Asch, N. A. Christakis. Epidemiology 1994: 5(3); 364-5. [Medline]
  37. A comparison of responses to mailed questionnaires and telephone interviews in a mixed mode health survey. D. J. Brambilla, S. M. McKinlay. American Journal of Epidemiology 1987: 126(5); 962-71. [Medline]
  38. Increasing response rates to postal questionnaires: systematic review. Phil Edwards, Ian Roberts, Mike Clarke, Carolyn DiGuiseppi, Sarah Pratap, Reinhard Wentz, Irene Kwan. BMJ 2002: 324(7347); 1183-. [Abstract] [Full text] [PDF]
  39. Measuring later health status of high risk infants: randomised comparison of two simple methods of data collection. D. Field, E. S. Draper, M. J. Gompels, C. Green, A. Johnson, D. Shortland, M. Blair, B. Manktelow, C. R. Lamming, C. Law. British Medical Journal 2001: 323(7324); 1276-81.
  40. Increasing response rates for mailed surveys of Medicaid clients and other low-income populations. P. J. Gibson, T. D. Koepsell, P. Diehr, C. Hale. Am J Epidemiol 1999: 149(11); 1057-62.
  41. Do postage-stamps increase response rates to postal surveys? A randomized controlled trial. R. A. Harrison, D. Holt, P. J. Elton. Int J Epidemiol 2002: 31(4); 872-4. [Medline]
  42. A comparison on nonresponse in mail, telephone, and face-to-face surveys. J. J. Hox, D De Leeuw. Quality and Quantity 1994: 28(4); 329-344.
  43. Does length of questionnaire matter? A randomised trial of response rates to a mailed questionnaire. C. Iglesias, D. Torgerson. J Health Serv Res Policy 2000: 5(4); 219-21. [Medline]
  44. Improving the measurement of quality of life in older people: the York SF-12. C.P. Iglesias, Y.F. Birks, D.J. Torgerson. QJM 2001: 94(12); 695-698. [Abstract]
  45. Increasing response rates to postal questionnaires. Cynthia P Iglesias, Yvonne F Birks, David J Torgerson, Paula-J Roberts, Chris Roberts, Bonnie Sibbald. BMJ 2002: 325(7361); 444-. [Full text]
  46. Response rate according to title and length of questionnaire. E. Lund, I. T. Gram. Scand J Soc Med 1998: 26(2); 154-60.
  47. Comparability of telephone and household breast cancer screening surveys with differing response rates. R. M. Mickey, J. K. Worden, P. M. Vacek, J. M. Skelly, M. C. Costanza. Epidemiology 1994: 5(4); 462-5.
  48. Methods for the design and administration of web-based surveys. T. K. Schleyer, J. L. Forrest. J Am Med Inform Assoc 2000: 7(4); 416-25. [Medline] [Abstract] [Full text] [PDF]
  49. Understanding Implementation. The mechanics of polling.. The Statistical Assessment Service. Accessed on 2003-10-20. www.stats.org/record.jsp?type=news&ID=378
  50. Question Time. The Statistical Assessment Service. Accessed on 2003-10-20. www.stats.org/record.jsp?type=news&ID=382
  51. Response to mail surveys: effect of a request to explain refusal to participate. The ARIC Study Investigators. E. Shahar, K. M. Bisgard, A. R. Folsom. Epidemiology 1993: 4(5); 480-2.
  52. A comparison of mail, telephone, and home interview strategies for household health surveys. J. Siemiatycki. Am J Public Health 1979: 69(3); 238-45.
  53. Nonresponse bias and early versus all responders in mail and telephone surveys. J. Siemiatycki, S. Campbell. Am J Epidemiol 1984: 120(2); 291-301.
  54. Improving the response rates to questionnaires. Liam Smeeth, Astrid E Fletcher. BMJ 2002: 324(7347); 1168-1169. [Medline] [Full text] [PDF]
  55. Increasing response rates in telephone surveys: a randomized trial. W. Smith, T. Chey, B. Jalaludin, G. Salkeld, T. Capon. J Public Health Med 1995: 17(1); 33-8.
  56. Using the Visual Analog Scale. Chad Starkey, Pete Koehneke, Daniel Sedory, Paula Turocy. Accessed on 2003-06-23. www.cewl.com/clined/acpm/app_c.html
  57. Is Shorter Always Better? Relative Importance of Questionnaire Length and Cognitive Ease on Response Rates and Data Quality for Two Dietary Questionnaires. Amy F. Subar, Regina G. Ziegler, Frances E. Thompson, Christine Cole Johnson, Joel L. Weissfeld, Douglas Reding, Katherine H. Kavounis, Richard B. Hayes. Am. J. Epidemiol. 2001: 153(4); 404-409.
  58. Comparative Response to a Survey Executed by Post, E-mail, & Web Form. Gi Woong Yun, Craig W. Trumbo. JCMC 2000: 6(1); [Full text]
  59. Refusal and information bias associated with postal questionnaires and face-to-face interviews in very elderly subjects. R. Hebert, G. Bravo, N. Korner-Bitensky, L. Voyer. J Clin Epidemiol 1996: 49(3); 373-81.
  60. Further reading -- Fraud
  61. Interviewer Falsification in Survey Research.. Section on Survey Research Methods, American Statistical Association. Accessed on 2003-05-15. www.aapor.org/interviewfalse.pdf
  62. Further reading -- Interviewer Effects
  63. Do interviewers' Health Beliefs and Habits Modify Responses to Sensitive Questions? A Study using Data Collected from Pregnant Women by Means of Computer-assisted Telephone Interviews. Anne-Marie Nybo Anderson, Jorn Olsen. American Journal of Epidemiology 2002: 155(1); 95-100.
  64. Further reading -- Nonresponse Bias
  65. Response and nonresponse bias in oral health surveys. D Locker. Journal of Public Health Dent 2000: 6072-81. [Medline]
  66. Separating Refusal Bias and Non-Contact Bias: Evidence from UK National Surveys. Peter J. Lynn, Paul Clarke, Institute for Social & Economic Research. Accessed on 2003-10-20. www.irc.essex.ac.uk/pubs/workpaps/wp2001-24.php
  67. PEDAKSI: methodology for collecting data about survey non-respondents. Peter J. Lynn, Institute for Social & Economic Research. Accessed on 2002-February. www.irc.essex.ac.uk/pubs/workpaps/2002-05.php
  68. Further reading -- Response Rates
  69. Response rates to mail surveys published in medical journals. D. A. Asch, M. K. Jedrziewski, N. A. Christakis. Journal Clinical Epidemiology 1997: 50(10); 1129-36. [Medline]
  70. Reported response rates to mailed physician questionnaires. S. M. Cummings, L. A. Savitz, T. R. Konrad. Health Serv Res 2001: 35(6); 1347-55. [Medline]
  71. Further reading -- Reliability and Validity
  72. seamonkey.ed.asu.edu/~alex/teaching/assessment/reliability.html Reliability and Validity by Chong Ho (Alex) Yu. This page discusses the issues surrounding reliability and validity.
  73. trochim.human.cornell.edu/kb/measure.htm Measurement by Bill Trochim. This page discusses various research topics in psychology including the various types of validity.
  74. www.yorku.ca/dept/psych/classics/Cronbach/construct.htm Construct Validity in Psychological Tests, Lee J. Cronbach and Paul E. Meehl (1955). First published in Psychological Bulletin, 52, 281-302. The full text of this classic paper on validity is available on the Internet.
  75. Rating health information on the Internet: navigating to knowledge or to Babel? Jadad, A. R. and A. Gagliardi (1998). Jama 279(8): 611-4.
  76. Developing a scale for measuring the barriers to condom use in Nigeria. Sunmola, Adegbenga M. Bull World Health Organ, 2001, vol.79 no.10. ISSN 0042-9686.

This webpage was written by Steve Simon on 2002-07-12, edited by Steve Simon, and was last modified on 2008-07-14. Send feedback to ssimon at cmh dot edu or click on the email link at the top of the page. Category: Ask Professor Mean, Category: Survey design