Writing and Analyzing surveys
From Eduwiki
Tests and Surveys are an essentially part of education research. Tests are designed to measure students' skills or knowledge in some area and surveys assess their ideas, beliefs and attitudes on different topics. Both are typically questions answered via paper-and-pencil or online forms that are completed individually. Tests done in pairs or in groups can be very effective instructional tools or assessments of a group - but it is difficult to determine an individual's contribution to the result.
Contents |
General tips
Generally you want to ask a lot of questions and collect a lot of data but are limited by the patience of the subjects. Very long surveys may turn off the subject and they may give up or may not read the questions carefully (or at all). Questions may be well written but if the subject doesn't take the seriously the data will not be useful. There is a constant trade-off between getting the information that you want without over-asking.
useful links:
- 20 Tips to Writing Effective Surveys
- 20 more tips
- Designing Surveys (Colorado State)
- Educational Research surveys (UTexas)
- Buy Essay
Question Types
The main question types are:
- Essay questions
- Short answer written questions
- multiple choice
- Likert scale - rating something from 1-5
- problem solving (tests only)
Some less common (but sometimes valuable) question types include:
- Fill-in questions (enter the missing word)
- Drawing or diagramming
- Matching (terms and definitions or pictures etc)
- Check all that apply
- Ranking/ordering items in a list
Usage Tips
Assessing Understanding
Multiple choice questions are so common in tests that students are becoming better at guessing answers when they don't know. To really assess students understanding, several related questions are needed to get more confidence that students really do know something and not just guessing correctly.
Essay questions can reveal a great deal about understanding, but they can be confounded by poor writing skills. Many researchers prefer to use short answer questions (2-4 sentence responses) because they are focused writing that reveals students' thinking without forcing them to structure an essay. It can be a very effective technique to combine a multiple choice question with a short answer. In the example below, the multiple choice helps focus students thinking (hopefully on the correct answer) and then allows them to write an explanation:
1. Which of the following is primarily responsible for earthquakes? (circle one)
a. volcanoes b. motion of the plates c. meteors d. moon
Please explain how your choice creates earthquake:
Drawing and diagramming questions can be effective assessments of understanding and can be easy for teachers to evaluate. However, because these activities are less common, students need practice in doing this before putting a lot of weight on their responses. There are not currently good tools to ask these questions online.
Assessing Attitudes
Likert scale questions are typically used to assess the strength of someone's attitude. For example "how likely are you to try the following...?" or "rate how much you like the following topics". The question needs to provide a scale to describe the choices. For example: "1 - not at all likely, to 5 - very likely" or "1 - do not like, 5 - like very much". The advantage of the scale questions is that you can correlate the strength of one attitude with another attitude. The drawback of the scale question is that you do not know the reason why people feel the way they do. You can follow up by asking for an explanation for the rating ("please explain your rating above") - but this can be a burden if asked too much.
Short answer and essay questions are valuable tools for assessing attitudes because they can help explain the motivations and reasons behind the attitudes. Quotations explaining the thought process can be an important part of your paper - and help illustrate the meaning of statistical data.
Analysis
Multiple choice questions are often used because their analysis is often done easily - simply report the percentages answering different choices. These questions can also be use in statistical analysis to divide students into different categories.
Essay questions and short answer questions are generally analyzed by creating one or more rating scales and then scoring (coding) the essays on the scales. Rating scales are slightly different from grading rubrics because they attempt to separate different types of thinking - even if they are equally correct. In other words it does not score the essay but it does attempt to categorize the answers in useful ways. For example for the short answer questions about earthquakes (above) you might have a scale like this:
Source of earthquakes (Q.1) A. mentions plate movement builds pressure/tension that gets released in earthquake B. plate movement leads to earthquakes (vague) C. meteors shake earth D. volcanoes shake earth or create pressure (feasible argument) E. Other thoughtful (wrong) response F. Incomprehensible G. No response
To create a rating scale for a question, you should take a sample of the answers and read through them trying to get a sense of the range of responses relative to the information you want to get from this question. Then draft a scale that separates the important groups of responses. Once a draft is created, then you begin an iterative process of coding and revising the code. Generally this can be done as part of coder training.
When possible, two raters should score the data - including both scoring a sizable number (> 30) of responses. Both coders should participate in training where they code the same responses and then compare their codes. They need to discuss cases where they were unsure about a code and where they disagreed. This discussion can result in refinements in the rating scale and ultimately with a sufficiently high inter-coder reliability. Generally 90% reliability is a good figure to aim for depending on the specifics of the
Although process of analyzing essay questions sounds difficult, it goes fairly quickly and the results are more reliable information on students' thinking.
