Using inferential statistics Created by jdeboel on 10/26/2010 8:48:00 AM Learn how to correctly use inferential statistics and find out the KPI(key performance indicators) of a research, take into account psychographics from Qualitative research, likert scale items, independent and dependent variable. Data modeling: Use Concepts and Facets as your statistical indicators
The drobots approach is based on a generic measurement model in which question lists are composed in a structured fashion. The method uses a “topdown” approach in which an analysis model is set up where the core research topics are transformed into indicators. The model is inspired by Latent Variable analysis models used in psychographics.
The drobots measurement model allows analysis on the manifest survey question level and also on a more abstract (latent) level by defining facets and concepts. The use of this approach helps to simplify benchmarking: instead of comparing items or questions with each other; a more conceptual analysis is possible. It is also easier to compare different surveys performed at different points in time. (ie. Follow-up questionnaires) Other than that, aggregating information to a more conceptual level allows for certain parts (ie. Price/quality of the services) to be compared to external data.
The grouping of items should of course still be statistically sound. One cannot just combine little groups of questions to achieve a higher measurement level. Statistical validation rules should still be respected.
A questionnaire that is composed through Key Performance Indicator (KPI) or conceptual methodology is comprised of 3 analysis levels: concepts, facets and items (the questions themselves). There isn’t always a need for 3 full levels however; sometimes the facet level isn’t used.
The concepts
Concepts are the highest level in a questionnaire. They are often the main axes discovered in a more qualitative research phase. A questionnaire can be comprised of one or more concepts. While composing the questionnaire, one could also choose to divide a concept into facets. (see below)
An example:
Theme= Course Evaluation --> examples of concepts are: Staff, Material, Tests,… |
These concepts are sometimes called Key Performance Indicators(KPI’s) in an operational benchmark business context.
The Facets
Concepts can be subdivided into facets or factors that are a further refinement of the concepts.
An example:
Concept = Material --> Facets= Course(material), Structure, Interesting,...
|
The items or questions
The last level is comprised of the questions themselves, this is called the item level. In order to visualize one particular facet, one or more questions can be asked.
An example:
Facet = Interesting
Questions = “Due to this course my interest in the subject has increased"
Or: “The course offered added value to me." |
The illustration below is a visual representation of the various information levels that can be achieved using the techniques of data modeling:
When combining items that belong together into facets and into concepts, we get 2 main benefits:
The reliability of the information increases substantially.
The number of anchor points increases, which makes the measuring level rise from ordinal to interval (metric). Simulation studies in the late ’70s have shown that, starting at 9 anchor points, an ordinal variable gets metric properties; by which more and more powerful 'parametric' analyses become possible. So, in combining 2 items with 6 anchor points (Completely disagree to completely agree), we yield 12 anchor points for the new facet.
Since an analysis on facet level or concept level requires less tests compared to an in depth analysis of all items, it’s easier to see the bigger picture and to perform a more efficient analysis and reporting. Because of this, the accumulation of type I & II errors, which is traditionally generated in inferential statistics, also diminishes.
Questionnaire construction
Questionnaire construction is a 3 step process that will enhance the level of inquiry and deliver reliable information. Of course, if you have your own approach to make questionnaires, drobots will happily analyze your data as well.
Step 1
Define the concepts and facets (if you choose to use facets) in your project. Then create the questions that will be good indicators of the facets or concepts. Use simple phrases that express a clear opinion or attitude. Use so called Likert scale items: items that express the level of agreement with statements. We advise to use between 4 and 6 so called anchor points. Too few anchor points implies poor measurement and too many will disorient your respondent. At this stage try to avoid binary data such as YES/NO answers as indicators of facets or concepts. And please do avoid so called creative mixtures of different question types and/or number of anchor points. It creates fatigue for the respondent and it turns a potentially simple analysis plan into a statistical nightmare.
Step 2
Think about the independent or split variables you want to use in your project. Sociodemographic variables such as age group or gender are typical examples. Do not ask information in your project that is irrelevant for the analysis.
Step 3
Pre test your questionnaire. Sometimes you think you have the ultimate questionnaire but it is not you who will be invited to share opinions, attitudes and needs. Ask 5 people in the target audience to fill out the questionnaire and make comments.
Model your data
Whether you are uploading your data yourself with the drobots interface or you send us the data, you need a structured approach that lists the following:
• Variable labels
• Level of measurement
• Value codes and labels
• Facet&Concept indicator information
The variable label is the label that will be used in the graphics and tables. As a result of this, we advise you to make them as short as possible, as that will in turn create clearer graphics. The level of measurement is nominal or ordinal or metric. The value codes are the valid data codes for the variables. The value labels are the meaning of the values (ie. 1:male; 2:female) If the question is used as a facet indicator, define to which facet.
If one concept or facet is the sole dependent variable, mention that as well in the appropriate stage. If you do not define a single dependent concept or facet, all the concepts and facets can be used as dependent variables in the impact analysis.
Other than that key information, you need to specify the name of the facets, the concepts and the way questions are linked to facets and facets to concepts. Make sure that negatively phrased questions as well as their scores are inverted when entering them in the platform.
An example:
"I don't go shopping alone." - Totally agree to Totally disagree
This is actually a negative question. You might give this variable the short description "Shopping alone". The problem will arise when the data load takes place. Because in the datafile, "Totally agree" will have for example score "6"; and "Totally disagree" will have score "1". The analyser treats these scores as mathematical values. So when analysing, the results will for example depict the "Shopping alone" variable as high, while the actual results should be the exact opposite.
Only changing the short description to "Not shopping alone" will make it a little clearer, but still isn't the best solution; because when we cross variables to compare them to each other, it will be hard to see true tendencies in the results.
|
Uploading data
With drobots, you either do everything yourself using the appropriate modules for project definition, model build and data load, or if you are still unsure about taking a modeling approach yourself, you can send us your questionnaire information and data; and we will try to help you as best we can. Just send an email to
support@drobots.com and explain what your project is about. If it's your first introduction to drobots, this service is of course completely free.
