Why is qualitative research not numerical

Evaluation methods

Once the evaluation questions have been determined, it is a matter of defining the research methods in order to arrive at the desired answers.

In the social sciences, quantitative and qualitative research approaches have become established on the one hand, which differ in terms of interest in knowledge and methodology. As the terms suggest, a quantitatively oriented research design is designed to capture phenomena in their quantity, i.e. in terms of quantity; a qualitatively oriented design asks for differentiated opinions, perspectives, and backgrounds. Although this is often suggested, the individual survey methods and the corresponding survey instruments cannot be clearly assigned to these two categories. Most survey instruments can be used both quantitatively and qualitatively, depending on their degree of openness. In addition to quantitative, closed questions, a questionnaire can also contain open questions that can be qualitatively evaluated. In interviews, on the other hand, it is also possible to integrate quantitative elements, e.g. when interviewees are asked about assessments on a scale of 1-10, which can then be evaluated numerically.

Quantitative method

Countable properties are measured or facts are numerically mapped in quantitative methods. Results are determined by means of statistical correlations and calculations. Standardized questionnaires or observations are based on already known prior knowledge and are aimed at very specific aspects that they examine, check, and / or examine broadly to which you would like to obtain assessments.

Quantitative methods focus on given questions and are therefore limited in their perspective. New aspects that do not appear in the formulated answer categories are therefore often not taken into account.

Qualitative process

A qualitative approach is particularly productive for in-depth investigation of phenomena that are complex or for which there is little prior knowledge. Open interviews or observations can capture essential aspects that remain hidden with a strongly standardized procedure and especially with closed questions. Separate quality criteria are proclaimed for qualitative methods in contrast to the classic quality criteria of quantitative research, e.g. intersubjective comprehensibility, appropriateness of the subject, scientific foundation, reflexivity or contextuality (Steinke 1999, Studer 2011).

Qualitative methods are often used when the subject of research is relatively new or to explore the research area or question and to develop hypotheses. Quantitative methods are used more to make statements about the frequency or distribution of certain phenomena or to test existing hypotheses. Often qualitative and quantitative methods are combined.

If a certain object is illuminated from different methodological perspectives at the same time, examined with several e.g. qualitative and quantitative methods, one speaks of method triangulation (Flick 2004). The selection of the methods used depends on the purpose of the investigation, the object of investigation and the questions. In addition, the people to be interviewed, the resources available and the existing methodological know-how also play an important role.

Evaluation methods / instruments

There are a number of methods and associated tools that can be used in evaluations. Which is the best method in each case depends on the question, but also on the environment, the target group, financial and human resources and other aspects. So there are no recipes as to which methods and instruments are the right ones for which questions. Nevertheless, there are orientation aids that can be useful when choosing. A few such orientations are given in the following overview:

Evaluation methods
Instruments

Remarks

Document analysis

Existing documents (reports, concepts, images, videos, ...) can be used for individual questions or aspects so that no additional data collection is necessary.

Written surveys (paper version)
questionnaire

The written survey is the most common survey method when information is to be collected from medium-sized or larger population groups in order to obtain a statistically representative overview of the responses from these groups. Questionnaires can contain both closed questions (with predefined answer categories) and open questions.

Online surveys
Online questionnaire

Compared to other written surveys, online surveys have the advantage that subsequent electronic data collection is not required, which saves a lot of time. However, not all people or groups of people can be reached this way.

Telephone interviews
questionnaire

However, surveys are also being carried out by telephone more and more frequently. Telephone surveys are very time-consuming compared to written surveys, especially for larger population groups, but are an efficient alternative for smaller groups. Telephone surveys have the advantage over written surveys that questions can be asked in the event of uncertainties or for more in-depth information.

Telephone interviews
Interview guide

In addition to quantitative surveys, qualitative interviews can also be carried out by telephone in order to elaborate on certain questions. Telephone interviews save travel time and costs compared to face-to-face interviews.

Personal interviews
Interview guide

Personal interviews usually create greater familiarity than telephone interviews and they allow additional insights into the context of the respondents, insofar as the interviews are carried out in the field.

Group surveys
Interview guide

Group surveys have the advantage over individual interviews that new aspects emerge and can be worked on together. Group interviews are therefore particularly suitable when consensus is to be worked out in assessments. However, the perspectives of the individuals cannot be explored to the same extent and independently of the other participants.

Observations
Observation grid

Observations are used in order to be able to assess what is happening on site at first hand, i.e. without this having already been filtered by the respondents. The subliminal, the unconscious and the non-explicit can also be captured in this way.

However, only aspects can be assessed that are directly accessible to observation, not, for example, the attitudes of people.

Measurements
Measuring instruments

Certain aspects can be measured objectively with appropriate instruments, such as height and weight to calculate the body mass index of people and track it over time, or time to determine how long it takes to perform certain work. Measuring instruments in these cases would be scales, measuring tape and watch.

Testing
Test arrangements and corresponding aids

Although less frequently used, tests can be used in a variety of ways in evaluations to assess intervention approaches or products (e.g. pretests for poster campaigns, test purchases of alcohol or tobacco products by minors, usability tests on websites, ...).

In addition to these widespread survey methods, there are a number of other methods that can also be used for evaluations, but whose potential is far from being exhausted. Interesting methods are e.g. story-telling, structure constellations (teams, organizations, settings), games, experiments, car photography and video work, repertory grid, etc.

Bibliography

  • Mayring, Ph. (2002). Introduction to qualitative research. Weinheim: Beltz.
  • Flick, U. (2006). Qualitative evaluation research. Concepts, methods, implementations. Reinbek: Rowohlt.
  • Müller-Benedict, V. (2001). Basic course in statistics in the social sciences. 5th edition. Wiesbaden: VS-Verlag.
  • Flick, U. (2004). Triangulation. An introduction. Wiesbaden: VS Verlag.
  • Studer, H. (2011). Reconstructive Research Practice - Approaches to an Object-Based Methodology. Dissertation 2007. Saarbrücken: Südwestdeutscher Verlag für Hochschulschriften.
  • Porst, Rolf (2000). Question Wording - For formulating questionnaire questions. ZUMA How-to series, No. 2.
  • Steinke, I. (1999). Qualitative Research Criteria. Approaches to the evaluation of qualitative-empirical social research. Weinheim: Juventa

What could prevent you from considering these aspects

  • They lack the professional competence to choose suitable evaluation methods and instruments
  • You have not allocated enough resources to search for or develop and translate survey instruments

What you can win

A well-founded choice of evaluation methodology and instruments ensures that the data collected actually provides the desired information or is suitable for answering your evaluation questions. At the same time, it can be prevented that too much data is collected that cannot (or cannot) be evaluated later.

What you can actually do

After you have defined the evaluation questions, check what data is already available that can be used to answer the questions. Look for methods and procedures that promise the greatest possible gain in knowledge with as little effort as possible. Let experienced evaluation experts advise you on the determination of evaluation methods and the development of evaluation instruments.

Reflection questions

  • Does a more quantitative or a more qualitative research approach appear suitable to enable an assessment of the project in the desired sense?
  • Are the chosen methods appropriate to assess the achievement of the project objectives and to answer the evaluation questions?