Pilot Studies and the Aims of Piloting

Pilot Study Psychology

Let's begin by defining what a pilot study is:

Typically, researchers conduct pilot studies before a large-scale study, which allows them to refine the study design, thus increasing the validity and reliability of the study. They can test parts of the study, such as procedures and materials, to ensure they are effective.

In psychological research, new measures must be tested in a pilot study before they can be used in a full-scale experiment. This is to ensure that the measure is reliable and valid.

Pilot studies help save time and money by detecting any errors in design. Researchers do not need to test proven measurement methods because their reliability and validity have been tested.

These are some of the purposes that pilot studies can serve:

• It can determine the validity of newly constructed measurement tools such as questionnaires.
• Identify areas of the research needing improvements, such as changes in equipment, procedure, participant instructions, or data handling and analysis procedures.
• Estimate sample size or identify potential target populations or populations to be excluded.
• The results can serve as evidence to show funding bodies whether they should grant funding.

Pilot Study in Research: Methodology

The steps researchers take when designing a pilot study do not differ significantly from the steps of regular research.

Pilot studies can successfully detect flaws in the study's design, which may impact its reliability and validity.

In terms of research methodology, there are many ways that pilot studies are used not only to improve the research but also to determine how an investigation will be carried out.

For instance, it can determine the sample size and how participants will be selected. The researcher may identify potential participant variables such as poor eyesight affecting the study's results. So in their later investigation, they may have an exclusion criterion for people with poor eyesight.

Researchers should pilot all measurement instruments, such as interviews and questionnaires, that will be used in an investigation. These should be tested under conditions similar to those later used, e.g., using standardised protocols, under specific research conditions, or in a particular setting.

In pilot studies, researchers should replicate the exact procedure used later for data entry and analysis. The researcher should keep data privacy and confidentiality in mind regardless of whether it is a pilot study or not.

Generally, pilot studies do not test experimental hypotheses. Therefore, these studies rarely test inferential statistics.

Fig. 1. One of the uses of pilot studies is to test the measuring instruments.

Pilot Study in Quantitative Research

Occasionally, researchers need to create new metrics when researching because no established measures are appropriate for testing their hypothesis. Or, the previous metrics do not measure variables in which the researcher is not interested.

Researchers must ensure their reliability and validity when developing new measurement tools, such as questionnaires. They conduct pilot studies to verify this.

A pilot study should measure the reliability and validity of new measures and could do so using the following procedures:

• High construct validity ensures that the measurement captures the intended variables. An example would be recruiting two groups of participants. One should be more knowledgeable about the phenomenon, and the other should be non-experts. If the results of the two groups are similar, construct validity is high.
• High internal consistency reliability occurs when researchers measure all aspects of a variable (e.g., a questionnaire assessing bullying experiences must measure all aspects of bullying: physical, verbal, social, and cyberbullying). A statistical test measuring Cronbach's alpha determines these aspects (a measure of how closely related a set of items is). A value of 0.7 is generally considered acceptable.
• High test-retest reliability is measured by asking the same participants to complete the measure under the same conditions but on different days. Researchers perform a correlational analysis with the test results. A significant positive correlation refers to a reliable measurement, which shows the results are consistent.

Pilot Study Example

A well-known example of a pilot study is the one that the World Health Organisation conducted in 1972. Given that the presence of patients showing schizophrenic profiles was increasing, researchers saw the need to develop procedures that could assist in diagnosing the illness.

The purpose of the pilot study was to identify if it was possible to create standardised measures and procedures for the diagnosis of schizophrenia that is both reliable and valid cross-culturally.

The pilot study suggested that cultural alignment between countries was possible, given that similar schizophrenia profiles were present in different countries. These, in turn, inspired future research to the development of classification manuals and diagnosis tools.

Advantages and Disadvantages of Pilot Studies in Psychology

Researchers typically conduct pilot studies before a large-scale study. Pilot studies have many advantages, but there are also limitations.

The advantages of pilot studies are:

• It avoids wasting time and resources.
• Increases the validity of later research projects if conducted properly.
• It allows the researcher to determine the most appropriate design and method of data collection to test the hypothesis.
• The researcher can identify potential ethical issues or obstacles affecting later research.
• Researchers can determine if new scales are reliable or valid.

And the disadvantages of pilot studies are:

• They are time-consuming and can be costly.
• Researchers should not use pilot studies to guess the expected results.
  • As pilot studies are small, the results will likely differ from large-scale studies, meaning that pilot studies are sensitive research techniques.
• Funding may be denied because of the results or researchers' difficulty recruiting participants.
• Different participants are needed for subsequent research due to practice effects.