Sampling Briefing Note (Appendix 3)

Sampling Briefing Sheet

What is a sample?

A sample is a subset of a population that has attitudes, opinions, habits or characteristics that you want to study.

Sampling is unnecessary if the population to be surveyed is small, for example, if a group is to be surveyed that has only 10 members, then they can all be surveyed with little difficulty. However, for larger populations it is costly to perform a full census and so an unbiased sample of the population should be used to represent the views of the wider population.

An unbiased sample is a subset of the wider population that has the same characteristics as the wider population.

What size of sample?

Determining your sample size has as much to do with how much resources are available to do the work as with the sample size needed to represent the total population. Generally the sample size agreed is a compromise between these two constraints. The following sections give some ideas on appropriate sample sizes.

Generally it is accepted that a sample of 1,500 is the upper limit of any sample size. Beyond this sample size the returns on precision diminishes greatly against the extra cost involved in collecting and processing the information.

The rule of thumb in sampling is having a sample size that allows analysis of the important subgroups of the population. This means for example, that if you are interested in young people’s views on drugs you would have a certain sample size, but if you also want to know how different subsets of these young people think, such as those of school age and older, then the sample size would need to be bigger to accommodate both sub-groups. Many analysts for national surveys assume around 100 cases in each sub-group of a sample.

Sample sizes and non-responses

For a given sample size, it is important to adjust the number actually sampled to allow for the expected non-response rate. This means having a slightly larger sample size than actually required to ensure the appropriate number of people from all categories respond. For example, if the target sample size for a survey is 100 responses, and the estimated non-response rate is 20%, then you would need to send out 125 questionnaires to get 100 back.

The amount of non-response can be quite high.  It isn’t unusual to send out 100 questionnaires in the post and get anything from 35 to 85 replies. The response rate depends upon the level of interest in the subject matter and the incentives used to encourage people to respond (for example, free prize draw entry with every response).

However, incentives can produce their own problems in terms of anonymity of respondents and the types of respondents attracted specifically by the type of incentive.

The issue of non-response is important because the accuracy of the results depends upon an unbiased sample of the population. So if, for example, a survey on people’s views of secondary education is sent out to households, and most of the respondents are females, over the age of 55, who are home owners, the results are not likely to be representative of the wider population. The respondents need to mirror the characteristics of the wider population so that their views can be held to reflect those of the wider population. Any bias in the respondents might heavily influence the survey results and so be misleading.

Most survey software packages allow the survey data to be reweighted to deal with any significant bias in the final response.  For example, if you expect a 50/50 gender balance to your survey but you get a 60% response from females and 40% from males, you can reweight the data.  In this case, the response from males would be scaled by a factor of 1.25 and the response from females would be scaled by 0.83.

A certain amount of guesswork is involved in estimating the amount of expected non-response to a survey, but, after the survey has been completed, checks can be made to assess the characteristics of the respondents against those of the wider target population. If the response is biased to a certain part of the population, then further survey work can be done to rectify the balance.

Type of sampling

There are two main types of sampling: 

• random or unbiased

• purposive

A random or unbiased sample is one in which every member of the wider population has an equal chance of being included in the sample, and so the sample reflects the characteristics of the wider population. You would use this type of sample to understand how the wider customer base is behaving or thinking.

One way of creating a sample is to select a name or address on a systematic basis, so if there are 400 households in an area and the desired sample size is 70 households, then every 6th household on the list would be selected to create the list of sample addresses.

A purposive sample is one that targets a part of the population with specific characteristics of interest, for example, a survey of young adult males between the ages of 18 and 24, who are also unemployed. 

Sampling, confidence levels and intervals

This is where the technical language around the subject of sampling becomes more obscure and drifts into probability theory but it is important to understand in terms of the size of the sample you’ll choose and why. The sampling stage of the survey research process is so vital that the usefulness and quality of the survey’s results depends upon how well it is accomplished.

A sample represents an estimate of the various population characteristics that interest the researcher. Using probability theory, a researcher can estimate their sample’s accuracy and establish a certain level of confidence in their estimate.

The general idea behind sampling is that the higher the degree of accuracy and confidence required – the higher the size of the sample will need to be. Great diversity in the population and a desire to analyse sub-groups in a population also make larger samples necessary. The constraint on bigger samples is cost, so compromises are made in terms of the accuracy and confidence level in the ability of the sample to reflect reality.

What does sampling accuracy and confidence mean?

Sampling accuracy is the measure of what variance can be expected around an estimated statistic for a given sample of the main population. For example, if a sample of a population shows that 65% of the population are “very satisfied” with a service provided by the Council, and the margin of error (the measure of accuracy or precision) between this estimated or sample statistic and the “real” statistic for the whole population is +/- 5%, so the “real” population statistic for “very satisfied” is anything between 60% and 70%.

The question then, is how important is it to know the precise statistic for the wider population? If precision is important then the margin of error needs to be reduced (hence increasing the need for a larger sample – see Table 3 below).

The confidence level is a measure of how representative the sample is of the wider population, that is if the confidence level is 99%, and if 100 different samples were selected from the same population, then 99 times out of 100 the statistics from each sample should fall into the same range.

Together the margin of error (measure of accuracy) and the confidence level constitute sample error. In most cases the aim is to reduce sampling error given the constraints of resources available to do the work.

Table 3 gives some idea of sample sizes for different populations, margins of error and confidence levels.

Table 3: Sample sizes for 95% confidence interval at 3% or 5% margin of error

Back to Contents Page