Key Concepts and Best Practices in Statistical Studies

Posted on Feb 17, 2025 in Naval Architecture

Key Elements of a Statistical Study

Representative Sample: The sample should extend beyond the participants and be representative of a larger group. The sample is the group actually studied, chosen from the larger population.
Sufficient Sample Size: Small samples can lead to erroneous conclusions. The required size depends on the natural variability of the data. More diverse or variable data requires a larger sample.
Study Type: Observational or Randomized Experiment: Simply observing isn’t always sufficient. To establish a causal connection, randomly assigned experiments are needed. Researchers cannot influence participants’ assignment to groups. Cause-and-effect conclusions cannot be based on observations alone, as groups may differ in ways that influence the outcome. Randomized experiments allow for cause-and-effect conclusions.

Essential Definitions

Unit: A single individual or object to be measured.
Sample: The collection of units actually measured or the collection of measurements obtained.
Population: The larger group from which the sample was chosen; the entire collection of units about which we have interest.
Sampling Frame: A list of units from which the sample is chosen. Ideally, it includes the whole population.
Sample Survey: Measurements are taken on a subset (sample) of units from the population. It’s a subgroup questioned on a set of topics. Results represent the population if the sample is chosen correctly. No manipulation or intervention is involved. It is faster to collect than a census if the population is large.
Census: A survey in which the entire population is measured.
Observational Study: Researchers merely observe things about the sample.
Randomized Experiment: Participants are randomly assigned to one of various treatment groups.
Data: A collection of numbers or information to which meaning has been attached. The meaning and conclusions depend on how well the information was acquired and summarized. Be cautious of media reports, as they rarely show the actual data.

Randomized Experiments Explained

Randomized experiments create differences in an explanatory variable and examine the results on a response variable. They measure the effect of manipulating the environment. This manipulation is assigned to participants randomly.

Randomization: Helps make groups approximately equal in all respects except for the explanatory variable, evening out confounding variables across treatments. This allows for the determination of cause and effect.
Explanatory Variable: The feature being manipulated. It may explain or cause differences in the response variable.
Response/Outcome Variable: The outcome of interest or the result of the manipulation.
Treatment: One or more combinations of categories of the explanatory variable(s) assigned by the researcher.

Observational Studies Explained

Observational studies observe differences in the explanatory variable and note whether these are related to differences in the response variable. Manipulation occurs naturally, not imposed. However, one cannot assume the explanatory variable is solely responsible for observed differences in the response variable. It’s difficult to establish a causal connection due to the lack of a control group. It can be unethical to assign people to receive a specific treatment, and certain explanatory variables are inherent traits that cannot be randomly assigned.

Case Study: An in-depth examination of one or a small number of individuals. It is descriptive, without statistical methods, and cannot be extended to a larger population.

Margin of Error

The margin of error is the amount by which the proportion obtained from the sample is likely to differ from the true population proportion. It is calculated as 1 divided by the square root of the number of people in the sample. Sample surveys can be quite accurate, with a margin of error typically less than 5%.

Critical Components of Statistical Studies

It’s important to know the details of a study to understand its applicability (e.g., age groups). It’s also crucial to know the magnitude of differences to assess their relevance to specific individuals.

Source of Research and Funding: Studies may be funded by government, private companies, universities, or institutions for various purposes.
Researchers in Contact with Participants: Consider who interacted with participants, the message conveyed, and potential biases.
Individuals/Objects Studied and Selection Method: Be aware of potential biases, such as voluntary responses.
Nature of Measurements/Questions: Consider the numbering, wording, and potential for suggestions in questions.
Setting of Measurements: Where and when the data was collected.
Differences in Groups Being Compared: Identify other ways the groups may differ that might influence the comparison.
Extent/Size of Claimed Effects: Assess whether the reported effects are practically significant.

Potential Pitfalls in Studies

Deliberate Bias: Questions deliberately worded to support a cause.
Unintentional Bias: Questions worded such that meanings are misinterpreted.
Desire to Please: Respondents may underreport undesirable social habits.
Asking the Uninformed: People may provide fictitious information.
Unnecessary Complexity: Questions may be overly complex.
Ordering of Questions: The order of questions can influence responses.
Confidentiality vs. Anonymity: Anonymity is generally preferred.

Open vs. Closed Questions

Open Questions: Allow respondents to answer freely. They are difficult to summarize, but offer more nuanced responses.
Closed Questions: Provide limited answer choices. They are easy to administer and analyze, but may not capture the full range of responses. Randomizing the order of choices can mitigate the “recency effect” in phone interviews.

Measuring Concepts and Attitudes

It can be difficult to measure abstract concepts like self-esteem or happiness. A common method involves respondents reading statements and indicating their level of agreement.

Understanding Variability

Variability: Refers to differences in measurements. It reflects an inconsistent pattern.
Variables: Characteristics that take on different values. They can lead to uncertainty.
Natural Variability: Natural differences across individuals.
Measurement Error: The difference between a measurement and the true value. It can be due to an unreliable measuring device.

Types of Variables

Categorical Variables: Those that can be placed into categories.
- Ordinal Variables: Have a natural ordering.
- Nominal Variables: Do not have a natural ordering.
Measurement Variables (Quantitative Variables): Those that can be assigned numerical values.
- Continuous Variables: Can take on any value within a given range (“amount of”).
- Discrete Variables: Can be counted (“number of”).

Confounding and Interacting Variables

Confounding Variables: Related to the explanatory variable and affect the response variable. Their effect cannot be separated from the explanatory variable’s effect. This is a major problem in observational studies. The best solution is to measure the variable to see if it is also related to the response variable.
Interaction: Occurs when the effect of one explanatory variable on the response variable depends on the value of another explanatory variable. If two variables interact, results should be presented separately for each combination.

Measurement Concepts

Valid Measurement: Actually measures what it claims to measure.
Reliable Measurement: Gives approximately the same result repeatedly when taken on the same object or individual.
Biased Measurement: Consistently off the mark in the same direction.

Sampling Methods

Simple Random Sampling: The best way to get a representative sample. Everyone in the population has a specified chance of being selected. Requires a list of units in the population and a source of random numbers.
Stratified Random Sampling: Divides the population into groups (strata) and takes a simple random sample from each stratum. Provides individual estimates for each group and can be more accurate than simple random sampling.
Cluster Sampling: Divides the population into groups (clusters), takes a random sample of clusters, and measures only the selected clusters. Requires a list of clusters, not all individual units.
Systematic Sampling: Divides the population list into consecutive segments, randomly chooses a starting point in the first segment, and samples at that same point in each segment. Can lead to a biased sample.
Random Digit Dialing: Approximates a simple random sample of households with telephones.
Convenience Sampling: *Non-random/Non-probability*. Units are selected because they are easy to sample. Not representative of the entire population.
Self-Selection Sampling: *Non-random/Non-probability*. Units volunteer. May not represent the entire population.
Multistage Sampling: Combines different sampling methods in various stages.

Difficulties in Sampling

Using the Wrong Sampling Frame: May include unwanted units or exclude desired units.
Not Reaching Selected Individuals: Even with a proper sample, units may not be reachable.
Low Response Rate: Response rates should be reported. Low response rates can limit generalizability.

Disasters in Sampling

Volunteer or Self-Selected Sample: Relying on volunteers is problematic.
Convenience or Haphazard Sample: Choosing the most convenient group can produce misleading results.

Control Groups and Placebos

Control Groups: Handled identically to treatment groups, except they do not receive the active treatment.
Placebos: Inert substances that resemble the real treatment but have no active ingredients.

Difficulties and Disasters in Experiments

Confounding variables
Interacting variables
Placebo effects
Generalizability

Difficulties and Disasters in Observational Studies

Confounding variables and causation
Extending results appropriately
Using the past as data

Useful Information of a Data Set

Centre/Mean
Outliers: Unusual values far removed from the rest of the data.
Standard Deviation/Variability: How spread out the values are.
Shape: Normal, bimodal, bell-shaped, etc.

Percentiles

The Pth percentile is a value where P% of the values are smaller. For example, the 10th percentile has 10% of values smaller than it. The 25th percentile is Quartile 1.

Calculating Standard Deviation

Find the mean.
Find the deviation of each value from the mean (value – mean).
Square the deviations.
Sum the squared deviations.
Divide the sum by (number of values – 1). This is the variance.
Take the square root of the variance. This is the standard deviation.

Effect of Outliers

Extreme values (outliers) can easily influence the mean and standard deviation. The median and quartiles are more robust to outliers.

Visualizing Data

Stem Plot: Quick and easy way to order data and visualize its shape.
Histogram: Better for larger data sets and visualizing shape.
Frequency Curve: Shows possible values for a measurement and can be used to find the proportion of the population falling into various ranges. It’s a smoothed-out histogram.

Normal Distribution

Evenly spread, mean = median = mode, half the population lies below the mean, and values further from the mean are less likely to occur.

Standardized Score (Z-score): The number of standard deviations a measurement falls above (positive) or below (negative) the mean.
Z-score = (Observed Value – Mean) / Standard Deviation
Observed Value = Mean + (Standardized Score)(Standard Deviation)

Bell-Shaped Measurements and the Empirical Rule

If the mean and standard deviation are known, you can find the percentile for any measurement without additional information.

Empirical Rule:

68% of values fall within 1 standard deviation of the mean.
95% of values fall within 2 standard deviations of the mean.
99.7% of values fall within 3 standard deviations of the mean.