Analysis of experimental data

ANU BDSI
workshop Design and Analysis of Experiments

Emi Tanaka

Biological Data Science Institute

20th September 2024

Current learning objective

  • -Comprehend the differences between experimental and observational data
  • -Demonstrate proficiency in designing experiments, including defining research questions, selecting appropriate treatments or factors, and identifying potential sources of variation
  • -Understand the principles of experimental design, including randomization, control, replication, and blocking
  • Understand the fundamental concepts of causal inference for experimental data
  • Formulate a statistical analysis plan for the given experimental design

Analysis of experimental data

  • Analysis of experimental data depends on various context.
  • We cannot cover all types of analysis in a short period!
  • We briefly cover
    • two-sample t-test,
    • linear model and
    • analysis of variance (ANOVA) for some types of experimental data.
  • Two-sample t-test and ANOVA can be in fact just formulated as a linear (mixed) model.

Completely randomised design

  • Let’s suppose we have a completely randomised design as below.

Simulate the experimental data

  • You can simulate in various ways - we are going to use a lazy approach using autofill_rcrds from the edibble R package

Comparing two samples

  • Suppose we have two independent samples like below.
  • Milk yield for the control supplement:
  • And its corresponding average:
  • And sample standard deviation:
  • Milk yield for the new supplement:
  • And its corresponding average:
  • And sample standard deviation:

Two-sample t-test

  • Is the average milk yield using the new supplement (\bar{y}_2) significantly greater than the average milk yield using the control supplement (\bar{y}_1)?
  • Suppose that we have two groups Y_1 \sim N(\mu_1, \sigma_1^2) and Y_2 \sim N(\mu_2, \sigma_2^2) with n_1 and n_2 observations from each group, respectively.
  • H_0: \mu_1 = \mu_2 vs H_1: \mu_1 < \mu_2
  • t = \frac{\bar{y}_2 - \bar{y}_1}{s_p\sqrt{\frac{1}{n_1} + \frac{1}{n_2}}} where s_p = \sqrt{\frac{s_1^2(n_1 - 1)+s_2^2(n_2 - 1)}{n_1 + n_2 - 2}} is the pooled standard deviation and \bar{y}_k and s_k is the sample mean and sample standard deviation, respectively, for the k-th group.
  • p-value is P(t_{n_1+n_2 - 2} \geq t) where t_{n_1+n_2 - 2} is a t-distribution with n_1 + n_2 - 2 degrees of freedom.
  • Low p-value (usually <0.05) indicates evidence that the new supplement provides a signficantly higher milk yield.

Paired observations

Paired t-test

  • A paired t-test is the same as the one-sample t-test for the differences between the paired observations.
  • Suppose that we have n paired observations and Y_d \sim N(\mu_d, \sigma^2_d).
  • H_0: \mu_d = 0 vs H_1: \mu_d > 0
  • t = \frac{\bar{y}_2 - \bar{y}_1}{s_d\sqrt{\frac{2}{n}}} where s_d is the sample standard deviation of the differences between paired observations.
  • p-value is P(t_{2n - 1} \geq t).

Two-sample t-test as linear regression

  • The t-statistic above for the suppnew is the same as the t-statistic from the two-sample t-test below.
  • If there are more than 2 groups, you should not do a pairwise comparison of all groups without adjustment for multiple comparison (not covered today).

Example: horse skin grafting

  • Suppose we have an experiment (with ethic approval) to compare 3 different skin grafting methods on 9 horses.
  • The grafting method is independently applied to each horse.
  • The skin is cut into 6 pieces and thickness is measured for each skin piece.
  • Let’s simulate the data.

Statistical analysis plan

  • A statistical analysis plan (SAP) outlines the statistical methods that will be used for analyzing the data.
  • It is good practice to have SAP written before the data collection (and required for some fields, e.g. clinical trials).
  • But what model should we use for the horse skin grafting data?
  • In principle, all treaments (graft_method) and blocking factors (horse) in the design should be included in the model.
  • Blocking factors are generally assumed to have no interaction effect with the treatment.

Analysis of variance (ANOVA) table

  • ANOVA table show the decomposition of the total variance into different sources of variation.

ANOVA mathematically

  • Suppose that y_{ij} is the measured thickness for the j-th skin piece of the i-th horse for i = 1, ..., 9 and j = 1, ..., 6.
  • Total sum of squares (\text{Total SS}) is \sum_{i=1}^9 \sum_{j=1}^6 (y_{ij} - \bar{y})^2 where \bar{y} is the overall mean.
  • Treatment (grafting method) sum of squares (\text{Treatment SS}) is 18 \times \sum_{k\in \{A, B, C\}} \left(\frac{1}{18}\sum_{i\in\mathcal{T}(k)} \sum_{j=1}^6y_{ij} - \bar{y}\right)^2 where \mathcal{T}(k) is the set of (three) horses that received treatment k.
  • Experimental unit (horse) residual sum of squares (\text{EU Residual SS}) is 6 \times \sum_{i=1}^9 \left(\frac{1}{6}\sum_{j=1}^6 y_{ij} - \bar{y}\right)^2 - \text{Treatment SS}.
  • Residual sum of squares (\text{Residual SS}) is \text{Total SS} - \text{Treatment SS} - \text{EU Residual SS}.
  • The mean square values are obtained by dividing the sum of squares by the corresponding degrees of freedom.
  • The F-value is given by mean square value divided by mean square of residual.

F-test

  • ANOVA can be used to compare means of more than two groups.
  • The most apprororpiate analysis for testing the treatment effect is to compare the corresponding mean square value with the experimental unit residual mean square value.

Linear (mixed) models

  • ANOVA is a special case of linear (mixed) models
  • We can reformulate the F-test using linear (mixed) models framework.
  • The benefit of casting it as a linear (mixed) model is that we can more extensively study the relationship between response and covariates.
  • But we won’t delve into this today.

Model diagnostics

  • When fitting a model, it is important to check the assumptions of the model.
  • Common model diagnostics include:
    • Residual plot (for checking violations of linearity assumptions, homoscedasticity and more)
    • Normal Q-Q plot (for assessing if residuals are normally distributed)
    • Cook’s distance (for identifying outliers)
  • Occasionally the response may need to be transformed (e.g. log, square root) to make the relationship between response and covariates linear.
  • But it’s beyond the scope of today’s session.

Summary

  • Have a statistical analysis plan for your experiment.
  • Simulate the analysis before running the experiment.
  • t-test is a common approach to compare two group means.
  • ANOVA can show the decomposition of the total response variation into different sources and compare multiple groups.
  • Both t-test and ANOVA can be reframed as a linear (mixed) model.
  • Linear (mixed) models can be used to model the relationship between the response and treatment and other peripheral variables.
  • When analysing the actual data, be sure to do some model diagnostics (e.g. check assumptions, outliers, etc.)