---
title: "Searching for rate shifts"
author: "Silvia Castiglione, Carmela Serio, Pasquale Raia"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{search.shift}
  %\VignetteEngine{knitr::rmarkdown_notangle}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
if (!requireNamespace("rmarkdown", quietly = TRUE) ||
     !rmarkdown::pandoc_available()) {
   warning(call. = FALSE, "Pandoc not found, the vignettes is not built")
   knitr::knit_exit()
}

misspacks<-sapply(c("plotrix","scales","RColorBrewer","kableExtra"),requireNamespace,quietly=TRUE)
if(any(!misspacks)){
  warning(call. = FALSE,paste(names(misspacks)[which(!misspacks)],collapse=", "), "not found, the vignettes is not built")
   knitr::knit_exit()
}

knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)

require(RRphylo)
require(kableExtra)
options(rmarkdown.html_vignette.check_title = FALSE)
options(knitr.kable.NA = '')
source("functions4vignettes.R")
```

## Index

1.  [search.shift basics](#basics)
2.  [Testing rate shifts pertaining to entire clades](#clade)
3.  [Testing rate shifts pertaining to phylogenetically unrelated species](#sparse)
4.  [Guided examples](#examples)

## search.shift basics {#basics}

One of the main advantages of the [RRphylo](RRphylo.html) method is the computation of phenotypic evolutionary rates for each branch of the phylogeny. This is particularly welcome as it allows to test for significant rate shifts occurring in groups of species belonging to a certain rate regime. For a rate shift to be real, the $β$ coefficients attached to the branches evolving under a distinctive rate regime must be statistically larger/smaller than the average coefficients calculated for the other branches of the tree. Since rate values are branch-specific, it is feasible to search for shifts indifferently when different rate regimes pertain to distinct [clades](#clade), or to a number of [unrelated species across the phylogeny](#sparse). The strategy to assess rate shifts is to test the difference in mean rates between the branches hypothesized to evolve under different rate regimes and the rest of the tree through randomizations.

The function `search.shift` (Castiglione et al. 2018) takes an object produced by `RRphylo`. It can be used to automatically locate the shifts, test different clades where distinct rate shifts are presumed to occur, or test for rate differences among different tip categories. It also allows to account for the effect of a covariate on the rate of evolution, provided the covariate has not been used to produce the RRphylo object (see [`RRphylo`](RRphylo.html) for further explanation about the covariate, and the [guided examples below](#examples)).

## Clade condition {#clade}

Under the `clade` condition, the function searches for shifts in absolute evolutionary rates applying to entire clades, using rates computed at both nodes and tips. The argument `node` allows the user to specify the nodes subtending to presumably shifting clades. If no specific hypothesis is available, `search.shift` automatically scans the phylogeny to locate significant instances of rate shift.

Under the automatic mode, the function selects all the clades ranging from one tenth to one half of the tree size (yet, the smallest clade size can be specified by setting the argument `f`), and computes the difference between the mean absolute rates of each of them and the rest of the tree. The significance of such difference is assessed by comparing it to a random distribution of differences, obtained by shuffling rate values across tree branches (the object `$all.clades` within the function output). A rate shift is identified if the rate difference is significantly higher (p \> 0.975) or lower (p \< 0.025) than expected by chance.

```{r,echo=13,warning=FALSE,message=FALSE,fig.dim=c(6,4),out.width="99%",dpi=220}
require(ape)
require(phytools)
require(scales)
cc<- 2/parallel::detectCores()

set.seed(22)
rtree(100)->Tree
fastBM(Tree)->y

y[tips(Tree,161)]*5->y[tips(Tree,161)]

RRphylo(Tree,y,clus = cc)->RR
search.shift(RR,status.type = "clade")->SSauto

layout(matrix(c(1,2),ncol=2),width=c(0.64,0.36))
plot(Tree,no.margin = TRUE, show.tip.label = FALSE)
nodelabels(bg="white",frame="none",node=as.numeric(rownames(SSauto$all.clades)),col="red",cex=0.8)

fontm<-cbind(rep(2,nrow(SSauto$all.clades)),c(rep(1,nrow(SSauto$all.clades))),c(rep(1,nrow(SSauto$all.clades))))
plotTab(tab=cbind(rownames(SSauto$all.clades),round(SSauto$all.clades,3)),
        text.cex=0.7,text.font=fontm,text.box=c(box="n"),
        text.grid=data.frame(rown=1:(nrow(SSauto$all.clades)-1),coln=rep(0,nrow(SSauto$all.clades)-1),col="gray80"),
        colN.highlight=c(col="gray97"),colN.box=list(box="o",col=c("black","gray97"),lwd=1),colN.grid = "n",colN.font=2,colN.cex=0.7,
        main="all clades differences",main.font=1,main.cex=0.7)

```

Of course, the way clades are selected results in many of them being nested in each other. To avoid results redundancy, the function picks among each array of significant and nested clades the one having the largest absolute rate difference with the rest of the tree (the object `$single.clades` within the function output).

```{r,echo=FALSE,warning=FALSE,message=FALSE, fig.dim=c(4,4),out.width="70%",dpi=220}
SSauto$single.clades->tab2
knitr::kable(tab2,digits=3,align="c", caption="single clades differences") %>%
  kable_styling(full_width = FALSE, position = "center")  %>%
  column_spec(1, bold = TRUE)
```

If the clades presumed to shift are indicated (argument `node`), the function computes the difference between mean rate values of each clade and the rest of the tree, and compares it to a random distribution of differences generated by shuffling rates across tree branches (`$single.clades$singles`).

Additionally, if more than one clade is indicated, the rate difference for one clade is computed by excluding the rate values of the others from the rate vector of the rest of the tree (the object `$single.clades$no.others` within the function output). Also, all the clades are considered as to be under a common rate regime and compared as a single group to the rest of the tree (the object `$all.clades.together` within the function output).

```{r,echo=1:2,warning=FALSE,message=FALSE,out.width="100%"}
search.shift(RR,status.type = "clade",node=c(162,134,179))->SSnode

SSnode$single.clades$singles->tab3
data.frame(rownames(tab3),tab3)->tab3
SSnode$single.clades$no.others->tab4
data.frame(rownames(tab4),tab4)->tab4
rownames(tab3)<-rownames(tab4)<-NULL
SSnode$all.clades.together->tab5
data.frame("all.clades",tab5)->tab5
cbind(tab5,rep(NA,3),tab3,rep(NA,3),tab4)->tabtot
tabtot[2:3,1:3]<-NA
# cbind(rbind(tab5,rep(NA,ncol(tab5)),rep(NA,ncol(tab5))),
#       rep(NA,3),tab3,rep(NA,3),tab4)->tabtot
colnames(tabtot)<-rep(c("","rate.difference","p.value",""),length.out=11)

knitr::kable(tabtot,digits=3,align="c") %>%
  kable_styling(full_width = FALSE, position = "center")  %>%
  column_spec(c(1,5,9), bold = TRUE) %>%
  column_spec(c(3,7),border_right = "1px solid lightgray") %>%
  add_header_above( c("\\$all.clades.together"=3," "=1,"\\$single.clades\\$singles"=3," "=1,"\\$single.clades\\$no.others"=3))


```

## Sparse condition {#sparse}

Under the `sparse` condition, the function searches for shift in absolute evolutionary rates occurring in groups of phylogenetically unrelated species belonging to specific categories. In this case no estimation of categories at internal nodes is performed, so the rate shift pertains only to species. A character vector of category for each species must be supplied as `state` argument within the function.

If a binary category is under testing, the function computes the difference between the mean absolute rates of the species within the two groups and compares it to a random distribution of differences obtained by shuffling the `state` across the species.

When the `state` vector includes more than two categories, the function computes the difference in mean absolute rates between each category and the rest of the tree, and the same figure for each possible pair of categories. Again, the significance level is assessed by comparing each difference to a random distribution of differences obtained by shuffling states across species.
```{r,echo=13:14,warning=FALSE,message=FALSE}
set.seed(14)
rep("a",100)->categ
categ[sample(1:100,30)]<-"b"
names(categ)<-Tree$tip.label
y[which(categ=="b")]*5->y[which(categ=="b")]
categ->two_categ

categ[sample(1:100,20)]<-"c"
y[which(categ=="c")]*1.5->y[which(categ=="c")]
categ->three_categ
RRphylo(Tree,y,clus=cc)->RR

search.shift(RR,status.type = "sparse",state=two_categ)->SSstate2
search.shift(RR,status.type = "sparse",state=three_categ)->SSstate3

cbind(rownames(SSstate2$state.results),SSstate2$state.results)->tab2
cbind(rownames(SSstate3$state.results),SSstate3$state.results)->tab3
cbind(tab2,rep(NA,6),tab3)->tabtot
tabtot[2:6,1:3]<-NA
colnames(tabtot)<-rep(c("","rate.difference","p.value",""),length.out=7)

knitr::kable(tabtot,digits=3,align="c",) %>%
  kable_styling(full_width = FALSE, position = "center")  %>%
  column_spec(c(1,5), bold = TRUE) %>%
  column_spec(c(3,7),border_right = "1px solid lightgray") %>%
  add_header_above(c("SSstate2\\$state.results"=3," "=1,"SSstate3\\$state.results"=3))

```



## Guided examples {#examples}

```{r,out.width='99%',fig.dim=c(7,6),message=FALSE,dpi=220,warning=FALSE,echo=1:5}
# load the RRphylo example dataset including Ornithodirans tree and data
DataOrnithodirans$treedino->treedino # phylogenetic tree
DataOrnithodirans$massdino->massdino # body mass data
DataOrnithodirans$statedino->statedino # locomotory type data
log(massdino)->lmass


require(plotrix)
require(RColorBrewer)
statedino->colo
as.character(colo)->colo
names(colo)<-names(statedino)
colo[which(colo=="B")]<-brewer.pal(n = 6, name = "Set1")[6]
colo[which(colo=="BQ")]<-brewer.pal(n = 6, name = "Set1")[5]
colo[which(colo=="F")]<-brewer.pal(n = 6, name = "Set1")[3]
colo[which(colo=="Q")]<-brewer.pal(n = 6, name = "Set1")[1]
ladderize(treedino,FALSE)->treed
#par(mar=c(9,4,1,3))
plot(treed,type="fan",show.tip.label = FALSE,edge.color = "gray40",no.margin = TRUE)
tiplabels(text=rep("",419),bg=colo,frame="circle",cex=.4)
plotinfo<-get("last_plot.phylo",envir =ape::.PlotPhyloEnv)
sqrt(plotinfo$xx[416]^2+plotinfo$yy[416]^2)->rad

c(422,516,583,746,689)->clads
c("Ornithischia","Sauropodomorpha","Theropoda","Pterosauria","Avialae")->nams
#c(2,4,3,3,1)->poss
list(c(0.5,0),c(0,1),c(0.5,-0.5),c(0.3,0),c(1,0))->poss
for(k in 1:length(clads)){
  clads[k]->aa
  plotinfo$yy[which(treed$tip.label%in%tips(treed,aa)[c(1,length(tips(treed,aa)))])]->yc
  plotinfo$xx[which(treed$tip.label%in%tips(treed,aa)[c(1,length(tips(treed,aa)))])]->xc
  sqrt(xc[1]^2+yc[1]^2)->r1
  if(yc[1]>=0) acos(xc[1]/r1)->ang1 else 2*pi-acos(xc[1]/r1)->ang1
  sqrt(xc[2]^2+yc[2]^2)->r2
  if(yc[2]>=0) acos(xc[2]/r2)->ang2 else 2*pi-acos(xc[2]/r2)->ang2
  
  if(k<5){
    draw.arc(x=0,y=0,radius=rad+10,angle1 =ang1+0.05,angle2 =ang2,lwd=3,col=brewer.pal(n = 6, name = "Set1")[2]) 
    c(plotinfo$xx[which(treed$tip.label==tips(treed,aa)[length(tips(treed,aa))/2])],
      plotinfo$yy[which(treed$tip.label==tips(treed,aa)[length(tips(treed,aa))/2])])->cc
    xcc=sqrt((cc[1]^2*(180+25)^2)/(cc[1]^2+cc[2]^2))
    if(cc[1]>=0) text(x=xcc,y=cc[2]/cc[1]*xcc,labels=nams[k],font=2,adj=poss[[k]]) else text(x=(-xcc),y=cc[2]/cc[1]*-xcc,labels=nams[k],font=2,adj=poss[[k]])
  }else{
    draw.arc(x=0,y=0,radius=rad+20,angle1 =ang1+0.05,angle2 =ang2,lwd=3,col=brewer.pal(n = 6, name = "Set1")[4])
    c(plotinfo$xx[which(treed$tip.label==tips(treed,aa)[length(tips(treed,aa))/2])],
      plotinfo$yy[which(treed$tip.label==tips(treed,aa)[length(tips(treed,aa))/2])])->cc
    xcc=sqrt((cc[1]^2*(180+35)^2)/(cc[1]^2+cc[2]^2))
    if(cc[1]>=0) text(x=xcc,y=cc[2]/cc[1]*xcc,labels=nams[k],font=2,adj=poss[[k]]) else text(x=(-xcc),y=cc[2]/cc[1]*-xcc,labels=nams[k],font=2,adj=poss[[k]])
  }
}

legend("topleft",legend=c("B","BQ","F","Q"),pt.bg =unique(colo)[c(1,3,4,2)],pch=21,col="black",pt.cex=1.5,bty="n")

```

```{r,message=FALSE,eval=FALSE}
# check the order of your data: best if data vectors
# are sorted in the same order of the species on the phylogeny
lmass[match(treedino$tip.label,names(lmass))]->lmass
statedino[match(treedino$tip.label,names(statedino))]->statedino

# perform RRphylo on the vector of (log) body mass
RRphylo(tree=treedino,y=lmass)->RRdinomass

# search for clades showing significant shifts in mass specific evolutionary rates
# (i.e. using the log body mass itself as a covariate)
search.shift(RRdinomass, status.type= "clade",cov=lmass)->SSauto

# search for shifts in mass specific evolutionary rates pertaining different locomotory types.
search.shift(RRdinomass, status.type= "sparse", state=statedino,cov=lmass)->SSstate
```

## References

Castiglione, S., Tesone, G., Piccolo, M., Melchionna, M., Mondanaro, A., Serio, C., Di Febbraro, M., & Raia, P.(2018). A new method for testing evolutionary rate variation and shifts in phenotypic evolution. Methods in Ecology and Evolution, 9, 974-983.
