batch_correction_analysis_LungAtlas.Rmd

---
title: "MultiNicheNet analysis: Integrated lung atlas analysis - correct for batch effects to infer differences between IPF and healthy subjects"
author: "Robin Browaeys"
package: "`r BiocStyle::pkg_ver('multinichenetr')`"
output: 
  BiocStyle::html_document
output_dir: "/Users/robinb/Work/multinichenetr/vignettes"
vignette: >
  %\VignetteIndexEntry{MultiNicheNet analysis: Integrated lung atlas analysis - correct for batch effects to infer differences between IPF and healthy subjects}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8} 
---

<style type="text/css">
.smaller {
  font-size: 10px
}
</style>

<!-- github markdown built using
rmarkdown::render("vignettes/batch_correction_analysis_LungAtlas.Rmd", clean = FALSE )
-->

```{r setup, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  # comment = "#>",
  warning = FALSE,
  message = FALSE
)
library(BiocStyle)
```

In this vignette, you can learn how to perform a MultiNicheNet analysis to compare cell-cell communication between two conditions of interest (one-vs-one comparison) while correcting for __batch effects__. A MultiNicheNet analysis can be performed if you have multi-sample, multi-condition/group single-cell data. We strongly recommend having at least 4 samples in each of the groups/conditions you want to compare. With less samples, the benefits of performing a pseudobulk-based DE analysis are less clear. For those datasets, you can check and run our alternative workflow that makes use of cell-level sample-agnostic differential expression tools.

As input you need a SingleCellExperiment object containing at least the raw count matrix and metadata providing the following information for each cell: the **group**, **sample** and **cell type**.

As example expression data of interacting cells, we will here use merged scRNAseq data from four studies comparing healthy lungs to lungs from patients with idiopathic pulmonary fibrosis (IPF)) (Adams 2020, Reyfman 2019, Morse 2019, and Habermann 2020). Harmonized cell type annotations across the 4 different studies were obtained through Azimuth [Azimuth meta-analysis of human scRNA-seq datasets](https://cellxgene.cziscience.com/collections/2f75d249-1bec-459b-bf2b-b86221097ced).

We will here demonstrate how MultiNicheNet can exploit the flexibility of generalized linear models in the pseudobulk-edgeR framework to correct for batch effects, here the source study: Adams 2020, Reyfman 2019, Morse 2019, or Habermann 2020. We will apply MultiNicheNet to compare cell-cell interaction changes between IPF and healthy tissue. Note that the only required input for a batch-correcting MultiNicheNet analysis is a merged scRNA-seq object containing raw counts and harmonized cell type annotations.

We will first prepare the MultiNicheNet core analysis, then run the several steps in the MultiNicheNet core analysis, and finally interpret the output.

# Preparation of the MultiNicheNet core analysis

```{r load-libs, message = FALSE,  warning = FALSE}
library(SingleCellExperiment)
library(dplyr)
library(ggplot2)
library(nichenetr)
library(multinichenetr)
```

## Load NicheNet's ligand-receptor network and ligand-target matrix

MultiNicheNet builds upon the NicheNet framework and uses the same prior knowledge networks (ligand-receptor network and ligand-target matrix, currently v2 version).

The Nichenet v2 networks and matrices for both mouse and human can be downloaded from Zenodo [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.7074291.svg)](https://doi.org/10.5281/zenodo.7074291). 

We will read these object in for human because our expression data is of human patients. 
Gene names are here made syntactically valid via `make.names()` to avoid the loss of genes (eg H2-M3) in downstream visualizations.

```{r}
organism = "human"
```

```{r, results='hide'}
options(timeout = 120)

if(organism == "human"){
  
  lr_network_all = 
    readRDS(url(
      "https://zenodo.org/record/10229222/files/lr_network_human_allInfo_30112033.rds"
      )) %>% 
    mutate(
      ligand = convert_alias_to_symbols(ligand, organism = organism), 
      receptor = convert_alias_to_symbols(receptor, organism = organism))
  
  lr_network_all = lr_network_all  %>% 
    mutate(ligand = make.names(ligand), receptor = make.names(receptor)) 
  
  lr_network = lr_network_all %>% 
    distinct(ligand, receptor)
  
  ligand_target_matrix = readRDS(url(
    "https://zenodo.org/record/7074291/files/ligand_target_matrix_nsga2r_final.rds"
    ))
  
  colnames(ligand_target_matrix) = colnames(ligand_target_matrix) %>% 
    convert_alias_to_symbols(organism = organism) %>% make.names()
  rownames(ligand_target_matrix) = rownames(ligand_target_matrix) %>% 
    convert_alias_to_symbols(organism = organism) %>% make.names()
  
  lr_network = lr_network %>% filter(ligand %in% colnames(ligand_target_matrix))
  ligand_target_matrix = ligand_target_matrix[, lr_network$ligand %>% unique()]
  
} else if(organism == "mouse"){
  
  lr_network_all = readRDS(url(
    "https://zenodo.org/record/10229222/files/lr_network_mouse_allInfo_30112033.rds"
    )) %>% 
    mutate(
      ligand = convert_alias_to_symbols(ligand, organism = organism), 
      receptor = convert_alias_to_symbols(receptor, organism = organism))
  
  lr_network_all = lr_network_all  %>% 
    mutate(ligand = make.names(ligand), receptor = make.names(receptor)) 
  lr_network = lr_network_all %>% 
    distinct(ligand, receptor)
  
  ligand_target_matrix = readRDS(url(
    "https://zenodo.org/record/7074291/files/ligand_target_matrix_nsga2r_final_mouse.rds"
    ))
  
  colnames(ligand_target_matrix) = colnames(ligand_target_matrix) %>% 
    convert_alias_to_symbols(organism = organism) %>% make.names()
  rownames(ligand_target_matrix) = rownames(ligand_target_matrix) %>% 
    convert_alias_to_symbols(organism = organism) %>% make.names()
  
  lr_network = lr_network %>% filter(ligand %in% colnames(ligand_target_matrix))
  ligand_target_matrix = ligand_target_matrix[, lr_network$ligand %>% unique()]
  
}
```

## Read in SingleCellExperiment Object

In this vignette, we will load in a subset of the integrated lung atlas data. For the sake of demonstration, this subset only contains 4 cell types. 
If you start from a Seurat object, you can convert it easily to a SingleCellExperiment object via `sce = Seurat::as.SingleCellExperiment(seurat_obj, assay = "RNA")`.

Because the NicheNet 2.0. networks are in the most recent version of the official gene symbols, we will make sure that the gene symbols used in the expression data are also updated (= converted from their "aliases" to official gene symbols). Afterwards, we will make them again syntactically valid. 

```{r, results='hide'}
sce = readRDS(url(
  "https://zenodo.org/record/8010790/files/sce_subset_lung.rds"
  ))
sce = alias_to_symbol_SCE(sce, "human") %>% makenames_SCE()
```

## Prepare the settings of the MultiNicheNet cell-cell communication analysis

In this step, we will formalize our research question into MultiNicheNet input arguments.

### Define in which metadata columns we can find the **group**, **sample** and **cell type** IDs

In this case study, we want to study differences in cell-cell communication changes between two patient groups (IPF patients vs healthy subjects), while considering the source dataset cells were derived from. Patient groups are indicated in the following meta data column: `disease`, which has 2 different values: idiopathic.pulmonary.fibrosis and normal.

Cell type annotations are indicated in the `annotation.l1` column, and the sample is indicated by the `donor` column. 
If your cells are annotated in multiple hierarchical levels, we recommend using a relatively high level in the hierarchy. This for 2 reasons: 1) MultiNicheNet focuses on differential expression and not differential abundance, and 2) there should be sufficient cells per sample-celltype combination (see later).

```{r}
sample_id = "donor"
group_id = "disease"
celltype_id = "annotation.l1"
```

__Important__: It is required that each sample-id is uniquely assigned to only one condition/group of interest. See the vignettes about paired and multifactorial analysis to see how to define your analysis input when you have multiple samples (and conditions) per patient.

If you would have batch effects or covariates you can correct for, you can define this here as well. Here, we want to correct for the source study, which is indicated in the following meta data column: `dataset_origin`, which has 4 different values: adams_2020, habermann_2020, morse_2019, reyfman_2019 and normal.

```{r}
covariates = NA
batches = "dataset_origin"
```

__Important__: for categorical covariates and batches, there should be at least one sample for every group-batch combination. If one of your groups/conditions lacks a certain level of your batch, you won't be able to correct for the batch effect because the model is then not able to distinguish batch from group/condition effects.

__Important__: The column names of group, sample, cell type, batches and covariates should be syntactically valid (`make.names`)

__Important__: All group, sample, cell type, batch and covariate names should be syntactically valid as well (`make.names`) (eg through `SummarizedExperiment::colData(sce)$ShortID = SummarizedExperiment::colData(sce)$ShortID %>% make.names()`)

### Define the contrasts of interest.

For this analysis, we want to compare how cell-cell communication differs between IPF and normal lungs.

```{r}
contrasts_oi = c("'idiopathic.pulmonary.fibrosis-normal','normal-idiopathic.pulmonary.fibrosis'")
```

__Very Important__ Note the format to indicate the contrasts! This formatting should be adhered to very strictly, and white spaces are not allowed! Check `?get_DE_info` for explanation about how to define this well. The most important points are that: 
*each contrast is surrounded by single quotation marks
*contrasts are separated by a comma without any white space 
*all contrasts together are surrounded by double quotation marks. 

If you compare against two groups, you should divide by 2 (as demonstrated in other vignettes), if you compare against three groups, you should divide by 3 and so on.

For downstream visualizations and linking contrasts to their main condition, we also need to run the following:
This is necessary because we will also calculate cell-type+condition specificity of ligands and receptors. 

```{r}
contrast_tbl = tibble(contrast =
                        c("idiopathic.pulmonary.fibrosis-normal", "normal-idiopathic.pulmonary.fibrosis"),
                      group = c("idiopathic.pulmonary.fibrosis", "normal"))
```

Other vignettes will demonstrate how to formalize different types of research questions.

### Define the sender and receiver cell types of interest.

If you want to focus the analysis on specific cell types (e.g. because you know which cell types reside in the same microenvironments based on spatial data), you can define this here. If you have sufficient computational resources and no specific idea of cell-type colocalzations, we recommend to consider all cell types as potential senders and receivers. Later on during analysis of the output it is still possible to zoom in on the cell types that interest you most, but your analysis is not biased to them.

Here we will consider all cell types in the data:

```{r}
senders_oi = SummarizedExperiment::colData(sce)[,celltype_id] %>% unique()
receivers_oi = SummarizedExperiment::colData(sce)[,celltype_id] %>% unique()
sce = sce[, SummarizedExperiment::colData(sce)[,celltype_id] %in% 
            c(senders_oi, receivers_oi)
          ]
```

In case you would have samples in your data that do not belong to one of the groups/conditions of interest, we recommend removing them and only keeping conditions of interest. This may be especially relevant in atlas settings where you may have cells from patients belonging to diseases that are outside of the scope of the current comparison.

```{r}
conditions_keep = c("normal", "idiopathic.pulmonary.fibrosis")
sce = sce[, SummarizedExperiment::colData(sce)[,group_id] %in% 
            conditions_keep
          ]
```

# Running the MultiNicheNet core analysis

Now we will run the core of a MultiNicheNet analysis. This analysis consists of the following steps:

* 1. Cell-type filtering: determine which cell types are sufficiently present
* 2. Gene filtering: determine which genes are sufficiently expressed in each present cell type
* 3. Pseudobulk expression calculation: determine and normalize per-sample pseudobulk expression levels for each expressed gene in each present cell type
* 4. Differential expression (DE) analysis: determine which genes are differentially expressed
* 5. Ligand activity prediction: use the DE analysis output to predict the activity of ligands in receiver cell types and infer their potential target genes
* 6. Prioritization: rank cell-cell communication patterns through multi-criteria prioritization

Following these steps, one can optionally 
* 7. Calculate the across-samples expression correlation between ligand-receptor pairs and target genes
* 8. Prioritize communication patterns involving condition-specific cell types through an alternative prioritization scheme

After these steps, the output can be further explored as we will demonstrate in the "Downstream analysis of the MultiNicheNet output" section. 

In this vignette, we will demonstrate these steps one-by-one, which offers the most flexibility to the user to assess intermediary results. Other vignettes will demonstrate the use of the `multi_nichenet_analysis` wrapper function.

## Cell-type filtering: determine which cell types are sufficiently present

In this step we will calculate and visualize cell type abundances. This will give an indication about which cell types will be retained in the analysis, and which cell types will be filtered out.  
Since MultiNicheNet will infer group differences at the sample level for each cell type (currently via Muscat - pseudobulking + EdgeR), we need to have sufficient cells per sample of a cell type, and this for all groups. In the following analysis we will set this minimum number of cells per cell type per sample at 10. Samples that have less than `min_cells` cells will be excluded from the analysis for that specific cell type.
 
```{r}
min_cells = 10
```

We recommend using `min_cells = 10`, except for datasets with several lowly abundant cell types of interest. For those datasets, we recommend using `min_cells = 5`.

```{r}
abundance_info = get_abundance_info(
  sce = sce, 
  sample_id = sample_id, group_id = group_id, celltype_id = celltype_id, 
  min_cells = min_cells, 
  senders_oi = senders_oi, receivers_oi = receivers_oi, 
  batches = batches
  )
```

First, we will check the cell type abundance diagnostic plots.

### Interpretation of cell type abundance information

The first plot visualizes the number of cells per celltype-sample combination, and indicates which combinations are removed during the DE analysis because there are less than `min_cells` in the celltype-sample combination. 

```{r, fig.width=25}
abundance_info$abund_plot_sample
```

The red dotted line indicates the required minimum of cells as defined above in `min_cells`. We can see here that some sample-celltype combinations are left out. For the DE analysis in the next step, only cell types will be considered if there are at least two samples per condition-batch combination with a sufficient number of cells. 

__Important__: Based on the cell type abundance diagnostics, we recommend users to change their analysis settings if required (eg changing cell type annotation level, batches, ...), before proceeding with the rest of the analysis. If too many celltype-sample combinations don't pass this threshold, we recommend to define your cell types in a more general way (use one level higher of the cell type ontology hierarchy) (eg TH17 CD4T cells --> CD4T cells) or use `min_cells = 5` if this would not be possible.


### Cell type filtering based on cell type abundance information

Running the following block of code can help you determine which cell types are condition-specific and which cell types are absent. 

```{r}
abundance_df_summarized = abundance_info$abundance_data %>% 
  mutate(keep = as.logical(keep)) %>% 
  group_by(group_id, celltype_id) %>% 
  summarise(samples_present = sum((keep)))

celltypes_absent_one_condition = abundance_df_summarized %>% 
  filter(samples_present == 0) %>% pull(celltype_id) %>% unique() 
# find truly condition-specific cell types by searching for cell types 
# truely absent in at least one condition

celltypes_present_one_condition = abundance_df_summarized %>% 
  filter(samples_present >= 2) %>% pull(celltype_id) %>% unique() 
# require presence in at least 2 samples of one group so 
# it is really present in at least one condition

condition_specific_celltypes = intersect(
  celltypes_absent_one_condition, 
  celltypes_present_one_condition)

total_nr_conditions = SummarizedExperiment::colData(sce)[,group_id] %>% 
  unique() %>% length() 

absent_celltypes = abundance_df_summarized %>% 
  filter(samples_present < 2) %>% 
  group_by(celltype_id) %>% 
  count() %>% 
  filter(n == total_nr_conditions) %>% 
  pull(celltype_id)
  
print("condition-specific celltypes:")
print(condition_specific_celltypes)
  
print("absent celltypes:")
print(absent_celltypes)
```
Absent cell types will be filtered out, condition-specific cell types can be filtered out if you as a user do not want to run the alternative workflow for condition-specific cell types in the optional step 8 of the core MultiNicheNet analysis. 

```{r}
analyse_condition_specific_celltypes = FALSE
```

```{r}
if(analyse_condition_specific_celltypes == TRUE){
  senders_oi = senders_oi %>% setdiff(absent_celltypes)
  receivers_oi = receivers_oi %>% setdiff(absent_celltypes)
} else {
  senders_oi = senders_oi %>% 
    setdiff(union(absent_celltypes, condition_specific_celltypes))
  receivers_oi = receivers_oi %>% 
    setdiff(union(absent_celltypes, condition_specific_celltypes))
}

sce = sce[, SummarizedExperiment::colData(sce)[,celltype_id] %in% 
            c(senders_oi, receivers_oi)
          ]
```

## Gene filtering: determine which genes are sufficiently expressed in each present cell type

Before running the DE analysis, we will determine which genes are not sufficiently expressed and should be filtered out. 
We will perform gene filtering based on a similar procedure as used in `edgeR::filterByExpr`. However, we adapted this procedure to be more interpretable for single-cell datasets.  

For each cell type, we will consider genes expressed if they are expressed in at least a `min_sample_prop` fraction of samples in the condition with the lowest number of samples. By default, we set `min_sample_prop = 0.50`, which means that genes should be expressed in at least 2 samples if the group with lowest nr. of samples has 4 samples like this dataset. 

```{r}
min_sample_prop = 0.50
```

But how do we define which genes are expressed in a sample? For this we will consider genes as expressed if they have non-zero expression values in a `fraction_cutoff` fraction of cells of that cell type in that sample. By default, we set `fraction_cutoff = 0.05`, which means that genes should show non-zero expression values in at least 5% of cells in a sample. 

```{r}
fraction_cutoff = 0.05
```

We recommend using these default values unless there is specific interest in prioritizing (very) weakly expressed interactions. In that case, you could lower the value of `fraction_cutoff`. We explicitly recommend against using `fraction_cutoff > 0.10`.

Now we will calculate the information required for gene filtering with the following command:

```{r}
frq_list = get_frac_exprs(
  sce = sce, 
  sample_id = sample_id, celltype_id =  celltype_id, group_id = group_id, 
  batches = batches, 
  min_cells = min_cells, 
  fraction_cutoff = fraction_cutoff, min_sample_prop = min_sample_prop)
```

Now only keep genes that are expressed by at least one cell type:

```{r}
genes_oi = frq_list$expressed_df %>% 
  filter(expressed == TRUE) %>% pull(gene) %>% unique() 
sce = sce[genes_oi, ]
```

## Pseudobulk expression calculation: determine and normalize per-sample pseudobulk expression levels for each expressed gene in each present cell type

After filtering out absent cell types and genes, we will continue the analysis by calculating the different prioritization criteria that we will use to prioritize cell-cell communication patterns.

First, we will determine and normalize per-sample pseudobulk expression levels for each expressed gene in each present cell type. The function `process_abundance_expression_info` will link this expression information for ligands of the sender cell types to the corresponding receptors of the receiver cell types. This will later on allow us to define the cell-type specicificy criteria for ligands and receptors.

```{r}
abundance_expression_info = process_abundance_expression_info(
  sce = sce, 
  sample_id = sample_id, group_id = group_id, celltype_id = celltype_id, 
  min_cells = min_cells, 
  senders_oi = senders_oi, receivers_oi = receivers_oi, 
  lr_network = lr_network, 
  batches = batches, 
  frq_list = frq_list, 
  abundance_info = abundance_info)
```

Normalized pseudobulk expression values per gene/celltype/sample can be inspected by:

```{r}
abundance_expression_info$celltype_info$pb_df %>% head()
```

An average of these sample-level expression values per condition/group can be inspected by:

```{r}
abundance_expression_info$celltype_info$pb_df_group %>% head()
```

Inspecting these values for ligand-receptor interactions can be done by:

```{r}
abundance_expression_info$sender_receiver_info$pb_df %>% head()
abundance_expression_info$sender_receiver_info$pb_df_group %>% head()
```

## Differential expression (DE) analysis: determine which genes are differentially expressed

In this step, we will perform genome-wide differential expression analysis of receiver and sender cell types to define DE genes between the conditions of interest (as formalized by the `contrasts_oi`). Based on this analysis, we later can define the levels of differential expression of ligands in senders and receptors in receivers, and define the set of affected target genes in the receiver cell types (which will be used for the ligand activity analysis).

We will apply pseudobulking followed by EdgeR to perform multi-condition multi-sample differential expression (DE) analysis (also called 'differential state' analysis by the developers of Muscat). 

```{r}
DE_info = get_DE_info(
  sce = sce, 
  sample_id = sample_id, group_id = group_id, celltype_id = celltype_id, 
  batches = batches, covariates = covariates, 
  contrasts_oi = contrasts_oi, 
  min_cells = min_cells, 
  expressed_df = frq_list$expressed_df)
```

### Check DE results

Check DE output information in table with logFC and p-values for each gene-celltype-contrast:

```{r}
DE_info$celltype_de$de_output_tidy %>% head()
```
Evaluate the distributions of p-values:

```{r}
DE_info$hist_pvals
```

These distributions look fine (uniform distribution, except peak at p-value <= 0.05), so we will continue using these regular p-values. In case these p-value distributions look irregular, you can estimate empirical p-values as we will demonstrate in another vignette.

```{r}
empirical_pval = FALSE
```

```{r}
if(empirical_pval == TRUE){
  DE_info_emp = get_empirical_pvals(DE_info$celltype_de$de_output_tidy)
  celltype_de = DE_info_emp$de_output_tidy_emp %>% select(-p_val, -p_adj) %>% 
    rename(p_val = p_emp, p_adj = p_adj_emp)
} else {
  celltype_de = DE_info$celltype_de$de_output_tidy
} 
```

### Combine DE information for ligand-senders and receptors-receivers

To end this step, we will combine the DE information of senders and receivers by linking their ligands and receptors together based on the prior knowledge ligand-receptor network.

```{r}
sender_receiver_de = combine_sender_receiver_de(
  sender_de = celltype_de,
  receiver_de = celltype_de,
  senders_oi = senders_oi,
  receivers_oi = receivers_oi,
  lr_network = lr_network
)
```

```{r}
sender_receiver_de %>% head(20)
```

## Ligand activity prediction: use the DE analysis output to predict the activity of ligands in receiver cell types and infer their potential target genes

In this step, we will predict NicheNet ligand activities and NicheNet ligand-target links based on these differential expression results. We do this to prioritize interactions based on their predicted effect on a receiver cell type. We will assume that the most important group-specific interactions are those that lead to group-specific gene expression changes in a receiver cell type.

Similarly to base NicheNet (https://github.com/saeyslab/nichenetr), we use the DE output to create a "geneset of interest": here we assume that DE genes within a cell type may be DE because of differential cell-cell communication processes. In the ligand activity prediction, we will assess the enrichment of target genes of ligands within this geneset of interest. In case high-probabiliy target genes of a ligand are enriched in this set compared to the background of expressed genes, we predict that this ligand may have a high activity. 

Because the ligand activity analysis is an enrichment procedure, it is important that this geneset of interest should contain a sufficient but not too large number of genes. The ratio geneset_oi/background should ideally be between 1/200 and 1/10 (or close to these ratios).

To determine the genesets of interest based on DE output, we need to define some logFC and/or p-value thresholds per cell type/contrast combination. In general, we recommend inspecting the nr. of DE genes for all cell types based on the default thresholds and adapting accordingly. By default, we will apply the p-value cutoff on the normal p-values, and not on the p-values corrected for multiple testing. This choice was made because most multi-sample single-cell transcriptomics datasets have just a few samples per group and we might have a lack of statistical power due to pseudobulking. But, if the smallest group >= 20 samples, we typically recommend using p_val_adj = TRUE. When the biological difference between the conditions is very large, we typically recommend increasing the logFC_threshold and/or using p_val_adj = TRUE.

### Assess geneset_oi-vs-background ratios for different DE output tresholds prior to the NicheNet ligand activity analysis 

Because we have data with many samples here, we will first inspect the geneset_oi-vs-background ratios in case of using the adjusted p-values:

```{r}
logFC_threshold = 0.50
p_val_threshold = 0.05
```

```{r}
p_val_adj = TRUE 
```

```{r}
geneset_assessment = contrast_tbl$contrast %>% 
  lapply(
    process_geneset_data, 
    celltype_de, logFC_threshold, p_val_adj, p_val_threshold
  ) %>% 
  bind_rows() 
geneset_assessment
```
We can see here that for all cell type / contrast combinations, all geneset/background ratio's are within the recommended range (`in_range_up` and `in_range_down` columns), and we will therefore proceed with these tresholds for the ligand activity analysis. When these geneset/background ratio's would not be within the recommended ranges, we should interpret ligand activity results for these cell types with more caution, or use different thresholds (for these or all cell types). 

### Perform the ligand activity analysis and ligand-target inference

After the ligand activity prediction, we will also infer the predicted target genes of these ligands in each contrast. For this ligand-target inference procedure, we also need to select which top n of the predicted target genes will be considered (here: top 250 targets per ligand). This parameter will not affect the ligand activity predictions. It will only affect ligand-target visualizations and construction of the intercellular regulatory network during the downstream analysis. We recommend users to test other settings in case they would be interested in exploring fewer, but more confident target genes, or vice versa. 

```{r}
top_n_target = 250
```

The NicheNet ligand activity analysis can be run in parallel for each receiver cell type, by changing the number of cores as defined here. Using more cores will speed up the analysis at the cost of needing more memory. This is only recommended if you have many receiver cell types of interest. 

```{r}
verbose = TRUE
cores_system = 8
n.cores = min(cores_system, celltype_de$cluster_id %>% unique() %>% length()) 
```

Running the ligand activity prediction will take some time (the more cell types and contrasts, the more time)

```{r}
ligand_activities_targets_DEgenes = suppressMessages(suppressWarnings(
  get_ligand_activities_targets_DEgenes(
    receiver_de = celltype_de,
    receivers_oi = intersect(receivers_oi, celltype_de$cluster_id %>% unique()),
    ligand_target_matrix = ligand_target_matrix,
    logFC_threshold = logFC_threshold,
    p_val_threshold = p_val_threshold,
    p_val_adj = p_val_adj,
    top_n_target = top_n_target,
    verbose = verbose, 
    n.cores = n.cores
  )
))
```

You can check the output of the ligand activity and ligand-target inference here:

```{r}
ligand_activities_targets_DEgenes$ligand_activities %>% head(20)
```

## Prioritization: rank cell-cell communication patterns through multi-criteria prioritization

In the previous steps, we calculated expression, differential expression and NicheNet ligand activity. In the final step, we will now combine all calculated information to rank all sender-ligand---receiver-receptor pairs according to group/condition specificity. We will use the following criteria to prioritize ligand-receptor interactions:

* Upregulation of the ligand in a sender cell type and/or upregulation of the receptor in a receiver cell type - in the condition of interest.
* Cell-type specific expression of the ligand in the sender cell type and receptor in the receiver cell type in the condition of interest (to mitigate the influence of upregulated but still relatively weakly expressed ligands/receptors). 
* Sufficiently high expression levels of ligand and receptor in many samples of the same group.
* High NicheNet ligand activity, to further prioritize ligand-receptor pairs based on their predicted effect of the ligand-receptor interaction on the gene expression in the receiver cell type. 

We will combine these prioritization criteria in a single aggregated prioritization score. In the default setting, we will weigh each of these criteria equally (`scenario = "regular"`). This setting is strongly recommended. However, we also provide some additional setting to accomodate different biological scenarios. The setting `scenario = "lower_DE"` halves the weight for DE criteria and doubles the weight for ligand activity. This is recommended in case your hypothesis is that the differential CCC patterns in your data are less likely to be driven by DE (eg in cases of differential migration into a niche). The setting `scenario = "no_frac_LR_expr"` ignores the criterion "Sufficiently high expression levels of ligand and receptor in many samples of the same group". This may be interesting for users that have data with a limited number of samples and don’t want to penalize interactions if they are not sufficiently expressed in some samples. 

Finally, we still need to make one choice. For NicheNet ligand activity we can choose to prioritize ligands that only induce upregulation of target genes (`ligand_activity_down = FALSE`) or can lead potentially lead to both up- and downregulation (`ligand_activity_down = TRUE`). The benefit of `ligand_activity_down = FALSE` is ease of interpretability: prioritized ligand-receptor pairs will be upregulated in the condition of interest, just like their target genes.  `ligand_activity_down = TRUE` can be harder to interpret because target genes of some interactions may be upregulated in the other conditions compared to the condition of interest. This is harder to interpret, but may help to pick up interactions that can also repress gene expression. 

Here we will choose for setting `ligand_activity_down = FALSE` and focus specifically on upregulating ligands.

```{r}
ligand_activity_down = FALSE
```

```{r}
sender_receiver_tbl = sender_receiver_de %>% distinct(sender, receiver)

metadata_combined = SummarizedExperiment::colData(sce) %>% tibble::as_tibble()

if(!is.na(batches)){
  grouping_tbl = metadata_combined[,c(sample_id, group_id, batches)] %>% 
    tibble::as_tibble() %>% distinct()
  colnames(grouping_tbl) = c("sample","group",batches)
} else {
  grouping_tbl = metadata_combined[,c(sample_id, group_id)] %>% 
    tibble::as_tibble() %>% distinct()
  colnames(grouping_tbl) = c("sample","group")
}

prioritization_tables = suppressMessages(generate_prioritization_tables(
    sender_receiver_info = abundance_expression_info$sender_receiver_info,
    sender_receiver_de = sender_receiver_de,
    ligand_activities_targets_DEgenes = ligand_activities_targets_DEgenes,
    contrast_tbl = contrast_tbl,
    sender_receiver_tbl = sender_receiver_tbl,
    grouping_tbl = grouping_tbl,
    scenario = "regular", # all prioritization criteria will be weighted equally
    fraction_cutoff = fraction_cutoff, 
    abundance_data_receiver = abundance_expression_info$abundance_data_receiver,
    abundance_data_sender = abundance_expression_info$abundance_data_sender,
    ligand_activity_down = ligand_activity_down
  ))
```

Check the output tables

First: group-based summary table

```{r}
prioritization_tables$group_prioritization_tbl %>% head(20)
```
This table gives the final prioritization score of each interaction, and the values of the individual prioritization criteria.

With this step, all required steps are finished. Now, we can optionally still run the following steps
* Calculate the across-samples expression correlation between ligand-receptor pairs and target genes
* Prioritize communication patterns involving condition-specific cell types through an alternative prioritization scheme

Here we will only focus on the expression correlation step:

## Calculate the across-samples expression correlation between ligand-receptor pairs and target genes

In multi-sample datasets, we have the opportunity to look whether expression of ligand-receptor across all samples is correlated with the expression of their by NicheNet predicted target genes. This is what we will do with the following line of code:

```{r}
lr_target_prior_cor = lr_target_prior_cor_inference(
  receivers_oi = prioritization_tables$group_prioritization_tbl$receiver %>% unique(), 
  abundance_expression_info = abundance_expression_info, 
  celltype_de = celltype_de, 
  grouping_tbl = grouping_tbl, 
  prioritization_tables = prioritization_tables, 
  ligand_target_matrix = ligand_target_matrix, 
  logFC_threshold = logFC_threshold, 
  p_val_threshold = p_val_threshold, 
  p_val_adj = p_val_adj
  )
```

## Save all the output of MultiNicheNet 

To avoid needing to redo the analysis later, we will here to save an output object that contains all information to perform all downstream analyses.

```{r}
path = "./"

multinichenet_output = list(
    celltype_info = abundance_expression_info$celltype_info,
    celltype_de = celltype_de,
    sender_receiver_info = abundance_expression_info$sender_receiver_info,
    sender_receiver_de =  sender_receiver_de,
    ligand_activities_targets_DEgenes = ligand_activities_targets_DEgenes,
    prioritization_tables = prioritization_tables,
    grouping_tbl = grouping_tbl,
    lr_target_prior_cor = lr_target_prior_cor
  ) 
multinichenet_output = make_lite_output(multinichenet_output)

save = FALSE
if(save == TRUE){
  saveRDS(multinichenet_output, paste0(path, "multinichenet_output.rds"))

}
```

# Interpreting the MultiNicheNet analysis output

## Visualization of differential cell-cell interactions

### Summarizing ChordDiagram circos plots

In a first instance, we will look at the broad overview of prioritized interactions via condition-specific Chordiagram circos plots.

We will look here at the top 50 predictions across all contrasts, senders, and receivers of interest.

```{r}
prioritized_tbl_oi_all = get_top_n_lr_pairs(
  multinichenet_output$prioritization_tables, 
  top_n = 50, 
  rank_per_group = FALSE
  )
```

```{r}
prioritized_tbl_oi = 
  multinichenet_output$prioritization_tables$group_prioritization_tbl %>%
  filter(id %in% prioritized_tbl_oi_all$id) %>%
  distinct(id, sender, receiver, ligand, receptor, group) %>% 
  left_join(prioritized_tbl_oi_all)
prioritized_tbl_oi$prioritization_score[is.na(prioritized_tbl_oi$prioritization_score)] = 0

senders_receivers = union(prioritized_tbl_oi$sender %>% unique(), prioritized_tbl_oi$receiver %>% unique()) %>% sort()

colors_sender = RColorBrewer::brewer.pal(n = length(senders_receivers), name = 'Spectral') %>% magrittr::set_names(senders_receivers)
colors_receiver = RColorBrewer::brewer.pal(n = length(senders_receivers), name = 'Spectral') %>% magrittr::set_names(senders_receivers)

circos_list = make_circos_group_comparison(prioritized_tbl_oi, colors_sender, colors_receiver)
```

### Interpretable bubble plots

Whereas these ChordDiagrams show the most specific interactions per group, they don't give insights into the data behind these predictions. Therefore we will now look at visualizations that indicate the different prioritization criteria used in MultiNicheNet. 

In the next type of plots, we will 1) visualize the per-sample scaled product of normalized ligand and receptor pseudobulk expression, 2) visualize the scaled ligand activities, 3) cell-type specificity. 

We will now check the top 50 interactions specific for the Tumor-tissue

```{r}
group_oi = "idiopathic.pulmonary.fibrosis"
```

```{r}
prioritized_tbl_oi_IPF_50 = get_top_n_lr_pairs(
  multinichenet_output$prioritization_tables, 
  top_n = 50, 
  groups_oi = group_oi)
```

```{r, fig.height=13, fig.width=28}
plot_oi = make_sample_lr_prod_activity_plots(
  multinichenet_output$prioritization_tables, 
  prioritized_tbl_oi_IPF_50)
plot_oi
```
Samples that were left out of the DE analysis are indicated with a smaller dot (this helps to indicate the samples that did not contribute to the calculation of the logFC, and thus not contributed to the final prioritization)

As a further help for further prioritization, we can assess the level of curation of these LR pairs as defined by the Intercellular Communication part of the Omnipath database

```{r}
prioritized_tbl_oi_IPF_50_omnipath = prioritized_tbl_oi_IPF_50 %>% 
  inner_join(lr_network_all)
```

Now we add this to the bubble plot visualization:
```{r, fig.height=13, fig.width=28}
plot_oi = make_sample_lr_prod_activity_plots_Omnipath(
  multinichenet_output$prioritization_tables, 
  prioritized_tbl_oi_IPF_50_omnipath)
plot_oi
```

Further note: Typically, there are way more than 50 differentially expressed and active ligand-receptor pairs per group across all sender-receiver combinations. Therefore it might be useful to zoom in on specific cell types as senders/receivers:

Eg CD16..Monocyte as receiver:

```{r}
prioritized_tbl_oi_IPF_50 = get_top_n_lr_pairs(
  multinichenet_output$prioritization_tables, 
  50, 
  groups_oi = group_oi, 
  receivers_oi = "CD16..Monocyte")
```

```{r, fig.height=13, fig.width=28}
plot_oi = make_sample_lr_prod_activity_plots_Omnipath(
  multinichenet_output$prioritization_tables, 
  prioritized_tbl_oi_IPF_50 %>% inner_join(lr_network_all))
plot_oi
```

Eg CD14.Monocyte as sender:

```{r}
prioritized_tbl_oi_IPF_50 = get_top_n_lr_pairs(
  multinichenet_output$prioritization_tables, 
  50, 
  groups_oi = group_oi, 
  senders_oi = "CD16..Monocyte")
```

```{r, fig.height=13, fig.width=30}
plot_oi = make_sample_lr_prod_activity_plots_Omnipath(
  multinichenet_output$prioritization_tables, 
  prioritized_tbl_oi_IPF_50 %>% inner_join(lr_network_all))
plot_oi
```

You can make these plots also for the other groups, like we will illustrate now for the S group

```{r}
group_oi = "normal"
```

```{r, fig.height=13, fig.width=28}
prioritized_tbl_oi_Normal_50 = get_top_n_lr_pairs(
  multinichenet_output$prioritization_tables, 
  50, 
  groups_oi = group_oi)

plot_oi = make_sample_lr_prod_activity_plots_Omnipath(
  multinichenet_output$prioritization_tables, 
  prioritized_tbl_oi_Normal_50 %>% inner_join(lr_network_all))
plot_oi
```

__Note__: We can use `make_sample_lr_prod_activity_batch_plots` to visualize batches on this plot!

```{r, fig.height=13, fig.width=28}
plot_oi = make_sample_lr_prod_activity_batch_plots(
  multinichenet_output$prioritization_tables, 
  prioritized_tbl_oi_IPF_50, 
  multinichenet_output$grouping_tbl, 
  batches)
plot_oi
```

## Visualization of differential ligand-target links

### Without filtering of target genes based on LR-target expression correlation

In another type of plot, we can visualize the ligand activities for a group-receiver combination, and show the predicted ligand-target links, and also the expression of the predicted target genes across samples.

For this, we now need to define a receiver cell type of interest. As example, we will take `CLEC9A` cells as receiver, and look at the top 10 senderLigand-receiverReceptor pairs with these cells as receiver.

```{r}
group_oi = "idiopathic.pulmonary.fibrosis"
receiver_oi = "CD16..Monocyte"
prioritized_tbl_oi_IPF_10 = get_top_n_lr_pairs(
  multinichenet_output$prioritization_tables, 
  10, 
  groups_oi = group_oi, 
  receivers_oi = receiver_oi)
```

```{r, fig.width=20, fig.height=20}
combined_plot = make_ligand_activity_target_plot(
  group_oi, 
  receiver_oi, 
  prioritized_tbl_oi_IPF_10,
  multinichenet_output$prioritization_tables, 
  multinichenet_output$ligand_activities_targets_DEgenes, contrast_tbl, 
  multinichenet_output$grouping_tbl, 
  multinichenet_output$celltype_info, 
  ligand_target_matrix, 
  plot_legend = FALSE)
combined_plot
```

What if there is a specific ligand you are interested in?

```{r}
group_oi = "idiopathic.pulmonary.fibrosis"
receiver_oi = "CD16..Monocyte"
ligands_oi = c("BMP1","BMP4","BMP5")
prioritized_tbl_ligands_oi = get_top_n_lr_pairs(
  multinichenet_output$prioritization_tables, 
  10000, 
  groups_oi = group_oi, 
  receivers_oi = receiver_oi
  ) %>% filter(ligand %in% ligands_oi) # ligands should still be in the output tables of course
```

```{r, fig.width=20, fig.height=20}
combined_plot = make_ligand_activity_target_plot(
  group_oi, 
  receiver_oi, 
  prioritized_tbl_ligands_oi, 
  multinichenet_output$prioritization_tables, 
  multinichenet_output$ligand_activities_targets_DEgenes, 
  contrast_tbl, 
  multinichenet_output$grouping_tbl, 
  multinichenet_output$celltype_info, 
  ligand_target_matrix, 
  plot_legend = FALSE)
combined_plot
```

### With filtering of target genes based on LR-target expression correlation

In the previous plots, target genes were shown that are predicted as target gene of ligands based on prior knowledge. However, we can use the multi-sample nature of this data to filter target genes based on expression correlation between the upstream ligand-receptor pair and the downstream target gene. We will filter out correlated ligand-receptor --> target links that both show high expression correlation (spearman or pearson correlation > 0.50 in this example) and have some prior knowledge to support their link. Note that you can only make these visualization if you ran step 7 of the core MultiNicheNet analysis.

```{r}
group_oi = "idiopathic.pulmonary.fibrosis"
receiver_oi = "CD16..Monocyte"
lr_target_prior_cor_filtered = multinichenet_output$lr_target_prior_cor %>%
  inner_join(
    multinichenet_output$ligand_activities_targets_DEgenes$ligand_activities %>% 
      distinct(ligand, target, direction_regulation, contrast)
    ) %>% 
  inner_join(contrast_tbl) %>% filter(group == group_oi, receiver == receiver_oi)

lr_target_prior_cor_filtered_up = lr_target_prior_cor_filtered %>% 
  filter(direction_regulation == "up") %>% 
  filter( (rank_of_target < top_n_target) & (pearson > 0.50 | spearman > 0.50))
lr_target_prior_cor_filtered_down = lr_target_prior_cor_filtered %>% 
  filter(direction_regulation == "down") %>% 
  filter( (rank_of_target < top_n_target) & (pearson < -0.50 | spearman < -0.50)) # downregulation -- negative correlation
lr_target_prior_cor_filtered = bind_rows(
  lr_target_prior_cor_filtered_up, 
  lr_target_prior_cor_filtered_down)
```

Now we will visualize the top correlated target genes for the LR pairs that are also in the top 50 LR pairs discriminating the groups from each other:

```{r}
prioritized_tbl_oi = get_top_n_lr_pairs(
  multinichenet_output$prioritization_tables, 
  50, 
  groups_oi = group_oi, 
  receivers_oi = receiver_oi)
```

```{r, fig.width=30, fig.height=25}
lr_target_correlation_plot = make_lr_target_correlation_plot(
  multinichenet_output$prioritization_tables, 
  prioritized_tbl_oi,  
  lr_target_prior_cor_filtered , 
  multinichenet_output$grouping_tbl, 
  multinichenet_output$celltype_info, 
  receiver_oi,
  plot_legend = FALSE)
lr_target_correlation_plot$combined_plot
```

You can also visualize the expression correlation in the following way for a selected LR pair and their targets:

```{r, fig.width=30, fig.height=6}
ligand_oi = "CXCL12"
receptor_oi = "CXCR4"
sender_oi = "Fibroblast"
receiver_oi = "CD16..Monocyte"
lr_target_scatter_plot = make_lr_target_scatter_plot(
  multinichenet_output$prioritization_tables, 
  ligand_oi, receptor_oi, sender_oi, receiver_oi, 
  multinichenet_output$celltype_info, 
  multinichenet_output$grouping_tbl, 
  lr_target_prior_cor_filtered)
lr_target_scatter_plot
```

## Intercellular regulatory network inference and visualization

In the plots before, we demonstrated that some DE genes have both expression correlation and prior knowledge support to be downstream of ligand-receptor pairs. Interestingly, some target genes can be ligands or receptors themselves. This illustrates that cells can send signals to other cells, who as a response to these signals produce signals themselves to feedback to the original sender cells, or who will effect other cell types. 

As last plot, we can generate a 'systems' view of these intercellular feedback and cascade processes than can be occuring between the different cell populations involved. In this plot, we will draw links between ligands of sender cell types their ligand/receptor-annotated target genes in receiver cell types. So links are ligand-target links (= gene regulatory links) and not ligand-receptor protein-protein interactions! We will infer this intercellular regulatory network here for the top100 interactions. You can increase this to include more hits of course (recommended). 

```{r}
prioritized_tbl_oi = get_top_n_lr_pairs(
  multinichenet_output$prioritization_tables, 
  100, 
  rank_per_group = FALSE)

lr_target_prior_cor_filtered = 
  multinichenet_output$prioritization_tables$group_prioritization_tbl$group %>% unique() %>% 
  lapply(function(group_oi){
    lr_target_prior_cor_filtered = multinichenet_output$lr_target_prior_cor %>%
      inner_join(
        multinichenet_output$ligand_activities_targets_DEgenes$ligand_activities %>%
          distinct(ligand, target, direction_regulation, contrast)
        ) %>% 
      inner_join(contrast_tbl) %>% filter(group == group_oi)
    
    lr_target_prior_cor_filtered_up = lr_target_prior_cor_filtered %>% 
      filter(direction_regulation == "up") %>% 
      filter( (rank_of_target < top_n_target) & (pearson > 0.50 | spearman > 0.50))
    
    lr_target_prior_cor_filtered_down = lr_target_prior_cor_filtered %>% 
      filter(direction_regulation == "down") %>% 
      filter( (rank_of_target < top_n_target) & (pearson < -0.50 | spearman < -0.50))
    lr_target_prior_cor_filtered = bind_rows(
      lr_target_prior_cor_filtered_up, 
      lr_target_prior_cor_filtered_down
      )
}) %>% bind_rows()

lr_target_df = lr_target_prior_cor_filtered %>% 
  distinct(group, sender, receiver, ligand, receptor, id, target, direction_regulation) 
```

```{r}
network = infer_intercellular_regulatory_network(lr_target_df, prioritized_tbl_oi)
network$links %>% head()
network$nodes %>% head()
```

```{r, fig.width=30, fig.height=12}
network_graph = visualize_network(network, colors_sender)
network_graph$plot
```

Interestingly, we can also use this network to further prioritize differential CCC interactions. Here we will assume that the most important LR interactions are the ones that are involved in this intercellular regulatory network. We can get these interactions as follows:

```{r}
network$prioritized_lr_interactions
```

```{r, fig.width=30, fig.height=12}
prioritized_tbl_oi_network = prioritized_tbl_oi %>% inner_join(
  network$prioritized_lr_interactions)
prioritized_tbl_oi_network
```

Visualize now the expression and activity of these interactions for the Tumor group
```{r}
group_oi = "idiopathic.pulmonary.fibrosis"
```

```{r, fig.height=12, fig.width=28}
prioritized_tbl_oi_IPF = prioritized_tbl_oi_network %>% filter(group == group_oi)

plot_oi = make_sample_lr_prod_activity_plots_Omnipath(
  multinichenet_output$prioritization_tables, 
  prioritized_tbl_oi_IPF %>% inner_join(lr_network_all)
  )
plot_oi
```


# Comparing the MultiNicheNet analysis with batch correction versus without

To assess how much difference the batch correction made, we will now run the MultiNicheNet analysis without batch correction with the wrapper function using the same parameters, and saving these analysis results in `multinichenet_output_noBC`
```{r}
batches =  NA

multinichenet_output_noBC = multi_nichenet_analysis(
  sce = sce, 
  celltype_id = celltype_id, sample_id = sample_id, group_id = group_id, 
  batches = batches, covariates = covariates, 
  lr_network = lr_network, ligand_target_matrix = ligand_target_matrix, 
  contrasts_oi = contrasts_oi, contrast_tbl = contrast_tbl, 
  senders_oi = senders_oi, receivers_oi = receivers_oi,
  min_cells = min_cells, 
  fraction_cutoff = fraction_cutoff, 
  min_sample_prop = min_sample_prop,
  scenario = "regular", 
  ligand_activity_down = ligand_activity_down,
  logFC_threshold = logFC_threshold, 
  p_val_threshold = p_val_threshold, 
  p_val_adj = p_val_adj, 
  empirical_pval = empirical_pval, 
  top_n_target = top_n_target, 
  n.cores = n.cores, 
  verbose = TRUE
  )
```

First we will show the top 50 interactions from the batch-corrected analysis (BC analysis) with non-corrected pseudobulk expression values

```{r}
group_oi = "idiopathic.pulmonary.fibrosis"
batches =  "dataset_origin"
```

```{r, fig.height=13, fig.width=28}
prioritized_tbl_oi_top_50 = get_top_n_lr_pairs(
  multinichenet_output$prioritization_tables, 
  50, 
  groups_oi = group_oi) # from BC-analysis

plot_oi = make_sample_lr_prod_activity_batch_plots(
  multinichenet_output_noBC$prioritization_tables, 
  prioritized_tbl_oi_top_50, 
  multinichenet_output$grouping_tbl, 
  batches)
plot_oi
```

For this dataset and these cell types, the non-corrected expression values look similar to the corrected ones.

Now we will combine the prioritization tables of both analyses. This will then later be used to define which interactions are most specific to the BC analysis compared to the non-BC analysis

```{r}
prioritized_tbl_oi_high5000_withBC = get_top_n_lr_pairs(
  multinichenet_output$prioritization_tables, 
  5000, 
  rank_per_group = FALSE
  )

prioritized_tbl_oi_high5000_noBC = get_top_n_lr_pairs(
  multinichenet_output_noBC$prioritization_tables, 
  5000, 
  rank_per_group = FALSE
  )

prioritized_tbl_oi_high5000 = prioritized_tbl_oi_high5000_withBC %>% 
  rename(score_BC = prioritization_score , rank_BC = prioritization_rank) %>% 
  full_join(
    prioritized_tbl_oi_high5000_noBC %>% 
      rename(score_noBC = prioritization_score , rank_noBC = prioritization_rank)
    )

head(prioritized_tbl_oi_high5000) # you can see some interactions NA in an analysis: this because ligand or receptor gene was filtered out

prioritized_tbl_oi_high5000 = prioritized_tbl_oi_high5000 %>% 
  mutate(
    diff_rank = rank_BC  - rank_noBC, 
    diff_score = score_BC - score_noBC) 
```

Inspecting `prioritized_tbl_oi_high5000` enables you to see how prioritization scores and ranks differ between both analyses.

```{r}
prioritized_tbl_oi_high5000 %>% View()
```

Now we will define which interactions are most specific to the BC analysis compared to the non-BC analysis

```{r}
BC_interactions = prioritized_tbl_oi_high5000 %>% 
  arrange(-diff_score) %>% 
  filter(group == group_oi) %>% 
  filter(rank_BC < 2000) %>% 
  pull(id) %>% 
  head(10)
noBC_interactions = prioritized_tbl_oi_high5000 %>%
  arrange(diff_score) %>% 
  filter(group == group_oi) %>% 
  filter(rank_noBC < 2000) %>% 
  pull(id) %>% 
  head(10)

BC_interactions_df = prioritized_tbl_oi_high5000 %>%
  arrange(-score_BC) %>% 
  filter(is.na(diff_rank)) %>% 
  filter(group == group_oi) %>% 
  head(10)

noBC_interactions_df = prioritized_tbl_oi_high5000 %>%
  arrange(-score_noBC)  %>% 
  filter(is.na(diff_rank)) %>% 
  filter(group == group_oi) %>% 
  head(10)

BC_specific_interactions_df = prioritized_tbl_oi_high5000 %>%
  arrange(-score_BC) %>% 
  filter(group == group_oi) %>% 
  filter(id %in% BC_interactions)

noBC_specific_interactions_df = prioritized_tbl_oi_high5000 %>%
  arrange(-score_noBC) %>% 
  filter(group == group_oi) %>% 
  filter(id %in% noBC_interactions)
```

### Visualization of scaled_LR_prod_activity per sample

#### BC-specific hits with BC expression values plot

```{r, fig.height=6, fig.width=28}
plot_oi = make_sample_lr_prod_activity_batch_plots(
  multinichenet_output$prioritization_tables, 
  BC_specific_interactions_df, 
  multinichenet_output$grouping_tbl, 
  batches)
plot_oi
```

#### BC-specific hits with noBC expression values plot

```{r, fig.height=6, fig.width=28}
plot_oi = make_sample_lr_prod_activity_batch_plots(
  multinichenet_output_noBC$prioritization_tables,  
  BC_specific_interactions_df, 
  multinichenet_output$grouping_tbl, 
  batches)
plot_oi
```

You can here indeed see the clear effects of the source-dataset in expression levels of some interactions (which may cloud the true DE signal that we pick up after correction).

#### noBC-specific hits with BC expression values plot

```{r, fig.height=6, fig.width=28}
plot_oi = make_sample_lr_prod_activity_batch_plots(
  multinichenet_output$prioritization_tables, 
  noBC_specific_interactions_df, 
  multinichenet_output$grouping_tbl, 
  batches)
plot_oi
```

#### noBC-specific hits with no BC expression values plots

```{r, fig.height=6, fig.width=28}
plot_oi = make_sample_lr_prod_activity_batch_plots(
  multinichenet_output_noBC$prioritization_tables, 
  noBC_specific_interactions_df, 
  multinichenet_output$grouping_tbl, 
  batches)
plot_oi
```
What you can see in these last two plots: some of these interactions are predicted as DE without batch correction because the expression is specific for the dataset of origin with the highest number of samples. This points to the need to correct for this during the analysis.

Important to note is that the interactions we visualize here were prioritized also by including ligand activity, so not only the DE of the LR pair! However, ligand activity may also be affected by batch effect correction because the underlying DE genes in receivers is affected by batch effect correction. Therefore, we will now check the effect of the dataset of origin on gene expression of some DE genes

### Check target genes!

Show now also DE genes that are different! -  this is important for the ligand activities!

```{r, fig.height=9, fig.width=27}
pval_df_targets = multinichenet_output_noBC$celltype_de %>% 
  filter(contrast == "idiopathic.pulmonary.fibrosis-normal") %>% 
  select(gene, cluster_id, p_adj, logFC) %>% 
  distinct() %>% 
  rename(p_adj_noBC = p_adj, logFC_noBC = logFC) %>% 
  inner_join(
    multinichenet_output$celltype_de %>% 
      filter(contrast == "idiopathic.pulmonary.fibrosis-normal") %>% 
      select(gene, cluster_id, p_adj, logFC) %>% 
      distinct() %>% 
      rename(p_adj_BC = p_adj, logFC_BC = logFC)
    )
## targets with more DE in noBC condition
pval_df_targets %>% 
  filter(logFC_noBC > 0) %>% 
  mutate(prop = p_adj_BC/p_adj_noBC, prop_logFC = logFC_noBC/logFC_BC) %>% 
  mutate(prop_pval_logFC = prop*prop_logFC) %>% 
  arrange(-prop_pval_logFC) %>% 
  filter(prop > 1 & p_adj_noBC < 0.05 & p_adj_BC > 0.05) 

pval_df_targets %>% 
  filter(logFC_noBC > 0) %>% 
  mutate(prop = p_adj_BC/p_adj_noBC, prop_logFC = logFC_noBC/logFC_BC) %>% 
  mutate(prop_pval_logFC = prop*prop_logFC) %>% 
  arrange(-prop_pval_logFC) %>% 
  filter(prop > 1 & p_adj_noBC < 0.05 & p_adj_BC > 0.05) %>% 
  group_by(cluster_id) %>% 
  count() %>% 
  arrange(-n)

## targets with more DE in BC condition
pval_df_targets %>% 
  filter(logFC_BC > 0) %>% 
  mutate(prop = p_adj_BC/p_adj_noBC, prop_logFC = logFC_noBC/logFC_BC) %>% 
  mutate(prop_pval_logFC = prop*prop_logFC) %>% 
  arrange(prop_pval_logFC) %>% 
  filter(prop < 1 & p_adj_noBC > 0.05 & p_adj_BC < 0.05) 
pval_df_targets %>% 
  filter(logFC_BC > 0) %>% 
  mutate(prop = p_adj_BC/p_adj_noBC, prop_logFC = logFC_noBC/logFC_BC) %>% 
  mutate(prop_pval_logFC = prop*prop_logFC) %>% 
  arrange(prop_pval_logFC) %>% 
  filter(prop < 1 & p_adj_noBC > 0.05 & p_adj_BC < 0.05) %>% 
  group_by(cluster_id) %>% 
  count() %>% 
  arrange(-n)
```

Define BC-specific DE genes in Fibroblast

```{r, fig.height=11, fig.width=17}
targets_oi = pval_df_targets %>% 
  filter(logFC_BC > 0) %>% 
  mutate(prop = p_adj_BC/p_adj_noBC, prop_logFC = logFC_noBC/logFC_BC) %>% 
  mutate(prop_pval_logFC = prop*prop_logFC) %>% 
  arrange(prop_pval_logFC) %>% 
  filter(p_adj_noBC > 0.05 & p_adj_BC < 0.05) %>% 
  filter(cluster_id == "Fibroblast") %>% 
  pull(gene) %>% head(5)
targets_oi %>% tibble(gene = .) 
```

Visualize these BC_specific_targets with corrected expression values:

```{r, fig.height=4.5, fig.width=17}
p_target = make_DEgene_dotplot_pseudobulk_batch(
  genes_oi = targets_oi, 
  celltype_info = multinichenet_output$celltype_info, 
  prioritization_tables = multinichenet_output$prioritization_tables, 
  celltype_oi = "Fibroblast", 
  batch_oi = batches, 
  grouping_tbl = multinichenet_output$grouping_tbl)
p_target$pseudobulk_plot
```

Visualize these BC_specific_targets with non-corrected expression values:

```{r, fig.height=4.5, fig.width=17}
p_target = make_DEgene_dotplot_pseudobulk_batch(
  genes_oi = targets_oi, 
  celltype_info = multinichenet_output_noBC$celltype_info, 
  prioritization_tables = multinichenet_output_noBC$prioritization_tables, 
  celltype_oi = "Fibroblast",  
  batch_oi = batches, 
  grouping_tbl = multinichenet_output$grouping_tbl)
p_target$pseudobulk_plot
```

Based on these plots, we can appreciate why they were DE more strongly after correction that before.

All together, these comparisons demonstrate that it is advisable to perform batch correction when possible.