Nb of OTus in final fasta versus tsv and a question on singletons

[natgiot]

Hello PEMA devs!
I was wondering why the number of OTUs in the finalTable.tsv file is different (higher) from the number of records in the final fasta file (Aligned_assignments.fasta). First of all, am I looking at the correct files? Both were found in the 7. directory, and I would need the fasta file to build my tree outside of pema.
My second question might be the reason why, but I would need confirmation:-) Does the remove singleton parameter in the parameters.tsv file affect both OTU and ASV pipelines? In that case, the removed singletons would be present in the table, but not in the fasta (to explain my different numbers)?
Thanks a lot for clarifications!
natassa

[hariszaf]

Hi @natgiot

Reg, the Aligned_assignments.fasta file, the OTUs that are not there, will be in the final_table.tsv file but will have a No hits assignment.

So I guess that this answers your second question as well. It's not the remove singletons parameter responsible for that, it's the CREST algorithm itself.

Hope this helps!

[natgiot]

Thank you Haris,
Is the fasta file containing all OTUs independently of their taxo status somewhere? I mean the one including the No hits. Or do I have to create it by checking in the tsv as you described? I guess I would also need the No hits for the phylogenetic tree (why not?)
Re the second question, can you still confirm that the remove_singletons parameter is also used when OTU mode is on?
Thanks,
Natassa

[hariszaf]

Is the fasta file containing all OTUs independently of their taxo status somewhere?
Yes, that is the all_sequences_grouped.fa file. There you can have your OTUs in a format like this:

>Otu1
TGGTGGAGTGATTTGTCTGGTTAATTCCGTTAACGAACGAGACCTTAACCTGCTAAATAGTTACGCGAATCTCGATTCGCGGCTAACTTCTTAGAGGGACTGTTGGTGTTCAACCAAAGTCAGGAAGGCAATAACA
>Otu2
TGGTGGAGCGATTTGTCTGGTTAATTCCGATAACGAACGAGACTCTAGCCTACTAAATAGTTCGAGGATTCTTTTACGCGTCCTCGCCAACTTCTTAGAGGGACAAGTGGCGTTTAGCCACACGAGATTGAGCAATAACA

In its current version (2.1.4) the remove_singletons parameter works for the ASV inference option only.
In case you think that would be necessary for the OTUs as well, feel free to open an issue.
It is my belief though that with a high vsearchId you should not have a lot of singletons.

[natgiot]

Thank you so much!
RE the remove_singletons, my understanding is that at a certain taxonomic level (defined by vsearchid), theree is still interest in removing the close-to-noise OTUs, which might represent low abundance OTUs; at low sequencing coverage of a community, one might not want to trust quantitations for such OTUs, so yes, I would need this parameter for OTU pipelines too. I am interested to hear other people's opinions on this, because I have been removing singleton OTUs in the past, too.
thanks!
Natassa

[natgiot]

I am confused that the number of ASVs in the extended file table (58456) is not identical to the Aligned_assignements.fasta (58677). I cannot fond in the pema bds scripts where this latter fasta is created as well, is this really an alignment file? Doesnot seem so. Based on my understanding, the fasta (and the tsv) would contain classified ASVs + the no-assigned-due-to-error ASVs ([WARNING] entrezpy.conduit.Conduit: {"empty response": {"queryid": "kkYBE6KMRRSlpNmWxmG7mA==", "action": "skip"}}. I counted the number of instances of "skip" in the output, and there are 3113. My final table contains 63538 ASVs, thus I expect that I have ~2K No-hit ASVs.
Do I count correct, why are the fasta and tsv files different by 221 ASVs and which script/process does this fasta file come from?