When featureCounts is used to count miRNAs present in genome aligned bam files, it gives miRNA size outputs, as well as long fragments (ie 8kb). I think this is happening because overlapping miRNAs exist, and thus featureCounts cannot distinguish where an individual miRNAs starts and ends in this situation (when there is an overlap of miRNAs in the same region, as confirmed by IGV visualization). In our protocol, we align the miRNAseq trimmed data to the genome, and we then use the same genome and its gtf format for miRNA counts. We do not use known miRNAs because we are trying to find novel miRNAs. We cannot use mirdeep2 for novel miRNAs because there are very few known miRNAs (and for some genomes there are 0 known miRNAs).

Thus I have the following questions: a. How can we change featureCounts settings to make this tool count individual miRNAs, rather than a cluster of overlapping miRNAs and label them as one fragment? b. If this is not possible, is there another tool that can be used in order to quantitate and discover all miRNAs (including novel miRNAs) from our genome aligned bam files? c. is it possible to tweak featureCounts to achieve this goal?

You seem be making some incorrect assumptions about what featureCounts does.

featureCounts simply assigns sequence reads to genes or exons defined in the GTF file, no more and no less. It treats each miRNA individually. It does not merge or cluster overlapping miRNAs and it does not do any relabelling. If overlapping miRNAs have been merged into one, then that must have been done when the GTF file was constructed. It was not done by featureCounts.

featureCounts does not return fragment sizes. It simply returns the total genomic length of exons in each feature that reads are assigned to.

I don't understand your remarks about known vs novel miRNAs. It is usual to include all potentially expressed features in the GTF file. featureCounts only uses those features that are in the GTF. It cannot discover novel miRNAs if they are not in the GTF file.