This is my first time using EdgeR or DeSeq2. I used HTSeq-Count to get counts for reads aligning to CDS (this is for ribosome profiling) for each gene. I obviously have a fair number of multi-aligning reads and reads that don't align to the feature (CDS). Should I remove these before running EdgeR or Deseq2?

I think you will find that edgeR and DESeq2 both remove these rows of the htseq output automatically, so you don't have to do it yourself.

But htseq isn't Bioconductor software. You might consider using the Bioconductor package Rsubread instead, which has been benchmarked as faster and more accurate. Rsubread produces a matrix of counts directly as an R object, so the issue you dealing with doesn't arise.

https://www.biorxiv.org/content/early/2018/07/26/377762