I am doing edgeR glmLRT() test because I don't have any replicates. I have three groups and I made the design as:

design <- model.matrix(~0+group+group:plant+treatment:group,data=y$samples)

I want to compare both between and within subjects. I am following section 3.5 of this edgeRUsersGuide. But I am getting the same DE genes with opposite expression for groupR:treatmentM and groupS:treatmentM.

Please help me why it happens?

The model parametrization you describe in your original post is very strange, so I'm just going to ignore it. Rather, it seems that the only possible design for edgeR is to do something like this:

Group <- rep(c("R", "S"), each=10)
Plant <- factor(rep(1:10, each=2))
Treatment <- rep(c("V", "M"), 10)
design <- model.matrix(~0 + Plant + Group:Treatment)
colnames(design) <- sub(":", "_", colnames(design)) # get rid of colon
design <- design[,-grep("TreatmentV", colnames(design))] # get to full rank

The first 10 coefficients are plant-specific blocking factors and aren't of interest for inference. The last two coefficients represent the effect of treatment (M over V) in each group. You can drop them separately to test for the effect of treatment within each group, or you can compare them to each other to identify group-specific responses to treatment:

con <- makeContrasts(GroupR_TreatmentM - GroupS_TreatmentM, levels=design)

... and feed that to glmLRT. This will test for whether the effect of treatment in the resistant group is the same as the effect of treatment in the susceptible group, which seems to be closest to what you're trying to do. On the plus side, the design I've described here has plenty of residual degrees of freedom for dispersion estimation; you actually do have replicates here.