In reference with Vignette, workflow and DESeq2 paper ,

                              rlog = log2(qij) =βi0+βij

where qij is a parameter proportional to the expected true concentration of fragments for gene i and sample j (see formula below), βi0 is an intercept which does not undergo shrinkage, and βij is the sample-specific effect which is shrunk toward zero based on the dispersion-mean trend over the entire dataset. The trend typically captures high dispersions for low counts, and therefore these genes exhibit higher shrinkage from the rlog.

can you please explain me how to calculate rlog with example.

xxxxxx C1 C2 C3 C4 C5 C6 T1 T2 T3 T4 T5 T6 geneA 929 1040 1048 582 887 1219 1078 777 998 952 602 776 geneB 236 457 306 153 284 327 196 184 299 149 89 112 geneC 4207 6238 5795 3240 4308 5783 3605 3323 6071 3311 2017 2365 geneD 772 1010 1087 632 962 963 534 430 754 477 292 396 geneE 1546 2966 1916 977 1625 1622 930 1073 1840 609 409 467 geneF 376 601 532 323 403 537 324 350 494 299 193 269 geneG 348 430 511 321 380 601 402 347 543 448 278 310 geneH 12574 15679 15938 22606 14236 26517 13365 10283 77330 24915 6187 13376

thanks in adv :)

You cannot calculate rlog with a simple formula (unlike the VST, which is one appealing aspect of the VST and why I tend to prefer the VST in our documentation and on the forum).

rlog is a sum of the posterior mode coefficients from a GLM with Normal prior distributions on non-intercept coefficients. There isn't a closed form solution.