Can someone please say, what the problem is?

error is shown at the bottom... Is it because I havenot enough RAM?

> bam <- system.file("extdata", "ENCFF653KEU.bam", package="nucleR")
> reads <- readBAM(bam, type="paired")

reading file /usr/local/lib/R/site-library/nucleR/extdata/ENCFF653KEU.bam
processing flags
processing strand +
processing strand -
> head(reads)

GRanges object with 6 ranges and 0 metadata columns:
      seqnames      ranges strand
         <Rle>   <IRanges>  <Rle>
  [1]     chr1 10019-10124      *
  [2]     chr1 10022-10129      *
  [3]     chr1 10025-10137      *
  [4]     chr1 10028-10129      *
  [5]     chr1 10028-10142      *
  [6]     chr1 10035-10148      *
  -------
  seqinfo: 25 sequences from an unspecified genome; no seqlengths

> cover_data <- coverage.rpm(reads)
> head(cover_data)

RleList of length 6
$chr1
numeric-Rle of length 249240340 with 4577529 runs
  Lengths:              10018                  3 ...                 45
  Values :                  0 0.0363107942550514 ... 0.0363107942550514

$chr2
numeric-Rle of length 243188437 with 4320187 runs
  Lengths:              10511                 97 ...                167
  Values :                  0 0.0726215885101028 ... 0.0363107942550514

$chr3
numeric-Rle of length 197928009 with 3582096 runs
  Lengths:              60070                216 ...                103
  Values :                  0 0.0363107942550514 ... 0.0363107942550514

$chr4
numeric-Rle of length 191044273 with 2993584 runs
  Lengths:              10119                 11 ...                 38
  Values :                  0 0.0363107942550514 ... 0.0363107942550514

$chr5
numeric-Rle of length 180904852 with 3030234 runs
  Lengths:              11482                  8 ...                 12
  Values :                  0 0.0363107942550514 ... 0.0363107942550514

...
<1 more element>
> cover_data

RleList of length 25
$chr1
numeric-Rle of length 249240340 with 4577529 runs
  Lengths:              10018                  3 ...                 45
  Values :                  0 0.0363107942550514 ... 0.0363107942550514

$chr2
numeric-Rle of length 243188437 with 4320187 runs
  Lengths:              10511                 97 ...                167
  Values :                  0 0.0726215885101028 ... 0.0363107942550514

$chr3
numeric-Rle of length 197928009 with 3582096 runs
  Lengths:              60070                216 ...                103
  Values :                  0 0.0363107942550514 ... 0.0363107942550514

$chr4
numeric-Rle of length 191044273 with 2993584 runs
  Lengths:              10119                 11 ...                 38
  Values :                  0 0.0363107942550514 ... 0.0363107942550514

$chr5
numeric-Rle of length 180904852 with 3030234 runs
  Lengths:              11482                  8 ...                 12
  Values :                  0 0.0363107942550514 ... 0.0363107942550514

...
<20 more elements>
> reads

GRanges object with 27540020 ranges and 0 metadata columns:
             seqnames      ranges strand
                <Rle>   <IRanges>  <Rle>
         [1]     chr1 10019-10124      *
         [2]     chr1 10022-10129      *
         [3]     chr1 10025-10137      *
         [4]     chr1 10028-10129      *
         [5]     chr1 10028-10142      *
         ...      ...         ...    ...
  [27540016]     chrM 16457-16570      *
  [27540017]     chrM 16460-16563      *
  [27540018]     chrM 16460-16569      *
  [27540019]     chrM 16461-16571      *
  [27540020]     chrM 16467-16568      *
  -------
  seqinfo: 25 sequences from an unspecified genome; no seqlengths

> cover_clean <- filterFFT(cover_data, pcKeepComp=0.02, showPowerSpec=TRUE)

Error: cannot allocate vector of size 878.2 Mb

Yes, that error says you don't have enough RAM. And that's a lot of RAM to need! I know next to nothing about nucleR, but presumably you can do things on smaller subsets of the genome rather than all in one go? If not, you might consider renting a big EC2 instance from Amazon.