Unable to run specific line of mixOmics code on R Studio server

[yaaminiv]

I am trying to run the following line of code: https://github.com/sr320/ceabigr/blob/main/code/exon-methylation-expression-mixomics.Rmd#L263

The code starts running so I close my R Studio server tab, but when I return to the R Studio server the next day, it looks like R was terminated. There's no new object in my environment and my last run command (the code I was trying to run) doesn't even show up in my console.

I'm now trying to run the same command but with less repeats (and hopefully less processing power) to see if I have the same issue:

exonPerf.plsFem <- perf(exonResult.plsFem, validation = "Mfold", folds = 5,
                     progressBar = FALSE, nrepeat = 5) #Use perf for repeated k-fold cross-validation to determine the number of components that should be included

Any other suggestions for trying to run this line of code?

[kubu4]

I'm having a difficult time finding that perf() function. Do you know if it's part of a package (if yes, which one) or a base R function?

[kubu4]

Looking at system logs (/var/log/syslog) from your earlier attempt, looks like two things are happening:

1. The computer is running out of memory.
2. This output path is not found: ../output/exon-methylation-expression-mixomics/. But, I'd be willing to guess that this error is related to the lack of memory.

We'll see how your current run goes, but It might be time to move this to Mox...

Apr  5 07:21:17 raven kernel: [2141941.032999] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/rstudio-server.service,task=rsession,pid=16955,uid=1009
Apr  5 07:21:17 raven kernel: [2141941.033222] Out of memory: Killed process 16955 (rsession) total-vm:470654744kB, anon-rss:255786688kB, file-rss:0kB, shmem-rss:0kB, UID:1009 pgtables:699092kB oom_score_adj:0
Apr  5 07:21:27 raven kernel: [2141951.949206] oom_reaper: reaped process 16955 (rsession), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Apr  5 07:21:34 raven rsession-yaaminiv[17362]: ERROR session hadabend; LOGGED FROM: rstudio::core::Error {anonymous}::rInit(const rstudio::r::session::RInitInfo&) src/cpp/session/SessionMain.cpp:675
Apr  5 07:23:08 raven kernel: [2142053.433270] TCP: request_sock_TCP: Possible SYN flooding on port 4813. Sending cookies.  Check SNMP counters.
Apr  5 07:24:49 raven rsession-yaaminiv[17362]: ERROR system error 2 (No such file or directory) [path: /home/shared/8TB_HDD_02/yaaminiv/ceabigr/code, path: ../output/exon-methylation-expression-mixomics/]; OCCURRED AT rstudio::core::Error rstudio::core::FilePath::completeChildPath(const string&, rstudio::core::FilePath&) const src/cpp/shared_core/FilePath.cpp:738
Apr  5 07:25:18 raven rsession-yaaminiv[17362]: ERROR system error 2 (No such file or directory) [path: /home/shared/8TB_HDD_02/yaaminiv/ceabigr/code, path: ../output/exon-methylation-expression-mixomics/]; OCCURRED AT rstudio::core::Error rstudio::core::FilePath::completeChildPath(const string&, rstudio::core::FilePath&) const src/cpp/shared_core/FilePath.cpp:738
Apr  5 07:25:19 raven dbus-daemon[2025]: [system] Activating via systemd: service name='org.freedesktop.timedate1' unit='dbus-org.freedesktop.timedate1.service' requested by ':1.1530' (uid=1009 pid=17542 comm="timedatectl " label="unconfined")
Apr  5 07:25:21 raven systemd[1]: Starting Time & Date Service...
Apr  5 07:25:21 raven dbus-daemon[2025]: [system] Successfully activated service 'org.freedesktop.timedate1'
Apr  5 07:25:21 raven systemd[1]: Started Time & Date Service.
Apr  5 07:25:40 raven rsession-yaaminiv[17362]: ERROR system error 2 (No such file or directory) [path: /home/shared/8TB_HDD_02/yaaminiv/ceabigr/code, path: ../output/exon-methylation-expression-mixomics/]; OCCURRED AT rstudio::core::Error rstudio::core::FilePath::completeChildPath(const string&, rstudio::core::FilePath&) const src/cpp/shared_core/FilePath.cpp:738```

[kubu4]

I think the job is about to crash... Just saw a major memory spike and all CPU threads just dropped...

[yaaminiv]

welp I'll move over to mox then. Is there R Studio server on mox still running?

[kubu4]

Might take a little while. I'm watching the memory swap file slowly creeping up (the red |s underneath the long line of green |s):

When that swap file is full, it will crash.

[kubu4]

That was that. Job was just killed 7mins ago due to memory constraints.

[kubu4]

welp I'll move over to mox then.

There is another option we could try before moving over to Mox, if we wanted.

That option is to increase the size of the swap file. We can probably increase it by another 50 - 100GB (I'd have to check available disk space before committing to an actual number). It could make the difference, but it's hard to guess what the peak memory required for the job would be.

BTW, the swap file is literally a file on the SSD that is utilized for extra memory when the RAM becomes fully utilized. So, we can increase the swap file to any size we want, but it's utilizing disk space. So, if the swap file gets too large, then it causes a bunch of errors related to available disk space

Is there R Studio server on mox still running?

Technically, no; but yes, it's still there. R Studio Server on Mox is only running when someone starts it (it's a Singularity image/container) as part of a SLURM job. Here's the R Studio on Mox guide if you need it.

[sr320]

So, we can increase the swap file to any size we want, but it's utilizing disk space.

Can we just plug in a new SSD and use it solely for swap?

[kubu4]

Yep, that'd be an option.

[yaaminiv]

There is another option we could try before moving over to Mox, if we wanted.

Happy to do whatever make the most sense. I've run the R Studio server on mox previously so I can certainly do it again. Is there a way to estimate how much processing power/memory a job would need?

[kubu4]

Is there a way to estimate how much processing power/memory a job would need?

Don't think so, at least not quickly/easily. You'd probably have to run a subset of smaller jobs to get an estimate of how much memory is require per some other unit (lines of code, DMLs, expressed genes?).

[kubu4]

Can we just plug in a new SSD and use it solely for swap?

The only thing with this is the concern that the external SSD might get mapped to a different "location" on restart. I'm fairly certain there's a way to set it up to map to the same location every time, but it'll take some sleuthing/refreshing on how to do this.