joblibやscikit-learnでOSError: [Errno 28] No space left on device


OSError: [Errno 28] No space left on device



Some scikit-learn estimators and utilities can parallelize costly operations using multiple CPU cores, thanks to the following components:

via the joblib library. In this case the number of threads or processes can be controlled with the n_jobs parameter.

via OpenMP, used in C or Cython code.

8.3. Parallelism, resource management, and configuration
Parallelism: Some scikit-learn estimators and utilities parallelize costly operations using multiple CPU cores. Depending on the type of estimator and sometimes...

joblib.Parallelのドキュメントを見てみると、shared memoryに関連する引数を発見。

temp_folder: str, optional

Folder to be used by the pool for memmapping large arrays for sharing memory with worker processes. If None, this will try in order:

a folder pointed by the JOBLIB_TEMP_FOLDER environment variable,

/dev/shm if the folder exists and is writable: this is a RAM disk filesystem available by default on modern Linux distributions,

the default system temporary folder that can be overridden with TMP, TMPDIR or TEMP environment variables, typically /tmp under Unix operating systems.

Only active when backend=”loky” or “multiprocessing”.

joblib.Parallel — joblib 1.4.dev0 documentation



import os
os.environ['JOBLIB_TEMP_FOLDER'] = '/tmp'


"No space left on device" error while fitting Sklearn model
I'm fitting a LDA model with lots of data using scikit-learn. Relevant code piece looks like this: lda = LatentDirichletAllocation(n_topics = n_topics, max_iter...
OSError: [Errno 28] No space left on device · Issue #127 · nalepae/pandarallel
Hi, Hi, How can I prevent the space error bellow ? Traceback (most recent call last): File "/anaconda/lib/python3.6/multiprocessing/", line 119, in worke...