Unexpected error while saving file in JupyterLab on ARE (Disk quota exceeded)

Technical
,
1

I’m getting

File Save Error for timeseries-Copy1.ipynb
Unexpected error while saving file:
g/data/v45/aek156/notebooks/github/aekiss/access-eval-recipes/ocean/timeseries-Copy1.ipynb
[Errno 122] Disk quota exceeded

In JupyterLab on ARE, but there’s plenty of room in g/data/v45 and in my home directory. Any suggestions? I guess I’m missing something obvious.

2

It looks like is trying to save it to ol01 project

3

Hi @Paola-CMS where did you see ol01?

I’m running on ol01 and I’ve included scratch/ol01+gdata/ol01 in the storage option.

4

I should have given you more details, sorry I was in a hurry, I just listed (ls -lt) the directory, and you can see that the file is there but with group ol01

5

OK thanks. That group is because I’m submitting under ol01.

I can duplicate files via the Jupyter file browser, or create them with the launcher, but when I save via the notebook menu or icon it creates zero-byte files with no content.

I wonder if File->Save Notebook first creates the empty file, then tries to write an auxiliary file (e.g. a temporary backup or log or something) somewhere that it can’t access, so it fails before proceeding to write the file’s contents.

6

Is it possible that ol01 doesn’t have any allocation on the disk that v45 is on? I mean /g/data3 ? As it’s trying to write on gdata3 as ol01 group.
More recent gdata disks don’t allow to write with a different group, but /g/data3 is an older one.

7

I don’t think that’s it, since I can create and duplicate non-empty files via the launcher and file browser.

8

I can even write files to /g/data/v45 from a jupyter script, but I can’t save the script itself.

9

Is it possible the jobfs is full ? In the past this has just crashed the kernel for me though (because some process was trying to spill memory to disk and failed).

10

hm, how would I check that?

11

Once the session is finished you can see in the Debug Log from the My Interactive Sessions log. While its going, I would guess you can tell using the message.log file after you click on the Session ID: link from the My Interactive Sessions log or through a terminal using qstat -f.

12

Thanks, but doesn’t look like that’s it

   JobFS requested:    100.0MB                JobFS used: 0B
13

Solved with the help of Ben Menadue (NCI). As @Paola-CMS suggested, this is because the files are being written as group ol01, but since ol01 resides on gdata6, it has only minimal disk allocation to /g/data3/v45.

The problem was fixed by moving to the parent directory and doing

find . -type d -exec chmod -c g+s {} +

which sets the setgid bit in all subdirectories, so that files will inherit the group of the parent directory . (v45 in this case) rather than the group I’m running as (ol01 in this case).

1 Like