Today we want to talk about some of the latest updates to the Sherlock system [this is a high-performance cluster of Stanford University - approx. per.], which greatly speed up the listing of files in directories with a large number of entries.
Unlike regular articles, this is more of an insider's report on how Sherlock is regularly worked on to keep it the best it can be for our users. We hope to publish more articles like this in the future.
Listing many files takes time
It all started with a question to technical support from the user. He reported a problem that doing ls takes several minutes in a directory with more than 15 entries in $SCRATCH [directory for temporary files - approx. per.].
Thousands of files in a single directory are usually hard on the file system and are definitely not recommended. The user knew this and admitted that it was not good, but mentioned that the listing was 1000 times faster on his laptop than in Sherlock. Of course, it hurt us. So we looked deeper.
Because ls looks nice
We've looked at what it actually does. ls when listing a directory, and why the process takes so long. Most modern distributions ls default is done as ls --color=autobecause everyone loves colors.
But beautiful colors come at a cost: for every file ls should get information about the file type, its permissions, flags, extended attributes and the like in order to select the appropriate color.
One simple solution to the problem is to turn off the color in ls altogether, but imagine the outrage of users. In no case should you take a colored output, we are not monsters.
So we looked deeper. ls colorizes entries via environment variable LS_COLORS, which sets dircolors(1) based on config file dir_colors(5). Yes, (and if you don't know about files (do) ββthen dir_colors , Despite everything).
Let's figure it out in more detail
To determine which color scheme is causing the slowdown, we created an experimental environment:
$ mkdir $SCRATCH/dont
$ touch $SCRATCH/dont/{1..10000} # don't try this at home!
$ time ls --color=always $SCRATCH/dont | wc -l
10000
real 0m12.758s
user 0m0.104s
sys 0m0.699s12,7 seconds for 10 files, not very good.
By the way, I need a flag
--color=always: although it is drawn tols --color=autoButlsdetects when it is not connected to a terminal (for example, by channel or with issuance redirection) and disables coloring if set toauto. Clever guy.
So what is taking so long? We looked with strace:
$ strace -c ls --color=always $SCRATCH/dont | wc -l
10000
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
44.21 0.186617 19 10000 lstat
42.60 0.179807 18 10000 10000 getxattr
12.19 0.051438 5 10000 capget
0.71 0.003002 38 80 getdents
0.07 0.000305 10 30 mmap
0.05 0.000217 12 18 mprotect
0.03 0.000135 14 10 read
0.03 0.000123 11 11 open
0.02 0.000082 6 14 close
[...]
Wow: 10 calls lstat(), 10 calls getxattr() (which all fail because our environment doesn't have the attributes that ls looks for), 10 calls capget().
Surely this can be optimized.
The capabilities attribute? nope
Following the advice , we tried to disable attribute validation :
$ eval $(dircolors -b | sed s/ca=[^:]*:/ca=:/)
$ time strace -c ls --color=always $SCRATCH/dont | wc -l
10000
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
98.95 0.423443 42 10000 lstat
0.78 0.003353 42 80 getdents
0.04 0.000188 10 18 mprotect
0.04 0.000181 6 30 mmap
0.02 0.000085 9 10 read
0.02 0.000084 28 3 mremap
0.02 0.000077 7 11 open
0.02 0.000066 5 14 close
[...]
------ ----------- ----------- --------- --------- ----------------
100.00 0.427920 10221 6 total
real 0m8.160s
user 0m0.115s
sys 0m0.961s
Wow, acceleration up to 8 seconds! We got rid of all those expensive calls getxattr(), and calls capget() also disappeared, great.
But there are still these annoying challenges lstat(), Althoughβ¦
How many flowers do you need?
Therefore, we took a closer look LS_COLORS.
First, just disable this variable:
$ echo $LS_COLORS rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36: $ unset LS_COLORS $ echo $LS_COLORS $ time ls --color=always $SCRATCH/dont | wc -l 10000 real 0m13.037s user 0m0.077s sys 0m1.092s
What!?! Still 13 seconds?
It turns out when the environment variable LS_COLORS not defined or missing only one of its elements <type>=color:, it uses the built-in database by default and uses colors anyway. So if you want to disable colorization for a particular file type, you need to override it with <type>=: or <type> 00 in file DIR_COLORS.
After a lot of trial and error, we narrowed the search down to this:
EXEC 00
SETUID 00
SETGID 00
CAPABILITY 00which is written as
LS_COLORS='ex=00:su=00:sg=00:ca=00:'
This means: do not colorize files by attribution , but bit by bit , nor by .
We speed up ls
And if you do not do any of these checks, then calls lstat() disappear, and now it's a completely different matter:
$ export LS_COLORS='ex=00:su=00:sg=00:ca=00:'
$ time strace -c ls --color=always $SCRATCH/dont | wc -l
10000
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
63.02 0.002865 36 80 getdents
8.10 0.000368 12 30 mmap
5.72 0.000260 14 18 mprotect
3.72 0.000169 15 11 open
2.79 0.000127 13 10 read
[...]
------ ----------- ----------- --------- --------- ----------------
100.00 0.004546 221 6 total
real 0m0.337s
user 0m0.032s
sys 0m0.029s0,3 seconds on a list of 10 files, a record.
Setting up Sherlock
From 13 seconds with default settings to 0,3 seconds with a little tweaking LS_COLORS means a 40-fold acceleration due to the absence of setuid / setgid and colored executable files. Not such a big loss.
Of course, this is now configured in Sherlock on a per-user basis.
But if you want to return the coloring, you can simply return to the default settings:
$ unset LS_COLORS
But then, on directories with a large number of files, be sure to brew coffee while it works ls.
Source: habr.com