cp team: correct copying of folders with files in *nix

This article will reveal some less obvious aspects related to the use of wildcards when copying, the ambiguous behavior of the command cp when copying, as well as methods to correctly copy a large number of files without omissions or crashes.

Let's say we need to copy everything from the folder /source to the folder /target.

The first thing that comes to mind is:

cp /source/* /target

Let's immediately correct this command to:

cp -a /source/* /target

— some characteristic of the node (for example, a number). The key is needed to identify the element of the tree corresponding to this key. Example of a binary search tree: -a will include copying all attributes, permissions, and will add recursion. When exact reproduction of permissions is not required, it is sufficient to use the key -r.

After copying, we will find that not all files were copied — files starting with a dot, like:

.profile
.local
.mc
and similar ones were ignored.

Why did this happen?

Because wildcards are processed by the shell (bash in the typical case). By default, bash ignores all files starting with dots, as it treats them as hidden. To avoid this behavior, we need to change the behavior bash using the command:

shopt -s dotglob

To preserve this behavior change after a reboot, you can create a file wildcard.sh with this command in the folder /etc/profile.d (it may be a different folder in your distribution).

And if there are no files in the source directory, then the shell will not be able to substitute anything for the asterisk, and the copying will also terminate with an error. To handle such a situation, there are the options failglob and nullglob. We will need to set failglob, which will prevent the command from executing. nullglob won't work, as it converts the string with wildcards that found no matches into an empty string (zero length), which for cp will cause an error.

However, if there are thousands of files or more in the folder, it's best to completely abandon the approach using wildcards. The thing is that bash expands wildcards into a very long command line like:

cp -a /souce/a /source/b /source/c …… /target

There is a limit to the length of the command line, which we can find out using the command:

getconf ARG_MAX

We will get the maximum command line length in bytes:

2097152

Or:

xargs --show-limits

We will get something like:

….
Maximum length of command we could actually use: 2089314
….

So, let's manage without wildcards at all.

Let's just write

cp -a /source /target

Here we encounter ambiguous behavior. cpIf the folder /target does not exist, we will get what we need.

However, if the target folder exists, the files will be copied into the /target/source folder.

We can't always delete the /target folder in advance, as it may contain necessary files, and our goal is, for example, to supplement the files in /target with files from /source.

If the source and target folders were named the same, for example, if we copied from /source to /home/source, we could use the command:

cp -a /source /home

And after copying, the files in /home/source would be supplemented with files from /source.

Here’s a logical puzzle: we can supplement files in the target directory if the folders are named the same, but if they are different, the source folder will be placed inside the target. How do we copy files from /source to /target using cp without wildcards?

To bypass this harmful limitation, we use a non-obvious solution:

cp -a /source/. /target

Those who are well-versed in DOS and Linux already understand: each folder contains 2 hidden folders '.' and '..', which are pseudo-folder links to the current and parent directories.

• When copying cp it checks for existence and attempts to create /target/.
• Such a directory exists and that is /target.
• Files from /source have been correctly copied to /target.

So, let's firmly remember this or put it on the wall:

cp -a /source/. /target

The behavior of this command is straightforward. Everything will work without errors, regardless of whether you have a million files or none at all.

Conclusions

If you need to copy all files from one folder to another, do not use wildcards; it’s better to use cp with a dot at the end of the source folder. This will copy all files, including hidden ones, and won't crash with millions of files or complete absence of files.

Afterword

vmspike suggested a similar command yielding the same result:

cp -a -T /source /target

Oz_Alex

cp -aT /source /target

WARNING: case sensitivity T matters. If you mix it up, you’ll end up with complete nonsense: the direction of copying will change.
Acknowledgments:

• Companies RUVDS.COM for support and the opportunity to publish on their blog on Habr.
• For the image TripletConcept. The image is very large and detailed, you can open it in a separate window.

P.S. Report any observed errors via private message. I increase karma for this.

Source: habr.com