🥇IPFS without pain (but this is not accurate)

Despite the fact that Habré was already more than one article about IPFS.

I will clarify right away that I am not an expert in this area, but I have shown interest in this technology more than once, but trying to play around with it often caused some pain. Today I started experimenting again and got some results that I would like to share. In short, the IPFS installation process and some features will be described (everything was done on ubuntu, I have not tried it on other platforms).

If you missed what IPFS is, it is written in some detail here: habr.com/en/post/314768

Installation

For the purity of the experiment, I suggest immediately installing it on some external server, since we will consider some pitfalls with working in local mode and remote. Then, if desired, it will not be demolished for a long time, there is not much.

Install go

Official documentation
See the current version at golang.org/dl

Note: it is better to install IPFS on behalf of the user who is supposed to use it most often. The fact is that below we will consider the option of mounting via FUSE and there are subtleties.

cd ~
curl -O https://dl.google.com/go/go1.12.9.linux-amd64.tar.gz
tar xvf go1.12.9.linux-amd64.tar.gz
sudo chown -R root:root ./go
sudo mv go /usr/local
rm go1.12.9.linux-amd64.tar.gz

Then you need to update the environment (more details here: golang.org/doc/code.html#GOPATH).

echo 'export GOPATH=$HOME/work' >> ~/.bashrc
echo 'export PATH=$PATH:/usr/local/go/bin:$GOPATH/bin' >> ~/.bashrc
source ~/.bashrc

Checking that go is installed

go version

Install IPFS

I liked the installation method the most ipfs update.

Install it with the command

go get -v -u github.com/ipfs/ipfs-update

After that, you can run the following commands:

ipfs-update versions - to see all available versions for download.
ipfs-update version - to see the currently installed version (until we have IPFS installed, it will be none).
ipfs-update install latest - install the latest version of IPFS. Instead of latest, respectively, you can specify any desired version from the list of available ones.

Installing ipfs

ipfs-update install latest

Checking

ipfs --version

Directly with the installation in general terms everything.

Start IPFS

Initialization

First you need to perform initialization.

ipfs init

In response, you will receive something like this:

 ipfs init
initializing IPFS node at /home/USERNAME/.ipfs
generating 2048-bit RSA keypair...done
peer identity: QmeCWX1DD7HnXXXXXXXXXXXXXXXXXXXXXXXXxxx
to get started, enter:
	ipfs cat /ipfs/QmS4ustL54uo8FzR9455qaxZwuMiUhyvMcX9Ba8nUH4uVv/readme

You can run the suggested command

ipfs cat /ipfs/QmS4ustL54uo8FzR9455qaxZwuMiUhyvMcX9Ba8nUH4uVv/readme

Experience the Power of Effective Results

Hello and Welcome to IPFS!

██╗██████╗ ███████╗███████╗
██║██╔══██╗██╔════╝██╔════╝
██║██████╔╝█████╗  ███████╗
██║██╔═══╝ ██╔══╝  ╚════██║
██║██║     ██║     ███████║
╚═╝╚═╝     ╚═╝     ╚══════╝

If you're seeing this, you have successfully installed
IPFS and are now interfacing with the ipfs merkledag!

 -------------------------------------------------------
| Warning:                                              |
|   This is alpha software. Use at your own discretion! |
|   Much is missing or lacking polish. There are bugs.  |
|   Not yet secure. Read the security notes for more.   |
 -------------------------------------------------------

Check out some of the other files in this directory:

  ./about
  ./help
  ./quick-start     <-- usage examples
  ./readme          <-- this file
  ./security-notes

Here, in my opinion, the interesting begins. The guys at the installation stage are already starting to use their own technologies. The proposed hash QmS4ustL54uo8FzR9455qaxZwuMiUhyvMcX9Ba8nUH4uVv is not generated specifically for you, but sewn into the release. That is, before the release, they prepared a welcome text, poured it into IPFS and added the address to the installer. I think it's very cool. And this file (more precisely, the entire folder) can now be viewed not only locally, but also on the official gateway ipfs.io/ipfs/QmS4ustL54uo8FzR9455qaxZwuMiUhyvMcX9Ba8nUH4uVv. At the same time, you can be sure that the contents of the folder have not changed in any way, because if it had changed, then the hash would also have changed.

By the way, in this case, IPFS has some similarities with the version control server. If you make changes to the source files of the folder and again pour the folder into IPFS, then it will receive a new address. At the same time, the old folder will not go anywhere just like that and will be available at its previous address.

Direct launch

ipfs daemon

You should receive a response like this:

ipfs daemon
Initializing daemon...
go-ipfs version: 0.4.22-
Repo version: 7
System version: amd64/linux
Golang version: go1.12.7
Swarm listening on /ip4/x.x.x.x/tcp/4001
Swarm listening on /ip4/127.0.0.1/tcp/4001
Swarm listening on /ip6/::1/tcp/4001
Swarm listening on /p2p-circuit
Swarm announcing /ip4/127.0.0.1/tcp/4001
Swarm announcing /ip6/::1/tcp/4001
API server listening on /ip4/127.0.0.1/tcp/5001
WebUI: http://127.0.0.1:5001/webui
Gateway (readonly) server listening on /ip4/127.0.0.1/tcp/8080
Daemon is ready

Opening the doors to the Internet

Pay attention to these two lines:

WebUI: http://127.0.0.1:5001/webui
Gateway (readonly) server listening on /ip4/127.0.0.1/tcp/8080

Now, if you have installed IPFS locally, then you will access IPFS interfaces using local addresses and everything will be available to you (For example, localhost:5001/webui/). But when installed on an external server, by default, the gateways are closed to the Internet. Gateways two:

webui admin (github) on port 5001.
External API on port 8080 (readonly).

So far, both ports (5001 and 8080) can be opened for experiments, but on a combat server, of course, port 5001 should be closed with a firewall. There is also port 4001, which is needed so that other peers can find you. It should be left open to outside requests.

Open ~/.ipfs/config for editing and find these lines in it:

"Addresses": {
  "Swarm": [
    "/ip4/0.0.0.0/tcp/4001",
    "/ip6/::/tcp/4001"
  ],
  "Announce": [],
  "NoAnnounce": [],
  "API": "/ip4/127.0.0.1/tcp/5001",
  "Gateway": "/ip4/127.0.0.1/tcp/8080"
}

Change 127.0.0.1 to the ip of your server and save the file, then restart ipfs (stop the running command with Ctrl+C and start it again).

Should get

...
WebUI: http://ip_вашего_сервера:5001/webui
Gateway (readonly) server listening on /ip4/ip_вашего_сервера/tcp/8080

Now the external interfaces should be available.

Check

http://домен_или_ip_сервера:8080/ipfs/QmS4ustL54uo8FzR9455qaxZwuMiUhyvMcX9Ba8nUH4uVv/readme

The above readme file should open.

http://домен_или_ip_сервера:5001/webui/

The web interface should open.

If webui works for you, then the IPFS settings can be changed directly in it, including viewing statistics, but below I will consider configuration options directly through the config file, which is generally not critical. It's just better to remember exactly where the config is and what to do with it, otherwise if the web face does not work, it will be more difficult.

Setting up a web interface to work with your server

Here is the first pitfall, which took about three hours.

If you installed IPFS on an external server, but did not install or run IPFS locally, then when you go to /webui in the web interface, you should see a connection error:

The fact is that webui, in my opinion, works very ambiguously. First, it tries to connect to the API of the server where the interface is open (based on the address in the browser, of course). and if it doesn't work there, it tries to connect to the local gateway. And if you have IPFS running locally, then webui will work fine for you, only you will work with local IPFS, and not external, although you opened webui on an external server. Then you upload the files, but for some reason you don’t see them just like that on an external server…

And if it is not running locally, then we get a connection error. In our case, the error is most likely due to CORS, which is also indicated by webui, suggesting adding a config.

ipfs config --json API.HTTPHeaders.Access-Control-Allow-Origin '["http://ip_вашего сервера:5001", "http://127.0.0.1:5001", "https://webui.ipfs.io"]'
ipfs config --json API.HTTPHeaders.Access-Control-Allow-Methods '["PUT", "GET", "POST"]'

I just registered a wildcard

ipfs config --json API.HTTPHeaders.Access-Control-Allow-Origin '["*"]'

The added headers can be found in the same ~/.ipfs/config. In my case it is

  "API": {
    "HTTPHeaders": {
      "Access-Control-Allow-Origin": [
        "*"
      ]
    }
  },

We restart ipfs and we see that webui has successfully connected (in any case, it should, if you opened the gateways for requests from outside, as described above).

Now you can upload folders and files directly through the web interface, as well as create your own folders.

Mounting the FUSE file system

Here's a pretty interesting feature.

Files (as well as folders), we can add not only through the web interface, but also directly in the terminal, for example

ipfs add test -r
added QmfYuz2gegRZNkDUDVLNa5DXzKmxxxxxxxxxx test/test.txt
added QmbnzgRVAP4fL814h5mQttyqk1aURxxxxxxxxxxxx test

The last hash is the hash of the root folder.

Using this hash, we can open a folder on any ipfs node (which can find our node and get the contents), we can in the web interface on port 5001 or 8080, or we can locally via ipfs.

ipfs ls QmbnzgRVAP4fL814h5mQttyqk1aUxxxxxxxxxxxxx
QmfYuz2gegRZNkDUDVLNa5DXzKmKVxxxxxxxxxxxxxx 10 test.txt

But you can still open it like a regular folder.

Let's create two folders at the root and grant rights to them to our user.

sudo mkdir /ipfs /ipns
sudo chown USERNAME /ipfs /ipns

and restart ipfs with --mount flag

ipfs daemon --mount

You can create folders in other places and specify the path to them through the ipfs daemon parameters -mount -mount-ipfs /ipfs_path -mount-ipns /ipns_path

Now reading from this folder is somewhat unusual.

ls -la /ipfs
ls: reading directory '/ipfs': Operation not permitted
total 0

That is, there is no direct access to the root of this folder. But you can get the content, knowing the hash.

ls -la /ipfs/QmbnzgRVAP4fL814h5mQttyqxxxxxxxxxxxxxxxxx
total 0
-r--r--r-- 1 root root 10 Aug 31 07:03 test.txt

cat /ipfs/QmbnzgRVAP4fL814h5mQttyqxxxxxxxxxxxxxxxxx/test.txt 
test
test

At the same time, even auto-completion works inside the folder when the path is specified.

As I said above, there are subtleties with such mounting: by default, mounted FUSE folders are available only to the current user (even root will not be able to read from such a folder, not to mention other users in the system). If you want to make these folders available to other users, then in the config you need to change "FuseAllowOther": false to "FuseAllowOther": true. But that's not all. If you run IPFS as root, then everything is OK. And if on behalf of a regular user (even sudo), then you will get an error

mount helper error: fusermount: option allow_other only allowed if 'user_allow_other' is set in /etc/fuse.conf

In this case, you need to edit /etc/fuse.conf by uncommenting the #user_allow_other line.

After that, restart ipfs.

Known issues with FUSE

The problem has been noticed more than once that after restarting ipfs with mounting (and maybe in other cases), the /ipfs and /ipns mount points become unavailable. There is no access to them, and ls -la /ipfs shows ???? in the list of rights.

Found this solution:

fusermount -z -u /ipfs
fusermount -z -u /ipns

Then restart ipfs.

Adding a Service

Of course, running in the terminal is only suitable for initial tests. In combat mode, the daemon should start automatically at system startup.

On behalf of sudo, create the file /etc/systemd/system/ipfs.service and write to it:

[Unit]
Description=IPFS Daemon
After=syslog.target network.target remote-fs.target nss-lookup.target

[Service]
Type=simple
ExecStart=/home/USERNAME/work/bin/ipfs daemon --mount
User=USERNAME
Restart=always

[Install]
WantedBy=multi-user.target

USERNAME, of course, must be replaced with your user (and perhaps the full path to the ipfs program will be different for you (you must specify the full path)).

We activate the service.

sudo systemctl enable ipfs.service

We start the service.

sudo service ipfs start

Checking the status of the service.

sudo service ipfs status

For the purity of the experiment, it will be possible to reboot the server in the future to check that ipfs starts successfully automatically.

Adding known to us feasts

Consider a situation where we have IPFS nodes installed both on an external server and locally. On an external server, we add some file and try to get it via IPFS locally by CID. What will happen? Of course, the local server most likely does not know anything about our external server and will simply try to find the file by CID by “asking” all IPFS peers available to it (with which it has already managed to “get acquainted”). Those in turn will ask others. And so on, until the file is found. Actually, the same thing happens when we try to get the file through the official gateway ipfs.io. If you're lucky, the file will be found in a few seconds. And if not, it will not be found even in a few minutes, which greatly affects the comfort of work. But we know where this file will first appear. So why don't we immediately tell our local server "Search there first"? Apparently, this can be done.

1. We go to the remote server and look in the ~/.ipfs/config config

"Identity": {
    "PeerID": "QmeCWX1DD7HnPSuMHZSh6tFuxxxxxxxxxxxxxxxx",

2. Run sudo service ipfs status and look for Swarm entries in it, for example:

Swarm announcing /ip4/ip_вашего_сервера/tcp/4001

3. We add from this the general address of the form "/ip4/ip_your_server/tcp/4001/ipfs/$PeerID".

4. For reliability, we will try to add this address to peers through our local webui.

5. If everything is OK, open the local config ~ / .ipfs / config, find “Bootstrap” in it: [...
and add the received address first to the array.

Restart IPFS.

Now let's add the file to the external server and try to request it on the local one. Should fly fast.

But this functionality is not yet stable. As far as I understand, even if we specify the address of a peer in Bootstrap, ipfs changes the list of active connections with peers during operation. In any case, the discussion of this and wishes regarding the possibility of specifying permanent feasts is underway here and it seems like is expected add some functionality to [email protected]+

The list of current peers can be viewed both in the webui and in the terminal.

ipfs swarm peers

And here and there you can add your feast manually.

ipfs swarm connect "/ip4/ip_вашего_сервера/tcp/4001/ipfs/$PeerID"

Until this functionality has been improved, you can write a tool to check for a connection to the desired peer and, if not, to add a connection.

Reasoning

Among those who are already familiar with IPFS, there are both arguments for and against IPFS. Basically, yesterday discussion and prompted me to dig into IPFS again. And with regards to the discussion mentioned above: I can’t say that I strongly oppose any argument of those who spoke (I disagree only with the fact that one and a half programmers use IPFS). In general, both are right in their own way (especially comment about checks makes you think). But if we discard the moral and legal assessment, who will give a technical assessment of this technology? Personally, I have some kind of inner feeling that "this must be done unequivocally, it has certain prospects." But why exactly, there is no clear formulation. Like, if you look at the existing centralized tools, then in many respects they are far ahead (stability, speed, manageability, etc.). Nevertheless, I have one thought that seems to make sense and which can hardly be implemented without such decentralized systems. Of course, I'm swinging too hard, but I would formulate it this way: the principle of disseminating information on the Internet must be changed.

Let me explain. If you think about it, now we have information distributed according to the principle “I hope that the one to whom I gave it will protect it and it will not be lost or received by those to whom it was not intended.” As an example, it is easy to consider various mail services, cloud storages, etc. And what do we end up with? On Habré hub Information Security is on the first line and almost every day we receive news about another global leak. In principle, all the most interesting things are listed in <irony> wonderful article Summer is almost over. There is almost no unleaked data left. That is, the main Internet giants are becoming larger, they are accumulating more and more information, and such leaks are a kind of information atomic explosions. This has never happened before, and here it is again. At the same time, although many understand that there are risks, they will continue to trust their data to third-party companies. Firstly, there is not much alternative, and secondly, they promise that they have patched up all the holes and this will never happen again.

What option do I see? It seems to me that data should initially be distributed openly. But openness in this case does not mean that everything should be easy to read. I'm talking about the openness of storage and distribution, but not total openness in reading. I assume that information should be distributed with public keys. After all, the principle of public / private keys is already old, almost like the Internet. If the information is not confidential and is intended for a wide circle, then it is laid out immediately with a public key (but still in encrypted form, just anyone can decrypt it with the available key). And if not, then it is laid out without a public key, and the key itself is transferred to what should have access to this information. At the same time, the one who should read it should only have a key, and where to get this information, he should not really soar - he just pulls it from the network (this is the new principle of distribution by content, not by address).

Thus, for a mass attack, attackers will need to obtain a huge number of private keys, and this is unlikely to be done in one place. This task, as I see it, is more difficult than hacking a particular service.

And here another problem is closed: confirmation of authorship. Now on the Internet you can find many quotes written by our friends. But where is the guarantee that it was they who wrote them? Now, if each such record was accompanied by a digital signature, it would be much easier. And it doesn’t matter where this information lies, the main thing is the signature, which, of course, is difficult to forge.

And here's what's interesting here: IPFS already carries encryption tools (after all, it is built on blockchain technology). The private key is immediately specified in the config.

  "Identity": {
    "PeerID": "QmeCWX1DD7HnPSuMHZSh6tFuMxxxxxxxxxxxxxx",
    "PrivKey": "CAASqAkwggSkAgEAAoIBAQClZedVmj8JkPvT92sGrNIQmofVF3ne8xSWZIGqkm+t9IHNN+/NDI51jA0MRzpBviM3o/c/Nuz30wo95vWToNyWzJlyAISXnUHxnVhvpeJAbaeggQRcFxO9ujO9DH61aqgN1m+JoEplHjtc4KS5
pUEDqamve+xAJO8BWt/LgeRKA70JN4hlsRSghRqNFFwjeuBkT1kB6tZsG3YmvAXJ0o2uye+y+7LMS7jKpwJNJBiFAa/Kuyu3W6PrdOe7SqrXfjOLHQ0uX1oYfcqFIKQsBNj/Fb+GJMiciJUZaAjgHoaZrrf2b/Eii3z0i+QIVG7OypXT3Z9JUS60
KKLfjtJ0nVLjAgMBAAECggEAZqSR5sbdffNSxN2TtsXDa3hq+WwjPp/908M10QQleH/3mcKv98FmGz65zjfZyHjV5C7GPp24e6elgHr3RhGbM55vT5dQscJu7SGng0of2bnzQCEw8nGD18dZWmYJsE4rUsMT3wXxhUU4s8/Zijgq27oLyxKNr9T7
2gxqPCI06VTfMiCL1wBBUP1wHdFmD/YLJwOjV/sVzbsl9HxqzgzlDtfMn/bJodcURFI1sf1e6WO+MyTc3.................

I am not a security specialist and cannot know exactly how to use it correctly, but it seems to me that these keys are used at the level of exchange between IPFS nodes. And also js-ipfs and example projects such as orbit-dbon which it works orbit.chat. That is, theoretically, each device (mobile and not only) can be easily equipped with its own encryption-decryption machines. In this case, it remains only for everyone to take care of saving their private keys, and everyone will be responsible for their own security, and not be a hostage of another human factor on some super-popular Internet giant.

Only registered users can participate in the survey. Sign in, you are welcome.

Have you heard about IPFS before?

I have never heard of IPFS, but it seems interesting
Haven't heard and don't want to hear
Heard but not interested
Heard, but did not understand, but now it seems interesting
I have been actively using IPFS for a long time.

69 users voted. 13 users abstained.

Source: habr.com

IPFS without pain (but this is not accurate)