An update on IRI-Playbook’s Roadmap

A about a week ago I’ve twitted about dockerizing all of IRI-playbook’s services.
This is a major step in the playbook’s evolution to become more versatile, enhance security and support additional distributions.

TL;DR

  • New “dockerized” release is being worked on
  • Will support Debian in addition to CentOS and Ubuntu
  • Support load-balancing
  • Users wanting to use the new release will have to re-install the node

Docker

Docker is a container technology that allows a user to package an application with all the parts it needs to run. Docker runs as a service on the server (the Host system) and manages containers running on the Host. To keep it simple: it can create, start, stop and restart containers. Containers are shipped as Docker images: you download (pull) an image for latest IRI and then tell docker to run it.

Applications in Docker Containers

IRI Playbook typically installs more than 10 services on a node. Just to name a few: IRI, Grafana, Prometheus, Nelson, Field and IOTA Peer Manager. You will not feel any difference when services run within containers. The advantages might not be visible to the user, but in terms of maintenance and versatility, the advantages are great.

I’ll try to briefly explain why:

Each application has its own dependencies on which it depends in order to run. For example, IRI requires Java and IOTA Peer Manager requires nodejs. These have to  be installed on the node before being able to use IRI or IOTA Peer manager.

The problem with having to install nodejs or Java on the Host system is that different applications may require different versions of Java or nodejs. That can lead to a lot of  mess on the Host.
It is way more efficient to bundle all the requirements into a single container: IRI’s Docker image already ships with right version of Java, Nelson’s Docker image with the right version of nodejs and so on. There is no conflict when running different applications in containers using a different version of nodejs, because the container filesystem is isolated from the Host. The last thing you want are sleepless nights trying to resolve package conflicts ;).

More Advantages to Docker

Here are a few more benefits to running applications inside of containers:

  • Easy to upgrade or downgrade
  • Security: being isolated from the host system, only required components are allowed access to the container. For example directories, files and network ports
  • Ability to limit CPU and Memory shares per container
  • Portability of applications

The new dockerized playbook makes it possible to support 3 major Linux distributions:
CentOS, Ubuntu and now also Debian!

What is Being Worked on at the Moment?

The new feature branch is where all the work is being done. All services have already been ported to run inside containers.

At the moment, some community members are helping test the new version (thanks to Ulairi, Mr. Andersen, uwec, dterandgo, Luca and Knobby)

Last but not least, I plan to integrate the iri-lb-haproxy project I have been working on. This will allow node operators who have multiple nodes to use one (or more) of their nodes as load balancers! (this feature might be added and fully working a little after the dockerized version is ready).

The Release Phase

The older version of the playbook will not be supported any longer. It is too time consuming to maintain and support 2 different versions of the playbook. Users will be encouraged to upgrade to the new version. This will require a complete re-install of the node.

Feel free to ping me on Twitter or IOTA’s Discord for any questions!

Thanks for the support!

If you like this project and would like to donate please use this address (and thanks!):

JFYIHZQOPCRSLKIYHTWRSIR9RZELTZKHNZFHGWXAPCQIEBNJSZFIWMSBGAPDKZZGFNTAHBLGNPRRQIZHDFNPQPPWGC

IRI Playbook – Some Facts

IOTA Reference Implementation (FullNode) Playbook

Installing and running a full-node made simple!

The iri-playbook uses the software Ansible which is widely used in professional IT environments:

Ansible is used to automate apps and IT infrastructure (Application Deployment + Configuration Management + Continuous Delivery)

A playbook is a set of instructions executed by Ansible.

  • The IRI playbook has been released in early December 2017 in an effort to help users have their full-node installed and configured in the simplest way possible.
  • It already has more than 1200+ installations to date (based on Github’s unique clone statistics)
  • It includes a “Getting Started Quickly” (one command) to get a node fully installed and configured in less than 15 minutes.
  • The IRI playbook can be used by anyone: no previous knowledge of Linux is required.

The Wiki and Repository

http://iri-playbook.readthedocs.io

https://github.com/nuriel77/iri-playbook

Overview

The iri-playbook installation includes:

  • Latest IRI configured and running
  • Can install & configure multiple nodes in parallel
  • Support for Linux distributions: CentOS and Ubuntu
  • IOTA Peer Manager GUI for node’s neighbor management
  • IOTA Prometheus Exporter for node’s metrics and Tangle statistics
  • Prometheus & Grafana to process metrics and display awesome graphs
  • Alertmanager to trigger (email) alerts based on configured rules
  • HAProxy reverse proxy for IRI’s API port for security policies and logging
  • CarrIOTA Field optional add-on to add the node to the CarrIOTA proxy/field
  • Nelson optional add-on to automatically manage node’s neighbors
  • Comprehensive Wiki including information about security hardening
  • Firewall configuration

The iri-playbook includes some useful utility scripts:

iric is a menu-driven configuration and management utility for the full-node’s services and maintenance tasks, database downloads, neighbor management and more. It makes maintenance of the node a child’s play.

nbctl is a script to help manage the node’s neighbors (add/remove/list). It is used by iric under the hood.

ps_mem a script which offers a nice overview of total memory consumption per process.

  • The IRI playbook configures Nginx as web-server to reverse proxy and set password access for several processes (e.g. IOTA Peer Manager, Prometheus etc.)
  • It provides automated SSL certificate creation and configuration (self-signed) which can later be changed to user’s supplied SSL certificate(s).
  • The project provides a fully-synced IOTA database to help kick-start new nodes.

Did you like the project?
Star the repository: https://github.com/nuriel77/iri-playbook
Or use the following IOTA address to leave a donation:

 

Security Considerations for IOTA Fullnode Operators

TL;DR

Enable firewall, change SSH default port, disable root SSH access, use SSH key authentication and disable password authentication.
In addition, disable any unnecessary commands on the API port (e.g. add/remove/getNeighbors and attachToTangle if you don’t want people using your node to perform PoW).

Head on to the bottom of the blog: I offer some links to security hardening instructions.

Intro

The IOTA network is growing very fast. Every day new nodes are being deployed. It is hard to estimate how many nodes are currently on the network, but there must be thousands at least.

For example, the last count I’ve heard of, Nelson nodes alone had more than 2000+ nodes on the network (Nelson is a software created by Roman Semko that runs on the full node and manages automatic neighbor peering). I would imagine at least 8000 non-Nelson nodes running along side those, if not more.

There are several installation guides out there meant to help the new node owner getting started. Rarely do any of those guides focus on the security aspects.

Most node owners are not tech-savvy, and simply find the easiest and fastest way to get a node onto the network, for any number of reasons: to connect their light wallet to their own node, offer others a node to which they can connect their light wallets to, help the network’s confirmations rate, etc.

Security plays a very important role. Unfortunately, this is often completely overlooked.

The Public Node

A fullnode owner can choose to make her/his node public. That means exposing the API port so that clients (e.g. wallets) can connect to perform transactions, check balances etc. It is not a requirement, as one can simply let the node be part of the network without exposing it as a public node.

The benefit of having one’s node publicly accessible is for the greater good, as it provides more options where wallet owners can connect to (taking off the load from “central” public nodes). In addition, one can choose to list their node in one of the public nodes’ listings so that they can easily be found by wallet clients.

Exposing the API port exposes the node to new types of threats. Some can easily be mitigated, and others are still unknown (just as any other system, this is a never ending task — to improve security as time progresses).

Not a Public Node

Have you chosen not to have your node publicly listed? Great! That’s fine.
However, these are no longer the early days of the IOTA’s network, when most IPs and host-names have been hidden. It is very simple to automate and collect nodes’ IPs and host-name should anyone wish to do so. You are never guaranteed that your node’s IP or host-name will not be leaked via your neighbors.

If you are using Nelson, it is rather futile to try hide your IP or host-name. Your IP will be present on multiple nodes within a short time. (For the record: I find it perfectly reasonable that node’s IP/hostname is known. If the node operator has taken some basic steps to secure her/his node, this shouldn’t be a problem).

Once you purchase and deploy your VPS, it is already exposed on the internet with a public IP. An attacker can scan your node’s ports and try to access services (SSH, webserver etc). This doesn’t necessarily have to be an attacker which is directly focused on attacking the IOTA network, but script kiddies who are trying to brute-force their way onto servers.

Just leave your SSH port on the default 22/tcp and you will notice brute-force attempts when viewing the server’s logs. These are just automated scripts trying to find weak nodes with password authentication where they can try to brute force weak passwords.

Recent Attacks on the Public Nodes

Recently, the IOTA network has been under a DDoS attack. It has been suggested that these attacks came in parallel with the stolen funds (due to the online seed generators scam). I won’t go into the exact details. If you want, you can read Ralf Rottmann’s detailed blog about those events.

I want to talk about the attack, and what happened which stalled almost all of the public nodes. The first reports of nodes being under attack appeared on IOTA’s Discord #fullnodes channel (where I spend a lot of time helping node owners). Very quickly did we realize that the nodes being attacked are public nodes and are listed on one of the fullnode listings.

I have been spending hours trying to figure what the node owners where experiencing. Unfortunately, my nodes are not public, and were thus left unaffected by the attacks. Having my nodes publicly listed would have helped me to inspect those attacks first hand.

I eventually managed to pin-point how the attacker(s) were getting nodes out-of-service: with much gratitude to the node owners who gave me access their nodes, I was able to inspect several nodes during the attack.

Discovering the Problem

Most nodes are not fitted with a reverse proxy (nginx or HAProxy) where you can decide to log the HTTP request body. That would help very quickly identifying which API calls are being made to the nodes which cause them to stall.

Luckily, our good ol’ friend tcpdump came to the rescue.
Having it dump the traffic to the API port, I was able to see the HTTP calls made to a node. I then proceeded to run those commands on my own server, trying to re-produce the error.

Most calls were not interesting: just asking for getNodeInfo and other commands which would receive a reply in no-time. Finally, I caught the problematic command:

This command stalled the node’s API port, it just kept processing. Having done “CTRL-C” didn’t seem to change anything, as the IOTA (IRI) software was still busy processing it in the background. New connections to the API port have gone into a queue and stalled as well. There was thus no need to hammer the node with a high-rate of calls, but simply run the above command, and any new connections would just get stalled.

What happens is, that those commands keep adding up, keeping the ports in CLOSE_WAIT state. After a while, you would find the node with 4000+ orphan connections. A Linux system (user’s namespace) has by default 4096 open files limit. Once it has reached this limit, it cannot do much more. Network connections add up to this. When IRI reached 4000+ open files, it stalled. Take into account those connections made to the database, for example. This attack would render your node unusable.

I contacted one of the core developers and reported to him about my finding (after having repeatedly tested this is in fact the problematic API call). The reason this command was problematic is because it was referencing a very old transaction. The developer has replied to me after a short while:

The api is stalled when calculating the weights for the random walk – this was overlooked from a DoS perspective.

This problem has been fixed within a day after having reported it.

Aftermath

The developers acted swiftly and came up with a fix quickly.

These events have prompted the community to step up the overall security, and become more aware of potential threats.

One example, is the work done to add HAProxy with security policies to run as a buffer between the API port and the clients. This is a very powerful basis on which new policies can be written and tweaked further. For example, invalid headers can be dropped, PoW (attachToTangle) can be globally rate-limited, specific regex’s rejects and so on.

Security Considerations

Once your node is out there, it is out in the wild. And you should probably consider taking some steps to make it more secure.

The iri-playbook includes some default security precautions related to the fullnode’s software and overall Linux OS. For example, running all the processes as unprivileged users, enabling restrictive firewalls and offering to run the API behind a load-balancer (HAProxy) with security policies enabled (e.g. rate-limiting).

In addition, the playbook includes an extensive chapter how to secure the system (SSH key authentication, disabling root, changing SSH default port etc.)

I encourage you to visit those links, get inspired, and take a pragmatic approach to securing your node.

Thanks for reading!

Donations

JFYIHZQOPCRSLKIYHTWRSIR9RZELTZKHNZFHGWXAPCQIEBNJSZFIWMSBGAPDKZZGFNTAHBLGNPRRQIZHDFNPQPPWGC

Configure IOTA (IRI) Full Headless Node on Centos 7

Update (Automated Installation):
I’ve created an automated installation(click-‘n-go) to install the full node and some additional add-ons (Monitoring Graphs, alerts and IOTA peer manager).
It includes extensive documentation on subjects such as security hardening and troubleshooting. I recommend to use it to get you up and running:

Automated IRI Installation

For anyone interested in running an IOTA fullnode on Centos, here are the steps I’ve taken to get it working nicely. I guess this should also work for any other RedHat based systems.
Some basic understanding of linux is required.

Note that this installation does not make use of Docker, so IRI runs directly on the system.
While there are some advantages running IRI in Docker, it is only optional.
(Sorry, still didn’t have the time to write a tutorial on installing IRI with Docker).

Install Epel

Install requirements:

Install IRI

Install maven (also libappindicator, java and git if you don’t have)

Choose a location where you want to install and run iri in.
In this example I chose /var/lib/iri but it can also be located elsewhere (like under /opt).

Note that I ran the above commands as root — because I am lazy (which is typically discouraged). You can run those commands as a privileged user prefixing the commands with ‘sudo’ where needed.

Now that iri is installed we can proceed to create a user and configure some files.

Configure

User and Home Directory

Let’s start by adding a user under which iri will run:

Now we can chown the entire home directory recursively and set correct permissions:

Startup File

Add an environmental variables file in /etc/sysconfig/iri:

Regarding IRI_VERSION, make sure you put the correct version (in time of writing this was 1.3.0). You can see the version with ls /var/lib/iri/target/iri-*.jar

You can read more about the startup options here: https://github.com/iotaledger/iri#installing

I chose to use command-line startup arguments, but you can also use a config.ini file as mentioned in the link above. You will have to provide the config file in the systemd drop-in file which is shown here below.

Neighbors List

A word about the neighbors list:

At this point you should already have at least one neighbor to add to the list. You can, at any time, add neighbors to the daemon while it is running using API commands. If you’ve added neighbors during run-time you should also add them to the IRI_NEIGHBORS list in /etc/sysconfig/iri so they are kept across restarts/reboots etc.

Systemctl

Next we can create a systemd drop-in file to control iri’s process.
We create it in /usr/lib/systemd/system/iri.service:

Next we need to reload systemctl to read the new drop-in file, enable the service and start it up:

Firewall

Depending on the ports you are going to use, you need to allow them through the firewall. By default the ports 15600/tcp and 14600/udp are used for peer communication.
I am using firewalld to control iptables. If you are using the default iptables you can search on the web how to enable the specific ports and keep the configuration persist across reboots/restarts.

Let’s say we want to allow neighbors connect to the default tcp and udp ports, we will issue:

You can also add the 14265/tcp if you chose to use the –remote startup option.
For security reasons I am keeping my 14265/tcp to listen only on localhost and if needed I do some SSH port forwarding to allow me to connect to it.

Verify

There are a few ways to verify everything is up and running.
First check the status of the daemon, you should see something like this (shoud be Active: active). Also look if any errors or if service exited for some reason.

To ensure the communication with the neighbors is working you can use tcpdump and should see something like this:

(For security reasons I’ve hidden most of the IP addresses).
You will notice bi-directional traffic, i.e. your IP address sending UDP packets to your neighbors and your neighbors sending UDP packets to your IP.

Node Monitor

I would also like to recommend this nice utility to help managing the daemon at run time:

https://github.com/akashgoswami/ipm

It allows to add/remove neighbors and visualize (live) the activity with each neighbor.

Good luck!
Please leave a comment or catch me on the iota slack channel if any questions: @nuriel77

If you liked this tutorial and would like to donate mIOTA:

JFYIHZQOPCRSLKIYHTWRSIR9RZELTZKHNZFHGWXAPCQIEBNJSZFIWMSBGAPDKZZGFNTAHBLGNPRRQIZHDFNPQPPWGC