Guide: Lightning Routing Node #2

In the following guide I will show you how to make your Lightning node a proper routing node and how to earn some sats with your channels. Among other things, I explain how to find good peers, set my transaction fees, and secure my node. If Lightning is the future of Bitcoin, it should be beneficial to build a good routing node as early as possible to benefit from that reputation later

This guide builds on our first guide.

The topics covered in this guide:

• Risks of a routing node
  • Hotwallet
  • Viruses
  • Damage
  • Bugs / Exploits
• Basic understanding
• Routing Nodes
  • Capacity
  • Number of channels
  • Good channel partners
  • Uptime
  • Transaction fees
    • charge-lnd
    • loop
  • Rebalancing
• Security & Backup
  • SCB
  • Watchtower
  • Raid-1
  • SSD vs HDD
  • UPS
  • Webservices
  • 2FA SSH
  • SSH over Tor
  • Anchor channels
  • Circuit Breaker

Risks of a Routing Node

Running a Routing Node does not mean risk-free profits. On the contrary, you get most of the profit for the risk you take with a Lightning Node. It’s not for nothing that the motto of the Lightning network is “#reckless”. Before transferring large amounts of Bitcoin to our Node, we should be aware of these risks

Hotwallet

Because our Lightning Node must always be online, it is a ‘hotwallet’. This means that your Private Keys are on a device that is connected to the Internet 24/7. So anyone who has access to your Node has access to your Bitcoins on the Lightning Node

Viruses

Usually you use a second computer to run Node maintenance. Programs such as RTL, Thunderhub, and LiT make it extremely easy to retrieve node information such as channel balances, routing fees, and channel fees

Should an unauthorized person be able to take control of your computer, whether via a virus or because they have physical access to the computer, your bitcoins on the Lightning Node are at risk. While most programs have an access password, I would not rely on it if in doubt. The same applies to other devices on your network that have access to the ports on the Node.

Avoidance: Turn off services, use SSH only, enable 2FA for SSH, use SSH only via TOR

Damage

Physical damage to the node is also a real risk to lose funds. This can be a house fire, a theft, or a hardware failure. If not properly backed up beforehand, your sats in the Lightning Channels will no longer be reliably recoverable

Avoidance: Hosted hardware, RAID, C-Lightning DB streaming, static channel backups.

Bugs / Exploits

Even though it is relatively unlikely, bugs and exploits can still occur in the Lightning network. The network is now almost 3 years old and I am not aware of any case where a node operator has lost funds through no fault of their own — but it cannot be ruled out

In theory, there are still attack scenarios that can be exploited. (Flood and Loot, Circuitbreaker)

Avoidance: Use Uptodate Software, Circuitbreaker, etc.

Basic understanding

The basic idea of a Lightning Channel is that a one-time large payment can be split into an infinite number of small payments. However, payments in this channel cannot flow in only one direction: Both participants can pay each other and thus change the state of the channel (the balance)

If the sender and receiver nodes do not have a direct channel with each other, transactions in the Lightning network flow over several “hops” from the sender to the receiver node. In order to enable payment, there must be enough liquidity between the participants of this transaction, the “hops”, for the transaction to take place.

The role of a routing node is to act as a “hop” to route payments from one channel to another. Because this routing of payments represents a cost in terms of labor and capital to you, you may charge a transaction fee.

LN Liquidity Learning Curve
1. Yay, channels for everyone!
2. Wait, liquidity is draining out
3. I pay channel close fees?
4. What happened to my forwards?
5. I can't receive payments?
6. OK, I'll just buy inbound
7. Wow where did that inbound go?
8. I should charge routing fees

— Alex Bosworth (@alexbosworth) June 4, 2021

Only the sender of a payment knows the complete route the payment takes. The hops each see from whom they receive a payment and to whom they should pass it. The recipient only sees the last hop

Routing Nodes

But how does a payer know which hops can be used to route a payment to the recipient node? Which channel must the payment take to arrive successfully? For this purpose, nodes are evaluated according to the following criteria, among others:

• Capacity of the channels of a node
• Number of channels of a node
• Number of channels with “good nodes
• How often has the node successfully routed for me?
• Is the node often offline?
• What are the transaction fees?

The paying node then tries different routes through peers that it judges to be “good”. For example, a broadcaster will not try to send a 200 000 sat payment over a 100 000 sat channel, but may rather take a channel that has a capacity of 1 000 000 sat

Nodes that optimize the above characteristics will be selected more often to route payments

Capacity

The majority of payments that go through my routing node are over 200 000 sat in size. So if your channels are only 500 000 sat in size, you can route two payments and then already have to rebalance or close the channel. This is not only very time-consuming, but can also be expensive. Therefore, you should choose the capacity of your channel as large as possible

Depending on how much capital or liquidity you have available, you should divide your channel sizes. Large channels in the Lightning network are rare and can therefore demand particularly high fees. An example

Someone wants to make a deposit on Bitfinex over 500 000 satoshi. The average channel size on the Lightning network is 800 000 sats. In total, more than 70 000 channels are eligible for this route

With a payment of 5Mil Satoshi, only 10 000 channels are still competing. So the probability to route this payment is much higher. At 10 mil, there are even only 6 000 channels left

So we want to have as big channels as possible. Currently I have set up a minimum channel size of 500 000 for incoming channels, but I myself only open channels above ~3 Mil Satoshis

Number of channels

To be eligible for as many routes as possible, you need enough channels. The more routes a transaction can flow through your node, the more likely this will happen

As a minimum number I would recommend 8 channels. Of course also under the condition that these have a sufficient size. A routing node should have a minimum capacity of 4-5 Mil Satoshis. Finally your node grows with the number of your channels. So there are different strategies, but it is not yet clear which one is the most profitable

Good channel partners

Now to the prize question: which nodes are good channel partners? This question is incredibly difficult to answer and I can only share my own experience

Let’s imagine a payment again: It should be quite large so that we can charge high fees. Where do these types of payments go? Mostly to service providers like lnmarkets.com, Bitrefill, Bitfinex, Loop or Opennode, where customers pay for something

To these nodes, we offer our satoshis as inbound liquidity so they can receive payments. However, when a payment is made, we need both an inbound peer and an outbound peer. This means that our channels should not only consist of payment services, but also, for example, wallet providers and generally very well-connected nodes

I already explained how you can best get inbound liquidity in the first Routing Node guide

You can find well connected nodes for example on the BOS score list or at Terminal Web. When choosing your channel partners, you should also make sure that your peers do not charge too high fees.

At amboss.space you can get a good overview of the fees a node takes from other channels. However, this alone is not meaningful. It can also mean that the node does not care about its channel maintenance / is not rebalanced

To find good channel partners, it also helps to look at other routing nodes. Look for private nodes on the BOS score list or on Terminal Web. Look at which channels these nodes take high fees from. These will likely be his active outbound channels.

Ultimately, you will have to open channels and wait to see if they are routed through. Once you’ve found a channel that demands decent liquidity, keep it at all costs! For channels that rarely route, you can first try to lower the fees a bit and if that doesn’t help, you should close it

My best routing peers so far are: WalletofSatoshi, River Financial, Bitfinex (bfx-lnd0), Loop,

Is the node often online/offline?

Your routing node should be online 24h a day. Your channel partners will notice how often you are offline. If you are often offline, you will be selected as a route much less often

You can view the uptime of your channel partners as follows:

lncli listchannels | jq '.channels | map({remote_pubkey: .remote_pubkey, chan_id: .chan_id, since: (now - (.lifetime | tonumber)) | strftime("%Y-%m-%d %H:%M:%S"), uptime_pct: ((.uptime | tonumber) / (.lifetime | tonumber)) * 100)}) | sort_by(-.uptime_pct)'

Rating services like Terminal Web ping your node periodically to determine its uptime. To be considered a “stable node” on Terminal Web, you need at least 99% uptime.

What are the transaction fees?

Channel fees are always highly dependent on the channel. How badly does your channel partner need the liquidity? Are you the only peer who still has liquidity in the channel? Then that peer will pay you whatever price you ask. Also important is how much your node wants liquidity. If you are hardly networked, it will be difficult to route through your node. So always scale your channel fees with the size of your node as well

In practice, you will notice relatively quickly which peers are sucking up your liquidity. If a channel constantly appears as an outbound channel in routings, you can increase the fee rate. Channels like lnmarkets.com were pure outbound traffic for me, where I could gradually increase the fee from 100ppm up to 1000ppm

Generally I would start with fees above 300ppm for channels with service providers (bitrefill, bitfinex etc., see above). You can keep the fees for smaller nodes and e.g. wallet operators low ~100ppm at first. This is also because the channel fees have to be chosen in a way that you can rebalance the channel economically later. But more about that later

For channels that are completely empty, I use a very high fee (1000ppm+) to prevent other nodes from trying to route in that direction and the payment fails

Charge-lnd

Charge-LND is a script that can dynamically adjust your channelfees. For example, you can set channels to use a lower fee when they are filled with liquidity than when they are empty. This should make your routing node route more, at least in theory.

The whole thing is relatively complex, as individual strategies have to be created and assigned to specific nodes. However, once the whole thing is set up, you save yourself a lot of work. I don’t use the tool yet, but I will have a closer look at it soon

Loop

Story-Time: When I found out last year that Loop has its own node, I thought “wow, so many channels, I’ll route a lot”. Channel opened overnight – in the morning the entire channel was already empty. Overnight I had routed 2-3 huge transactions going to Loop. Of course only with the standard fee rate of 1ppm.

Loop is a special channel partner. The service requires extremely high inbound liquidity as they exchange offchain bitcoin to onchain and vice versa. You can charge extremely high fees (~5000ppm) for Loop, but rebalancing the channel is practically impossible. So you get 0.5% of your channel capacity as payment for a channel to loop

Once the channel is emptied, Loop closes it. So in the end you are selling inbound liquidity to Loop. Of course, this only makes sense if you have a lot of inbound liquidity

Rebalancing

I already explained the basics of rebalancing in the first routing node guide. Basically, you should constantly refill the channels that always need liquidity and for which you can charge a high fee

I came across the rebalance-lnd script by Carsten Otte. In my experience, rebalancing works a bit better with it than with BOS. First, the script directly ignores routes that have a higher fee than you chose and second, it has the Econ-Fee-Factor-Function

With an Econ-Fee-Factor you can set that the fee you pay for rebalancing has to be lower than the fee rate of your channel by a certain factor. I usually use a factor of 0.6 for the rebalancing of my commercial channels, for smaller channels a factor of 0.8. Unlike BOS, rebalance-lnd does not work with peer names but with channel IDs.

Don’t be afraid to spend a lot of money for rebalancing. A 500ppm rebalance on a 1200ppm channel is still 700ppm profit!

With -e you can block channels, so that the liquidity is not taken from these channels during a rebalancing. With this you should mark your channels that need a lot of liquidity.

To install rebalance-lnd you have to download the repository first:

git clone https://github.com/C-Otto/rebalance-lnd/

change to the directory

cd rebalance-lnd

and then install the required dependencies

pip install -r requirements.txt

In general, I have much more success rebalancing with smaller amounts than large amounts. I usually use 20 000 – 300 000 sat amounts at a time to rebalance my channels. This makes sense because, as discussed above, there are significantly more channels for smaller payments. Instead of one rebalancing transaction per channel, I then use 5-20

Security & Backup

Arguably the most important issue when running a routing node is the security of your sats. Unfortunately, there is a trade-off between security and usability: To make it as hard as possible for third parties to access my node, I have to restrict myself

You don’t have to implement all of the following suggestions

Static Channel Backups (SCB)

Static Channel Backups are the simplest form of backup. They only store the information with which partners you had which channels. They do not include how many sats in that channel currently belong to you

SCBs are updated whenever you open or close a channel. The SCB is stored on both the SD card and the HDD. You can also download them manually in RTL via the backup section or automatically via the BOS bot once the backup updates.

SCBs can only be restored in combination with your LND seed. When you restore an SCB, you are relying on the honesty of your channel partner, as they may publish an old channelstate. You can get around this reliance by using a Watchtower.

Watchtower

A watchtower is a third node that monitors with you that your channel partner does not publish an old channelstate. In practice, you send information about the current channel state to the watchtower after each transaction

The watchtower checks each Bitcoin block to make sure that your channel partner does not close the channel using a channel state with an older index and can send a so-called “justice transaction” on your behalf, which pays out the entire channel capacity to your onchain wallet

We have already explained how to connect to a watchtower in an article. However, you can also run a watchtower yourself.

Raid-1

To protect yourself from losing your data due to a physical failure of your hard drive, you can use a Raid-1. With Raid-1, your data is not only written on one hard disk, but mirrored 1:1 on two hard disks. Rudimentarily, this works via a hardware raid enclosure, but the integrity of the data is not always guaranteed

According to Openoms, Raid-1 via BTRFS and ZFS is more secure. However, a hard disk must then be supplied with power externally, since the Raspberry Pi can only supply one hard disk

SSD vs HDD

For over a year, I ran my Routing Node with an HDD. Every now and then, problems would occur that I couldn’t explain. I tried everything possible: Changed the OS, reflashed the SD card several times, compacted Channel.db etc. – LND sometimes just restarted and took hours to start again

Eventually it turned out that my HDD was getting extremely slow, so I replaced it with an SSD. Since then, my Node has had no problems at all and is lightning fast when starting LND and other services. So anyone still using an HDD should watch out for similar problems and switch to an SSD if necessary

Uninteruptable Power Supply (UPS)

Power outages are probably not a big problem for most of us. But in some areas they happen relatively often. If the power supply to a device suddenly cuts out, errors can occur – especially if data has just been written

To prevent this, it is worthwhile to purchase a UPS. This is actually just a battery that immediately steps in when the regular power supply fails. This battery can be of different sizes. In some power outages, your Internet connection may even still work as long as the router is powered

In such cases, a larger UPS can be used to power the router and Raspberry Pi for a while. Otherwise, a small UPS for the Raspberry Pi makes sense. However, one should urgently pay attention to the compatibility with the Node case.

Disable web services

GUIs of programs like RTL and Thunderhub are first accessible for everyone in the network. This is advantageous in that you can reach your routing node from any device. Unfortunately, the security measures for web services are much weaker than for example SSH

For people who are familiar with the terminal, I would recommend disabling web services, closing ports, and using the node exclusively via SSH. Admittedly, this is a bit overcautious, of course. Scripts like Suez make it possible to manage your channels without a GUI. You can install Suez as follows:

Install Poetry:

curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -

after that you have to log out once and log in again.

Clone the repository:

git clone https://github.com/prusnak/suez

then change to the directory

cd suez

and install Suez:

poetry install

You can get the channel overview of Suez with the command:

poetry run ./suez

You can then deactivate the web services via the Raspiblitz menu.

Add 2FA to SSH

To protect your SSH login, you can enable two-factor authentication:

sudo apt update

sudo apt install libpam-google-authenticator

google-authenticator

You will be asked if you want the authentication token to be time based, answer“y“.

Now scan the QR code with your authenticator app and note the “Emergency Scratch Codes” at the bottom of the page.

Next, you will be asked if you want to update the authentication file. Answer that with“y” as well.

The next question is if we want to prevent 2 people from logging in with the same token at the same time. We confirm this with“y”

The script then asks if we want to allow a time tolerance. This can mean that codes are valid longer than 60s and two codes are valid at the same time. We deny this with“n”

Last we are asked if we want to enable rate limiting. This means that we get a timeout of 30 seconds after 3 failed input attempts. We confirm this with “y”

To use the 2FA module for SSH login, we first open the SSH configuration file:

sudo nano /etc/pam.d/sshd

and add the following line at the end of it:

auth required pam_google_authenticator.so

Save with CTRL+S and close the document with CTRL+X

Then we activate the challenge-response system of the SSH protocol:

sudo nano /etc/ssh/sshd_config

There we replace

ChallengeResponseAuthentication no

with

ChallengeResponseAuthentication yes

Finally we restart the SSH service:

sudo systemctl restart sshd

From now on, you will be asked for your 2FA token every time you log in to SSH.

SSH via Tor

To access your node from outside your network, you should use SSH over Tor. You can find a good tutorial on the Twenty-One Youtube channel:

Anchor channels

Anchor channels are an improved version of regular Lightning channels. They reduce the risk of a “flood & loot” attack. If the option is enabled, you can use CPFP (Child-Pays-For-Parent) to speed up the closing transaction for channel force closures if needed. This makes the commitment fee much smaller, which is why your channel size ends up being slightly larger

You can enable the option by opening the LND configuration file:

sudo nano /mnt/hdd/lnd/lnd.conf

and

protocol.anchors=true

to the file. Then restart LND:

sudo systemctl restart lnd

If the option is enabled, new channels with other peers that also support it will automatically become anchor channels.

Circuit Breaker

To reduce the risk of a griefing attack, Joost Jager (formerly of Lightning Labs) has developed Circuit Breaker. In a griefing attack, an attacker floods one of your channels with HTLCs that cannot be resolved. This renders the channel useless and makes channel closing too expensive. Circuit Breaker limits the number of active HTLCs in your channels

You can install Circuit Breaker from the Raspiblitz menu in the “Settings” submenu. Select Circuitbreaker in the settings menu with the space bar and confirm with Enter.

after the installation Circuit Breaker will be shown in the main menu.

Circuit-Breaker uses default values from the beginning. So you don’t have to change them. If you want to change them anyway, you can do so with:

sudo su circuitbreaker

to switch to the circuitbreaker user

First we duplicate the example file:

cd
cd circuitbreaker
cp circuitbreaker-example.yaml circuitbreaker.yaml

and then we can change our settings via:

nano circuitbreaker.yaml

to change them.

Final words

I hope this guide helps you stack some sats with routing and support the Lightning network. Don’t rely too much on evaluation criteria, like the BOS score. Alex Bosworth himself says that it is not really meaningful

Don’t expect DeFi-type gains, either. Lightning is not designed to make you rich. With a little practice, one routing node is enough for your daily coffee-to-go. For more, you need to be really active at the moment and have a deeper understanding of the Lightning network to find profitable routes that others may not have found yet

Last but not least, don’t be put off. Some days our node doesn’t route anything at all. Especially with new channels it sometimes takes a while until you route something. If a channel still hasn’t routed anything after a long time, close it and find a new peer

If you want to connect to our routing node, you can find it here: 03f51df0183b2083d678d867d7441ba7e8dbf1bfdd23729d702b81a8b128e3e876@uklxntmppdznkgd2r5tamnkly54ncenb53hk5yj6agq5r7vk73xmwaad.onion:9735

(Don’t be discouraged by our 1000ppm default fee, this drops to 100-200ppm once liquidity is on our side