/
onflow.org
Flow Playground

Node Operation Quick Guide

Step 0 - Cleaning Up your Previous State

You can skip this step if it is your first time running a node on Flow.
  1. Stop your Flow node
  2. Clear the contents of your data directory that you have previously created. The default location is /var/flow/data. The data directory contains the Flow chain state.

If you had a previous node running on candidate 9, you’ll need to turn it off just before joining the updated network and re-start your node with the updated configuration.

Step 1 - Run Genesis Bootstrap

You will need to run this process **for each** node that you are operating

Download the Bootstrapping Kit

Note: if you have downloaded the bootstrapping kit previously, ensure that you do this step again to get the latest copy of the bootstrapping kit

curl -sL -O storage.googleapis.com/flow-genesis-bootstrap/boot-tools.tar
tar -xvf boot-tools.tar

Generating your Node ID

A node ID is a 32 byte string that is unique to every node in Flow. When you register your node to stake with Flow, Generate a unique ID by hashing your staking public key.

Generate Your Node Keys

Addresses

You will need to have a real address, for example, a CNAME pointed at the node or just the IP address e.g. example.com:3869 or 189.23.42.12:3869.

Do not include in http:// format.

If you are running multiple nodes, please ensure you have different addresses for each node.

All your current keys and Flow genesis files should be in the bootstrap folder created earlier. Please take a back up of the entire folder.

## Skip if this is section if this is your first time ##
# If you joined our network previously, make sure to take a backup of your previously generated keys!
cp -r /path/to/bootstrap /path/to/bootstrap.bak
#########################################################
# Generate Keys
$ mkdir ./bootstrap
# YOUR_NODE_ADDRESS: FQDN or IP address associated to your instance
# YOUR_NODE_ROLE: The Flow nodes that you wish to run, it should be ONE of the following - [ access, collection, consensus, execution, verification ]
$ ./boot-tools/bootstrap key --address \"${YOUR_NODE_ADDRESS}:3569\" --role ${YOUR_NODE_ROLE} -o ./bootstrap

Upload your Public Keys

If you're running multiple nodes, you only need one token - it can be used to generate multiple keys.

Please use the following pattern when generating your token candidate-version-yourname (you can pick any name, it just helps us keep track of the most recent keys you've sent and would like us to include in the bootstrapping process)

$ ./boot-tools/transit -push -d ./bootstrap -t ${TOKEN} -role ${YOUR_NODE_ROLE}

Running push
Generating keypair
Uploading ...
Uploaded 400 bytes

Step 2 - Getting Ready to Start your Flow Node

Before proceeding to Step 2, ensure that Step 1 was completed successfully with the bootstrap directory handy (default: /var/flow/bootstrap)

  1. Expose TCP/3569 on your firewall. If you are running Flow access node you will also need to expose TCP/9000 and optionally TCP/8000
  2. Create the data directory if you have not previously created it and/or removed a previous version of it (default: /var/flow/data)

You can choose to run your Flow Node via docker or have systemd to manage your Flow node. Make sure to choose one of these options only

Systemd

If you are running your Flow Node via Docker you can skip this step and go to the Docker section below.

  1. Ensure that you pulled the latest changes from flow-go repository via git
## Clone the repo if you haven't already done so
git clone https://github.com/onflow/flow-go

## Get latest changes
cd flow-go
git pull origin master

  1. Copy your respective systemd unit file to: /etc/systemd/system

  2. Create directory sudo mkdir /etc/flow

  3. Copy the runtime-conf.env you updated in step 3 to: /etc/flow/

  4. Enable your service sudo systemd enable YOUR-FLOW-NODE.service

Wait for the Flow team announcement before you proceed to Step 3

Step 3 - Start your Flow Node

Once you receive an announcement from the Flow team that the network had been booted (via Discord you will need to fetch the genesis info, update your runtime configuration and then boot your Flow node up!

The Flow team will provide you a new token PULL_TOKEN to pull the genesis info from (Please do NOT use the original Token that you used in Step 1) The YOUR_NODE_TYPE must be the same ones that you used on step 1 - generating your node keys

  1. Run the transit script to fetch the new genesis info: ./boot-tools/transit -pull -d ./bootstrap -t ${PULL_TOKEN} -role ${YOUR_NODE_TYPE}
  2. Pull the latest changes from flow-go repository
  3. Get your node-id, you can find it at /path/to/bootstrap/public-genesis-information/node-id
  4. Update the FLOW_GO_NODE_ID inside runtime-conf.env to the node-id that you got from the previous step
  5. Start your Flow node via docker or systemd

NOTE if you are running an execution node, you must also move the execution state from the public-genesis-information directory, to the execution-state directory with-in the bootstrap folder: mv ./public-root-information/root.checkpoint ./execution-state/ This will eventually be taken care of by the transit script but currently requires manual input.

Docker

You can skip this section if you are running via systemd

  1. Update your environment variables: source /path/to/runtime-conf.env
  2. Run the following docker command based on the Flow node that you are running:
Remember: update your docker volume mounting path

You'll need to update the path for data and bootstrap volumes in the following commands. Those paths are from step 2.2 and 2.3a.

Access

docker run --rm \
	-v /path/to/bootstrap:/bootstrap:ro \
	-v /path/to/data:/data:rw \
	--name flow-go \
	--network host \
	gcr.io/flow-container-registry/access:v0.9.3 \
	--nodeid=${FLOW_GO_NODE_ID} \
	--bootstrapdir=/bootstrap \
	--datadir=/data/protocol \
	--rpc-addr=0.0.0.0:9000 \
	--http-addr=0.0.0.0:8000 \
	--collection-ingress-port=9000 \
	--script-addr=${FLOW_NETWORK_EXECUTION_NODE} \
	--bind 0.0.0.0:3569 \
	--loglevel=error

Collection

docker run --rm \
	-v /path/to/bootstrap:/bootstrap:ro \
	-v /path/to/data:/data:rw \
	--name flow-go \
	--network host \
	gcr.io/flow-container-registry/collection:v0.9.3 \
	--nodeid=${FLOW_GO_NODE_ID} \
	--bootstrapdir=/bootstrap \
	--datadir=/data/protocol \
	--ingress-addr=0.0.0.0:9000 \
	--bind 0.0.0.0:3569 \
	--loglevel=error

Consensus

docker run --rm \
	-v /path/to/bootstrap:/bootstrap:ro \
	-v /path/to/data:/data:rw \
	--name flow-go \
	--network host \
	gcr.io/flow-container-registry/consensus:v0.9.3 \
	--nodeid=${FLOW_GO_NODE_ID} \
	--bootstrapdir=/bootstrap \
	--datadir=/data/protocol \
	--bind 0.0.0.0:3569 \
	--loglevel=error

Execution

docker run --rm \
	-v /path/to/bootstrap:/bootstrap:ro \
	-v /path/to/data:/data:rw \
	--name flow-go \
	--network host \
	gcr.io/flow-container-registry/execution:v0.9.3 \
	--nodeid=${FLOW_GO_NODE_ID} \
	--bootstrapdir=/bootstrap \
	--datadir=/data/protocol \
	--triedir=/data/execution \
	--rpc-addr=0.0.0.0:9000 \
	--bind 0.0.0.0:3569 \
	--loglevel=error

Verification

docker run --rm \
	-v /path/to/bootstrap:/bootstrap:ro \
	-v /path/to/data:/data:rw \
	--name flow-go \
	--network host \
	gcr.io/flow-container-registry/verification:v0.9.3 \
	--nodeid=${FLOW_GO_NODE_ID} \
	--bootstrapdir=/bootstrap \
	--datadir=/data/protocol \
	--bind 0.0.0.0:3569 \
	--loglevel=error

Systemd

You can skip this section if you are running via docker

  1. Check that your runtime-conf.env as at /etc/flow/runtime-conf.env
  2. Check that you have updated the variables inside the runtime-conf.env
  3. Start your service: sudo systemctl start flow

Step 4 - Verify if your Node is Running

Here are a few handy commands that you can use to check if your Flow node is up and running

Docker

  • To get Flow logs: sudo journalctl -u flow-YOUR_ROLE
  • To get the status: sudo docker ps
$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1dc5d43385b6 gcr.io/flow-container-registry/verification:candidate8 \"/bin/app --nodeid=4…\" 30 hours ago Up 30 hours flow-go

Systemd

  • To get Flow logs: sudo journalctl -u flow-YOUR_ROLE
  • To get your Flow node status: sudo systemctl status flow
● flow-verification.service - Flow Access Node running with Docker
Loaded: loaded (/etc/systemd/system/flow-verification.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2020-05-20 18:18:13 UTC; 1 day 6h ago
Process: 3207 ExecStartPre=/usr/bin/docker pull gcr.io/flow-container-registry/verification:${FLOW_GO_NODE_VERSION} (code=exited, status=0/SUCCESS)
Main PID: 3228 (docker)
Tasks: 10 (limit: 4915)
Memory: 33.0M
CGroup: /system.slice/flow-verification.service
        └─3228 /usr/bin/docker run --rm -v /var/flow/bootstrap:/bootstrap:ro -v /var/flow/data:/data:rw --rm --name flow-go --network host gcr.io/flow-container-registry/verification:candidate8 --nodeid=489f8a4513d5bd8b8b093108fec00327b683db545b37b4ea9153f61b2c0c49dc --bootstrapdir=/bootstrap --datadir=/data/protocol --alpha=1 --bind 0.0.0.0:3569 --loglevel=error

Step 5 (optional) - Monitoring and Metrics

This is intended for operators who would like to see what their Flow nodes are currently doing. Head over to Monitoring Node Health to get setup.

Node Status

The metrics for the node should be able to provide a good overview of the status of the node. If we want to get a quick snapshot of the status of the node, and if it's properly participating in the network, you can check the consensus_compliance_finalized_height or consensus_compliance_sealed_height metric, and ensure it is not zero.

curl localhost:8080/metrics | grep consensus_compliance_sealed_height

# HELP consensus_compliance_sealed_height the last sealed height
# TYPE consensus_compliance_sealed_height gauge
consensus_compliance_sealed_height 1.132054e+06

Common Issues

Error: cannot create connection

20T18:34:21Z","message":"could not create connection"}
{"level":"error","node_role":"consensus","node_id":"6d3fac8675a1df96f4bb7a27305ae531b6f4d0d2bc13a233e37bb07ab6b852dc","target":"QmVcSQaCdhmk1CMeMN7HTgGiUY1i2KqgVE2vvEmQXK4gAA","error":"failed to dial : all dials failed
  * [/ip4/155.138.151.101/tcp/3569] dial tcp4 155.138.151.101:3569: connect: connection refused","retry_attempt":2,"time":"2020-05-20T18:34:21Z","message":"could not create connection"}

This is error is OK. Your fellow node operators have not turned on/joined the network yet. So no need to worry about it!

Flow Node Not Booting Up

If your Flow node is not able to boot up, or it exits right after it boots up. You will need to do a clean up of your state.

After cleaning up the state try booting it up again. If the problem persists, message a member from the Flow team on Discord.

Edit on GitHub