Ethereum Execution Clients Memory Comparison in Endurance

Introduction

A image from the community piqued my interest.

One of the Validators on our Endurance network is running a set of Clients, including Beacon, Execution, and Validator Nodes, on a NAS with just 4GB of RAM.

Moreover, the actual memory usage doesn’t even reach 1GB (738+181+25=944 MB), as seen in the image.

This is significantly below our expectations. Initially, to simplify deployment complexity, we did not tailor the official Validator server hardware configuration of Endurance2.0 to the memory size required for each client combination but instead uniformly opted for 32GB of RAM. Our internal tests showed that, in the worst-case scenario, one of our client combinations could consume over 20GB of RAM to complete initialization¹.

The need for such a significant amount of RAM for initialization owes to our massive Genesis file — in short, most Ethereum Execution Layer (EL) clients are not optimized for “huge Genesis files”. This isn’t a problem with the client teams! After all, who would have thought that a network’s Genesis.json file could nearly reach 1GB².

Performance Comparison

Disclaimer

Although we are not the core developers of any of the clients mentioned, we deeply value the trust and cooperation built over the years among the Ethereum client teams. The purpose of this article is not to privately judge “which client is better” but simply to:

• Highlight the performance differences between different clients under A. a specific version, and B. under some extreme conditions.
• Recommend client selection strategies to Solo Stakers with “potato machines”.
• Highlight the issues our development team wants to prioritize in the coming months.

Disclaimer 2

We made every effort to ensure the reliability of our test results, such as repeating each test at least three times. However, inaccuracies or ambiguities may still exist, and we welcome corrections if found.

Test Environment

This comparison is solely between EL clients regarding memory consumption. We did not include the memory usage of CL clients because their memory footprint is normal.

Hardware configuration:

cloud: GCP

instance: n2d-highmem-2 (AMD EPYC, 2vCPU, 16GB memory)

disk: SSD persistent disk

operating system: ubuntu 22.04

client version:

genesis file: approximately 1.1 GB³

We deliberately did not apply for a server with 32GB of RAM because we wanted to set the baseline at 16GB, with the shortfall made up through swap space. Our optimization goal is that 16GB can handle the most memory-intensive combination.

Initialization Start-up:

This refers to the start-up time on a completely clean system for the first time.

In this phase, reth clearly has an advantage with the lowest memory usage and startup time, using only 3.02 GB of RAM, significantly outperforming the runner-up.

Besu performed the worst, with memory usage similar to Geth but an initialization time of up to 100 minutes.

Regular Start-up:

This refers to the daily repeated restart tests after the completion of the initial start-up.

In this phase, the memory requirements generally decrease for everyone because most clients do not process the genesis file in its entirety during daily startup, or even avoid reading the genesis file altogether.

Geth and erigon, by completely skipping the genesis file read, emerge as winners, with clear advantages in both memory and time.

The testing for besu might not have been “fair” because its regular startup performs an additional check on the current block height, and if the height is 0, it goes through the initialization process again. This results in its results looking very similar to the initialization startup. This was discovered later, but by then, the test environment had already been destroyed. However, we have added another test at the end of the article.

Conclusion

If you are a developer with 32GB of RAM like us, then any client combination would suffice.

If you have only 8GB of RAM as a home staker and do not wish to complete client initialization on another Linux computer, we recommend reth for its speed.

For those seeking the ultimate in low memory usage (< 2GB), like the scenario depicted at the beginning of this article, we suggest using geth/erigon. However, you’ll need to prepare another computer to complete the client initialization and then transfer the folder to the server with less memory for startup.

Postscript

Regarding the high memory configuration requirement demonstrated in the besu test, future sole stakers might opt for other lighter clients, a direction in client diversity we’d prefer not to see.

Therefore, my colleague @lyfsn and I attempted some optimizations within a month after the successful mainnet upgrade, hoping to merge them as a series of general improvements into besu’s upstream branch.

In our internal tests (including a submitted PR and another not yet submitted), besu’s regular startup time has been optimized to a reasonable level.

On the developer’s computer, the startup time improved from 2min53s to 0min9s, with the Total Memory Allocated reduced from 125.47GB to 1.31GB. This represents a x20 speed increase and a x100 decrease in memory allocation.

If applied to the above test environment, both the startup time and peak memory would be highly competitive.