Nym Fast & Furious network performance tests I, II, and III

---

Learn how Nym team and operator community are optimizing one of the world’s most powerful privacy networks

Over the past two weeks, the Nym network and core team have been conducting a series of performance tests.

Languages: 中文 // Türkçe // 日本 // Française // Español // Português // Bahasa Indonesia // Русский // українська мова

The tests entails Nym’s core team setting up clients all over the world to push high volumes of traffic through the network to learn where the network can be improved. The third test is planned for 12 UTC today (Tuesday 9th of April), and will take place live during the weekly Operator AMA call.

These performance tests are preparing this global, decentralized community of operators for what is to come: the launch of the first commercial application to run on the Nym network, NymVPN.

The aim of today’s test is to:

• Push substantial traffic through the mixnet, pinpointing and logging any inconsistent error messages and their origins.
• Engage the operator community for better ipv6 support after seeing previous tests often fail in this area.
• Reach out to individual operators on specific actions, focusing on gateways that aren’t handling traffic properly.

Before we begin today’s test, this blogpost summarises some ofthe initial learnings from the first two performance tests, which were focused on fine-tuning the testing methodology, monitoring and communication with operators.

• Are you a Nym node operator? Join the operator Matrix channel

The Nym network

Nym network is a decentralized infrastructure by independent operators, running one of the world’s most advanced privacy overlay networks: a mixnet. These independent operators provide routing and mixing services for any internet traffic, effectively making patterns of communication across the entire internet untraceable. Nym operators are coordinated and rewarded via a token economic system, incentivising the provision of excellent privacy services to applications and end users.

Decentralized networks can offer unique privacy properties. But they also come with their own set of challenges–such as ensuring consistency of service across nodes. The Fast & Furious performance tests are geared towards solving these challenges so that the network supplies a high quality of service for any application to run on it.

Why run performance tests?

Guaranteeing strong reliability, high speed and throughput across a global, decentralized network of independent operators is a challenge. And assessing the performance of such a privacy network when it is explicitly designed to prevent any centralized monitoring of traffic adds additional challenges! But this is the aim with the Fast & Furious performance tests: to solve these, one step at a time, in partnership with the Nym operator community.

Lab tests are not enough

Nym Core dev team runs regular lab tests for benchmarking and to gain insights into how the Nym network might perform. The sandbox environment provides a place for the team to for example test improvements to binaries and for the Nym ecosystem to do their own tests, integrations, tooling and more. This has given some initial estimate of the number of clients that can be sustained per core for each node. However, the real world is complex and full of unexpected issues.

The real world number of clients per core that a real world Nym node can sustain is not clear. And it is not clear how many clients per node in 2-hop VPN mode. Why? Cloud providers are not always up front when it comes to bandwidth. There might be CPU limits and Gb/s limitations by VPS providers and data center performance varies. In fact, it is unclear exactly which metrics are the most important for improving overall network performance!

This has to be tested, improved and modeled in order to understand just how many simultaneous users the Nym network can sustain while providing high performance. The aim of the Fast & Furious tests is therefore for both core team and the operator community to gain important insights, including:

1. How many mixnet clients requesting large amounts of IP traffic from hosts all over the Internet the current set of Nym gateways can handle;
2. How much sustained IP traffic a subset of mainnet nodes (entry gateways, mix nodes and exit gateways) can sustain;
3. Improvements to the mix node and gateway binaries to improve the throughput on mainnet;
4. Measurements to inform required/ optimal machine specs;
5. Test and improve the quality operations of Nym nodes;
6. For operators to discover any unexpected rate limiting and throttling by VPS providers;
7. For operators to learn how to monitor their nodes in a distributed fashion.

What have we learned so far?

First of all, we have learned that the Nym operator community is responsive, engaged and are becoming experts and organically mentoring new joiners. In other words, a real community.

A lot of new operators have joined recently via the delegations and service grants programs, facilitated by community squads. These programs are enabling good communication and coordination with the core team. So far 340 nodes (212 mix nodes, 128 gateways) are registered for the performance test sessions. One of the main aims of the performance tests is to check how the new Internet Packet Routing for exit gateways are performing. As soon as the team began pushing load through the network, there were quick reactions to learnings. 75% of nodes are now operating with IPR enabled and operators are setting up their own monitoring.

To enable insights for the performance tests, operators are running a modified binary with an endpoint that monitors packets received, sent and dropped. This endpoint allows operators to easily connect their own monitoring system. This led to an important learning for most of the community on the importance of decentralized performance monitoring. Here is why…

Decentralized networks need decentralized measurements

Among the Nym node operator community are seasoned and highly professional Web3 infrastructure operators like Nodes Guru and Blockfend as well as activists running nodes for social good. This community has built some of the most relied upon and widely used essential tooling for operators and stakers: Nodes Guru explorer, No Trust Verify dashboard and the Explore Nym rewards calculator. And most recently, for the test, one of the most active ‘Shinobi’ in the community, pawnflake, wrote up a script to help monitor node performance. This led to a broader discussion on important differences between performance monitoring for centralized and decentralized privacy systems.

The Nym performance test seeks to understand how the Nym network and binaries can be improved. For this reason, the test entails tracing packets received, sent and dropped node-by-node using Prometheus and Grafana.

This test set-up works well if each node does their own independent monitoring. However, a problem arises if the monitoring is centralized. If nodes track their packets via one single service this will effectively reveal full paths of all packet routing in one place. At that point, decentralization loses its properties as someone would then have the full overview of network communication and be able to learn all kinds of information about who is speaking to who and about what.

Instead, in true decentralized fashion, it is essential for operators to run their own local monitoring and metrics and simply report back results for further analysis. Pawnflake’s script has therefore been incorporated into guidance for independent node monitoring rather than offered as a central service.

• Visit our guides to setup metrics template and learn how to operate them in self-custodial way

How can we know whether nodes are malicious?

Now, you might wonder, what would stop nodes from simply running centralized monitoring of paths now that the new binaries have made this possible for the tests?

Firstly, for a malicious actor to be able to aggregate all the paths, they would have to convince most operators to provide their data. And after the tests are over they would need to convince operators to run a modified version of the software to be able to log paths. This is socially quite difficult to achieve given the quantity, geographical spread and diversity of Nym node operators. And technically, this will become even more difficult as Nym advisors are currently working on secure enclaves to verify cryptographically that a node is running an unmodified version of the software.

To formalize the no logs policy further, Nym Core team is preparing new operator Terms & Conditions. This will outline the operator’s legal responsibilities in terms of the software, the Nym network, users and company.

Finally, it is in a node operator’s best interest to not attempt at collective aggregate metrics and hold unnecessary information as that effectively sets them up as a target for legal action and potential subpoena.

Overall, this is a major improvement over centralized relay networks like VPNs, who have a full overview of all user traffic.

What happens next?

The first two tests have enabled operators and the team to learn how best to do a performance test on a live, decentralized mixnet. Something that has never been done before! Nodes have now upgraded to new binaries, decentralized monitoring is set up and the Nym Q/A team have all systems set up. The aim is to understand packets per second, bytes per sec and begin to get a picture of which metrics affect overall performance the most.

The Nym Q/A team has had 50–100 clients all over the world pushing load through the network, sending and fetching content images, video all over the world. Now the question remains, who will break first? Nym Q/A team or the Nym operators? 😎

The third test is about to begin! Sign up here.

Resources

• Fast & Furious registration page
• Nym performance test docs
• Decentralized performance monitoring
• Nym node operator guide

Join the Nym Community

Discord // Telegram // Element // Twitter

Privacy loves company

English // 中文 // Русский // Türkçe // Tiếng Việt // 日本 // Française // Español // Português // 한국인