Most of our work as a guild is done behind closed doors, inside dark rooms where no one can hear us, or anyone else scream. It might not be that dramatic, but what we do most of the day is something you probably never find out about. We operate the infrastructure of the WAX blockchain which mainly mean we sit in text based terminal windows with white text and black background. Pretty much what you expect the dude in the picture below would do. If we do our job, you will never notice the majority of what we do, because things simply work... Well, it's not fully dependent on us, but at least this is the goal.
As a Guild we operate producer nodes, which take all the transfers of tokens and NFTs and put inside blocks which we sign and secure. Without this process, we don't have a blockchain, so it's rather essential for all our NFT fun times that this is optimized and working well. The great thing is that it's always operational, I should probably not say always, because you never know... But so far, WAX has been fully operational since it launched, and blocks have been produced without issues. Yay, we do our job. For this job, we do get paid by the WAX blockchain, so we can maintain our infrastructure, and you can continue to trade NFTs, play games and have fun.
But producing blocks is not everything we do, we run something called API's, which I will divide into two different sections, those that provide you with data (history nodes), and those that allow you to write data (push nodes). These so called API nodes are required for all things you consider as WAX. Each DAPP, website, wallet, NFT marketplace, NFT service and so forth rely on different API nodes to communicate with the blockchain nodes. They use the history API's to fetch everything that has happened so far on the blockchain, like when you bought that NFT and later want to look at it. The website (likely atomichub) is requesting this data from an API to showcase it for you. And when you want to buy an NFT, you need to send that request to the blockchain with the amount of WAX you want to pay, for this you will use a push API.
Now you might wonder why I walk through these aspects, probably you are the only one that ever will read this, but I feel that it's important to explain this regardless. With this highly simplified explanation you can quickly understand that without these API services, WAX and NFTs as you know it, wouldn't be available. Well, at least it would not be visible on the marketplace or wallet of your choice.
At times, these API services struggle to keep up with the blockchain, or they struggle from all the bursts of requests they get. At times, when there are huge NFT releases, these API services can easily become overrun. This also happen in smaller scale by poorly designed applications and games. And the majority of our time, while working on this infrastructure, is optimizing the different bottle necks of these API's. So that hopefully your game, NFT marketplace and wallet always works.
What have we been working on over the last month(s)?
Over the last few months our WAX API's usually have an average of 400-1000 requests per second, depending on time, day and event it can also be way higher. One of the biggest games available on wax does 27 requests to our API each time someone hit that login button. Then game actions use more requests and so forth. Another game fetched about 17 requests per user that logged in, but in a loop of 30 rounds within seconds. So each time you login to play that game we got hundreds of requests. Which works perfectly fine when there are a few users playing, that all spread out nicely. But doesn't work as well when there are many games with many users going at it at the same time.
This has required us to put in a so called rate limit on our API. This is a way for us to reduce how many times you can interact with out API in a set time frame. This is highly required for a public API such as ours where we on top of the normal users, DAPPs, games and marketplaces also have reoccurring attacks, some on purpose and some ironically by mistake. You may wonder how we can be attacked by mistake? Well, basically some programs built are not well thought through, or tests that has gone wrong and their application throws requests on our API at a rapid speed without giving us enough time to respond so we get clogged.
To defend against these mistakes and malicious attacks our rate limit blocks you from accessing the API after you reached your quota. We use a so called floating window, which is a nice phrase for a timer that starts when you hit your first requests and follows your requests forward. So if our floating window is 30 seconds long, and you do 10 requests directly, then those 10 will reset after 30 seconds. We also have limits on how many open connections you can have to our API at the same time, if you exceed the amount of requests, or amount of open connections, you get rate limited (error code 429). But if you continue to request data or open more and more connections to our API regardless of this rate limit, you get hit by a ban.
The exact numbers of the rate limiting and the floating window is under testing, and the details will be published in a later post. But what has happened is that a bunch of games and tools available has been designed when WAX was less utilized, or poorly designed, and when you do basic things as login they spin up 10, or 30, or 300 requests towards our API, sometimes within milliseconds. This is then seen as malicious by our defensive system and you get hit by rate limit or even a ban.
This banning of users that simply try to play alienworlds, rplanet or use a web tool is directly caused by the shear amount of requests being sent for basic actions. Well, this is partly true because thousands of the IPs we have banned are automated programs or bots, which is those we are aiming to affect. This has resulted in a lot of support work from our team to sort out users that have wrongfully been banned for simply playing one of the WAX games. We have seen normal users that after hitting the rate limit open another 170 connections to our API through these games.
TLDR: API work
With the growth of WAX, we get a large majority of DAPP requests, while we handle this with different measurements we have also wrongfully banned users. These bans are a mixture of the users trying to spam login button, and games/dapps that are poorly designed and spam requests. some eosjs modules does multiple requests that the developers are unaware of.
We are currently tweaking and testing different levels of rate limiting for different type of calls. When we have tweaked this a bit more for the public API we will also open up for the ability for DAPPs and games to get around these rate limits for an easier on-boarding and experience on wax.
If you are wrongfully banned, or if you have questions in regards to this, feel free to reach out to Anders or Eric over at our Telegram chat.
If you want to follow our work more closely, we are reporting to the Office of Inspector generals every month. Starting this month (July) we are doing it through github, which means you can easily find and read our updates.
https://github.com/eosswedenorg/waxguilds/tree/main/reports/waxswedenorg