Subscribe Now

* You will receive the latest news and updates on your favorite celebrities!

Trending News

By using our website, you agree to the use of our cookies.

Blog Post

The Need for Speed: A Toptal JavaScript Coding Challenge Retrospective
Computer science and engineering

The Need for Speed: A Toptal JavaScript Coding Challenge Retrospective 

Toptal started the JavaScript coding challenge web app as a way to attract people to our conference booths. Seeing how successful it was, we decided to make a pilot on the web, open for everyone in our community and their networks.

A screenshot of Toptal's JavaScript Speed Coding Challenge, showing one of the first questions. A function named "double" is supplied, and its body consists of a comment, "return x doubled."

At launch time, we had encouraged motivated developers to come up with creative ways to score high on the overall JavaScript coding challenge. Some of the key success factors of great independent freelancers are the ability to think outside the box and find creative ways to work within a set of constraints.

JavaScript Coding Challenge Questions

The gauntlet consisted of a number of JavaScript questions—akin to those that might be used as interview questions—ranging from really basic JavaScript challenge questions:

box.double = function double (x) {
    //return x doubled
    
    
};

…to more intermediate ones:

box.dateRank = function dateRank (x) {
    // x is a date in 2019 as string (example: "06/30/2019")
    // return the rank of the day in 2019 (i.e., "09/01/2019" translates to 244)
};

We wanted both beginners and advanced developers to have fun and invite their friends and colleagues to compete with them for the highest scores. The questions were sorted by points, so that even junior developers could score some points. The order for questions having the same amount of points was random, so the experience was slightly different every time, while the full set of questions stayed the same throughout the week.

The goal was to complete as many tasks as possible within the time limit of three minutes. In case someone found a way to complete all the tasks in the JavaScript coding challenge, 10 points would be awarded for each second that was left. We allowed multiple attempts to complete the challenge.

The first thing that we anticipated would happen is that people would take some time to solve the questions at a leisurely pace, and then copy and paste the answers into the web application.

After one hour and 40 minutes of launching the challenge, the first person followed this approach and got the 1445 maximum points that the application allowed, plus some extra points that we awarded for every second left.

A Turning Point in the JavaScript Coding Challenge

With the copy-and-paste approach, top contestants had nothing more to gain from focusing on coding the answers themselves. Instead, they turned their attention to bringing the speed to their automation skills.

The easiest approach at this point was to write some JavaScript that would solve each task while waiting on a loop until the buttons were ready, and copy-paste it into the browser console:

const solutions = {
  'double': 'return x*2',
  'numberToString': '...',
  'square': '...',
  'floatToInt': '...',
  'isEven': '...',
  'squareroot': '...',
  'removeFirstFive': '...',
  // ...
  'dateRank': '...',
  // ...
};
const get_submit_button = () => document.querySelector('.task-buttons > .col > .btn');
const solve = () => {
  const ace_editor = ace.edit(document.querySelector('.ace_editor'))
  const submission = ace_editor.getValue()
  for (const key in solutions) {
    if (submission.includes('box.' + key + ' ')) {
      ace_editor.insert(solutions[key])
      get_submit_button().click()
      setTimeout(() => {
        get_submit_button().click()
        setTimeout(() => {
          solve()
        }, 400)
      }, 900)
      return
    }
  }
}
solve()

This part could also be automated using regular automation tools like Selenium. But a faster way would be to automate use of the API, sending the solutions to the tasks:

const request = require('request');
const runTask = (data, entryId, callback) => {
  const tests = data.nextTask.tests_json;
  const results = Object.fromEntries(
    Object.entries(tests).map(([key, value]) => [key, value.result])
  );
  request.post(`https://speedcoding.toptal.com/webappApi/entry/${entryId}/attemptTask`, {
    form: {
      attempt_id: data.attemptId,
      tests_json: JSON.stringify(results),
    },
  }, (error, res, body) => {
    if (error) throw error;
    const next = JSON.parse(body).data
    if (next.isChallengeEntryFinished) {
      callback(next)
      return
    }
    runTask(next, entryId, callback)
  });
}
const runEntry = (callback) => {
  request.post('https://speedcoding.toptal.com/webappApi/entry', {
    form: {
      challengeSlug: 'toptal-speedcoding',
      email: ...,
      leaderboardName: ...,
      isConfirmedToBeContacted: ...,
      dateStop: ...
    },
  }, (error, res, body) => {
    if (error) throw error;
    const {
      data
    } = JSON.parse(body);
    const entryId = data.entry.id
    runTask(data, entryId, callback)
  });
}
runEntry(console.log)

One thing to note is that, in this version of the speedcoding challenge, the code was tested only on the client side. Because of this, it was possible to simply send out the answers to the test cases instead of the code. This allowed for an optimization and to cut some milliseconds on the client side.

After three days, the competition really began to heat up, with the number of attempts per three-hour bucket going suddenly from under 1,000 to nearly 100,000.

For three days, the scores remained the same. Some people were writing micro optimizations for their code, and several people were submitting their solutions in a loop, hoping the server would become less crowded so they could get a few points ahead. We were due for a big surprise.

Breaking the JavaScript Coding Challenge’s Maximum Score

First, let’s do some quick math: We had a total of 1445 points awarded for the completion of all tasks, and a time allowance of 180 seconds. If we award 10 points per second left in the application, the maximum theoretical achievable score would be 3245—in the case of submitting all answers instantly.

One of our users achieved a score of over 6000, which kept steadily growing over time.

How could someone get such a high score?

After a brief review, we found what had been happening. Our top contestant, a professional full-stack developer and Toptaler with more than 15 years of competitive programming experience, found a loophole in the setup of the coding challenge. He spawned multiple bots, which slowed down the server; meanwhile, he could complete the same task (the one that awarded the most points) as many times as possible and assign their scores to a single entry, continuously adding to that entry’s score.

This wasn’t against the rules, as we allowed for creative solutions; however, the particular method that he was using caused the server to be considerably busier, making network requests slower for everyone else. The first thing that we did was increase our server power, which only made him go from 56,000 to 70,000 points and stay in the first place.

While we didn’t want to intervene in the way people were interacting with the challenge, these attempts slowed down the server to the extent that the challenge was hard to use for other users, so we decided to fix the loophole.

The fix made it impossible for other people to achieve the same score during the last day of the JavaScript coding challenge. Because of this, we decided to extend the number of awards given to the top contestants. Originally, the top prize—a pair of AirPods—was supposed to go to the one top contestant only. In the end, AirPods were given to those holding the first six places.

Humble Beginnings and Ferocious Ends: Some JavaScript Coding Challenge Stats

Even our highest scorers had some difficulty starting out. In fact, of all the people who made five attempts or more, the top score for anyone’s first attempt was 645, and the median score for first attempts in that group was just 25 points.

By the end of the contest, one could guess the strategy being attempted from the total number of attempts. While some were more efficient than others, the top contestant far and away had the highest attempt count.

A combined, logarithmic bar graph showing the top 20 contestants' highest scores and total attempt counts. While the top attempt count went to the top scorer, the second-, third-, and fourth-highest attempt counts went to the 13th-, ninth-, and second-place contestants, respectively. Exactly half of the top 20, including the fourth- and sixth-place contestants, had under 1,000 attempts. Two contestants had under 100 attempts, and another one even had under 10.

Moving Forward

What does the future hold?

This was only the first JS coding challenge iteration. We want to do many more JavaScript programming challenges in the future, learn lessons from the previous runs, and make it even more exciting. The first thing we want to do is to implement attempt throttling to limit the number of submissions. We also want to expand beyond coding challenges in JavaScript, making them available in a wide range of programming languages.

Lastly, while we are taking measures to make the JavaScript coding challenge more secure, we plan to keep allowing the use of bots and other creative approaches for future challenges.


Special thanks to Pavel VydraAnton AndriievskyiTiago Chilanti, and Matei Copot for their contributions to the challenge and this article, and to @Zirak for the open-source project that formed the basis of the contest app. Likewise, thanks to everyone who participated in—and ran—the contest.

Related posts

1 Comment

  1. Katrina Shaff

    First time visiting your website, I enjoy your website!

Leave a Reply

Required fields are marked *