SMUMods
  • Reviews
  • Books
  • Analytics
  • Blog
Behind-The-Scenes: Building SMUMods Analytics
a month ago – February 8, 2021
This article was originally published by Renata Dharma as a Medium article and is reposted by SMUMods with permission. All credit goes to the original author.

For anyone who’s just started venturing into the data analytics realm, one might ask:

What’s data analytics all about?

Simply put, Data Analytics transforms data into visualizations where you can observe trends and eventually, extract actionable insights. Its potential to empower your business decision is powerful and recognized by many companies.

So what’s it like to perform data analytics? Two students from Singapore Management University, Cindy and Jia Cheng embarked on a quest with SMUMods, challenging themselves to produce useful visualizations that aid undergraduates with their bidding decisions.

Who’s the BOSS? 👔

In context, BOSS refers to our university’s unique platform, an abbreviation for “Bidding Online SyStem”. It pools hundreds of modules together and opens multiple bidding windows for undergraduates to bid for their semester’s modules. Prices of each module begin at e$10 (e-credits) and can be highly sensitive to factors like course popularity, section timing, professors, past bid prices, and more.

Notably, it’s also the platform that gets undergraduates stressed out over just how much to bid — the two front-end developers of SMUMods can definitely relate.

“My despair over BOSS’s tacky interface and timing out every 2 minutes inspired me to create a better-looking version,” said Cindy, cheekily. She is currently a year 3 undergraduate, pursuing Software Development and Financial Technology tracks in the School of Information Systems (SIS). “The co-founder of SMUMods, Gabriel, had asked me what feature I would like to implement and this came to my mind.”

Jia Cheng, also a year 3 undergraduate at SIS pursuing the Software Development track, mentioned, “I met Gabriel at Product Circle in 2019/20 Term 1 and talked about the whole interface of BOSS — Long story short, he introduced Cindy to me to work together on this feature, so here we are.”

“Our ultimate goal is to provide an alternative to the BOSS Overall Results page where students can view bidding prices and past trends on the go!”

With that, the two set off to build the analytics feature from scratch.


Developing the Analytics Feature 🔨

Analytics1

A draft of the Analytics feature in its premature, vanilla JavaScript stages

But first – will people use it? 🤔

Picking this idea up from late last year, the two developers set out to understand the opinions of this potential analytics feature, to further validate the problem, and to find out whether this problem is experienced by the rest of the SMU community as well. Jia Cheng added, “We would be more motivated to build this feature if we knew it served a wide segment of the SMU community that has been facing this problem too.”

“With the guidance of Zachery and Gabriel, we carefully curated a survey that gathered responses on students’ perceptions of our potential feature and their “bidding habits” to find out how we could build a product that would suit their needs”, recounted Cindy.

Analytics2

A sneak-peek at the survey results: nearly 140 out of over 180 students relied on past bidding results of more than a semester to determine their bidding prices.

“Thankfully, many of the students who responded expressed their interest and saw value in the proposed analytics feature. Jia Cheng and I were encouraged by the response and went ahead to transform our ideas into reality.”

Over the summer of 2020, the pair began working toward the next step.

Ready, Set, Collect 🔍

Data collection was quite simple: with a click of a button on BOSS, all the past bidding data can be downloaded immediately.

But the real challenge comes when all of the downloaded data have been combined…

$Unc1ean datA, and the issues they bring

Analytics3

Behind the Scenes — sample JSON data used for the feature

Ever opened up a file, expecting neat rows of texts and numbers, all in the columns they should be in? For data miners, it’s a rare sight.

Data cleaning is a stage that often follows after data collection. The main goal here is to detect corrupted record entries and replace or modify the dirty data. Along the way, the pair came across a few issues.

Jia Cheng had initially written a script to convert the past BOSS bidding results (released here) from Excel files into comma-separated values (CSV) files and then into JavaScript Object Notation (JSON) format, first using JavaScript before moving to Python which he was more comfortable with.

The pair soon found the challenges of dirty data emerging:

  • Several modules had commas in their names, which made it difficult to split the data where commas are used as delimiters

  • The files downloaded from BOSS contained replication of professor names, and older datasets comprised of columns with maximum bids — a feature BOSS has removed ever since. For this, the compromise of excluding maximum bids for the entire feature had to be made.

  • BOSS had included in their datasets rows of data for rounds that were not biddable or had no bids. This meant that the datasets contained rows with meaningless data. Thankfully, these rows were easy to remove as they could be identified with bid amounts equating to 0 — this highlights the importance of being contextually aware as all modules unanimously required a minimum of e$10.

  • BOSS has many rounds of bidding — including “Incoming Round 1C” and “Incoming Freshmen Round” — messing up the alphabetical order of the rounds required for visualization, as these were rounds that typically occur after “Round 1A”, “Round 1B” and so on.

In hindsight, Jia Cheng felt that the data cleaning process would have been faster if he had used Pandas (a Python built-in library used widely in data analysis) instead of searching each error and fixing it.

Visualization in Vue 📊

The pair used a JavaScript framework called Vue to construct the future. Being new to the framework did not stop them from attempting, as they first ventured into vanilla JavaScript before progressing further.

Analytics4

”When we were still figuring everything out”

Of course, the visualization wasn’t done in just one try — it took multiple views and alterations to achieve the feature’s final look.

Analytics5

A premature draft the two did on Figma

In a series of trial and errors, Cindy noted a few lessons from visualizing data:

  • As seen in the first draft, the numeral bidding prices were in both a bar chart and a line chart. Admittedly, it did not make much sense for similar data types (one representing the minimum bid price and the other representing median bid price) to be represented by two different types of visualizations. Moreover, discrete data like the number of vacancies were not well suited for line charts in this context — instead, bar charts are generally better at representing discrete data.

  • The first draft also did not include the number of vacancies, represented by ‘Before Proc’ and ‘After Proc’ in the datasets. These columns refer to the number of available vacancies before the current round and the number of vacancies after the current round respectively. After much discussion, Cindy and Jia Cheng decided to include and display the trends of the number of vacancies before and after each round, as these statistics contribute significantly towards students’ bidding decisions.

Analytics6

Another attempt to visualize

“At first, we had difficulties using Vue due to the lack of familiarity with this framework. It was only through thorough research and experimentation, whilst cross-checking with other developers on the team — whom we are so thankful for — that both of us were able to overcome the technical challenges,” Cindy recounted. “(Vue) is a lightweight but highly capable framework that I’m keen on exploring in future projects.”

Exploring and Enhancing UI/UX 💻💯

To improve user experience, Jia Cheng mentioned a couple of things they brought to the project:

  • An auto-complete feature, as well as pairing the course codes and course titles together under the search field. As a user myself, I found this extremely helpful whenever I could not remember the full module name or only recalled the course code. This way, I was able to search for past results in a few seconds!

  • Integrating elements from Buefy. Buefy is a lightweight library of responsive UI components for Vue.js, and this contributed to the minimalistic look that we see live on their website.

Going once, going twice… and sold! 💰

Then came the big release day — 24 June 2020, the feature went live and hundreds of students flocked to the website.

The two received positive feedback on the feature and they were delighted to have been able to help the student cohort right before bidding began.

Analytics7

There were so many users that I too felt the website lag — you can show your support by donating to SMUMods so that they can afford larger servers to host the website more smoothly.


A Growth-Oriented Journey 🌱

Analytics8

From this...

Analytics9

...to this...

Analytics10

…and to this end product!

👨‍💻 For Cindy, it was pertinent to possess self-sufficiency — reading up on the documentation opened them up to the answers they were looking for. Additionally, she believes that this experience taught her to see data visualization beyond just pie charts; there were key considerations to be made, such as why one would present a certain type of data in such a manner — numbers with line graphs, and so on — which made her more strategic, and practical with how she would visualize her data.

👨‍💻 On the other hand, Jia Cheng shared his point of growth: Through this journey, he finds himself firmly believing in thoroughly going through with an idea. This meant to him dedicating more time, be it during the semester or over summer.

SMUMods’ Analytics feature may have been pioneered and led by both of them, but Jia Cheng is convinced that such a feat is surely possible for anyone else with the passion to finish their idea, and I definitely concur with this one.

It’s been amazing to hear about their development journey, learning alongside the pains and gains of building this feature from scratch. I personally can’t wait for their next release, so I got the two to share a little sneak peek:

Cool! What’s next in line? 🚧

Apart from the release of a minimum viable product (MVP) before the next bidding season, the software developer duo indeed has several ideas in mind. These include features to help compare prices between professors, terms, and class timings for the same module. “We also intend to do some web scraping to retrieve all the section details as well,” Cindy added.

These ideas piqued my interest and an undergraduate like me would most definitely benefit from these improvements!

Now that you’ve learned about the process involved in building this feature, head over to smumods.com/analytics/ to try it out for yourself. You’ll be guaranteed to heave a sigh of relief — say goodbye to BOSS’ old interface, and hello to SMUMods Analytics!

Analytics11Analytics12

This article would not have been possible without the assistance of CindyJia Cheng, Zachery, and the rest of the team from SMUMods.