Less book work … more museums!

School’d: five star ratings and high schoolers …

School’d is a series about the data we collect at Slader and what we’re learning from it.

Some of this data is pure novelty - fun stuff that we’ve become experts in from spending hours with our site and observing our users’ behavior.

Other learnings seem more significant - not just in terms of how we run our site, but in regards to how students today are learning, and how they’re using the Internet to support their learning.

Hundreds of thousands of students visit Slader.com each week to help them with their homework. They are here by choice, not at the urging of their parents, their schools, or their teachers, and they’re taking a proactive approach to their own learning.

What can we learn from them?

Five-star rating systems have their drawbacks. Most users will either rate something a 1 a 4, or a 5. They will either love something or hate it, but not much in between, or if they don’t feel strongly, they’ll just not rate. Users are used to the system, and it looks ‘normal’ … but is it useful to us?


Here’s a breakdown of how users rate content on Slader. A clear longtail with most users rating just a few pieces of content. But, there are still many having rated 10-30 pieces of content, and some that have rated thousands of pieces of content.


I thought it’d be telling to see how our ratings compare for different types of content and in relation to other behaviors. I’ve charted alongside each other a) all ratings, b) ratings for Calculus solutions, c) ratings for Pre-algebra solutions and d) ratings where the user also commented on the content.

The first piece of information that stands out is that if a user comments on a piece of content, they’re much more likely to rate it a 1 than otherwise. We see this often - a user will point out a mistake with a solution or comment that they’d like more clarification. It’s a good method of getting the user who submitted the content to provide some additional information or fix an error.

Also noticeable are the lower number of low scores on Pre-algebra content relative to overall ratings counts, and the high relative number of lower ratings on Calculus solutions … it’s anyone’s guess as to why. Perhaps younger students have a bit less negativity in them?

Note: I’ve taken slight liberties in the y-scale of this chart in order to be able to better compare the datasets side-by-side.


And now, for how those ratings break down by user. Because many users have rated just a few pieces of content, we see obvious spikes (which I’ve cropped) at the round numbers (users can’t rate content less than a whole number of stars). And the average ratings skew toward the higher end of the scale and are somewhat scattered otherwise. An interesting dataset, that even given the hundreds of thousands of datapoints we’re looking at doesn’t show a clear pattern. Need more data!


What this chart mostly tells us is that users rate content a 1 and a 5. If they have an average rating near 2.5, they have a lot of variance. Closer to the ends we see more consistency, they either hate or love all content.


In summary, what’s clear from a quick review of this data is that using star ratings for our content on Slader perhaps isn’t the ideal rating system. Users seem to use the 1 and 5 star buttons to rate content, and mostly ignore the rest of the buttons. We learn if they hate or love something, but more rarely do we get a nuanced assessment of the quality of the content.

And how does this data differ from elsewhere? Sites like Netflix have users who seem to rate content with a bit more keen eye. Most other systems (movies, restaurants, products) tend to see a heavy loading of high star ratings (4s and 5s), but we see more polarization. One reason is that most ratings on Slader apply to math solutions and if a user perceives that solution as ‘wrong’, then it’s a much more polarized feeling than not really liking the latest Tarantino film or the Chicken Parmesan at the local Italian joint.

Is it worth changing the system we use? Or do we make improvements to the star ratings? I tend to think that the possibility of gathering more nuanced data is better than assuming we’ll never get it …

Peter Bernheim is CTO of Slader.com. Questions? Comments? Something to add? Email me at peter.bernheim@slader.com

New version of the Slader iPhone app!

We’ve released a new version of the Slader iPhone app - and it’s FREE and AWESOME. Download it now!

New features:

  • Place bounties!
  • Buy gold in-app!
  • Upload photo solutions!
  • Improved graphics and layout!
  • Much much more!

School’d: Trick or Treat

School’d is a series about the data we collect at Slader and what we’re learning from it.

Some of this data is pure novelty - fun stuff that we’ve become experts in from spending hours with our site and observing our users’ behavior.

Other learnings seem more significant - not just in terms of how we run our site, but in regards to how students today are learning, and how they’re using the Internet to support their learning.

Hundreds of thousands of students visit Slader.com each week to help them with their homework. They are here by choice, not at the urging of their parents, their schools, or their teachers, and they’re taking a proactive approach to their own learning.

What can we learn from them?

Some teachers realize that students aren’t going to have time for homework on Halloween. Others don’t. It’s much better when Halloween falls on a Friday or Saturday!

Peter Bernheim is CTO of Slader.com. Questions? Comments? Something to add? Email me at peter.bernheim@slader.com

School’d: Textbooks over time

School’d is a series about the data we collect at Slader and what we’re learning from it.

Some of this data is pure novelty - fun stuff that we’ve become experts in from spending hours with our site and observing our users’ behavior.

Other learnings seem more significant - not just in terms of how we run our site, but in regards to how students today are learning, and how they’re using the Internet to support their learning.

Hundreds of thousands of students visit Slader.com each week to help them with their homework. They are here by choice, not at the urging of their parents, their schools, or their teachers, and they’re taking a proactive approach to their own learning.

What can we learn from them?

Since most of our content at Slader is based on textbooks, we’ve learned a lot about the textbook industry, the key players (publishers and authors) and the way that textbooks change over time. New editions of textbooks are released every few years, often with very little changed except a problem here, or a new state standard addressed there.

Below, I take a quick look at the way that two textbook families have changed over time. The families include everything from state-specific versions of a textbook to truncated versions of the same material, as can be seen in the varying lengths of the books shown.


Book 1 above is a popular upper level textbook family. While newer versions of this book have been introduced recently, it’s clear that many students are still using older versions. It’s common for books upwards of 10 years old to still be quite popular, especially with upper level math; the state standards are less subject to change, plus I’d guess the students tear out and doodle on pages less often.


Book 2 is a lower level textbook. More users have clearly shifted to the latest edition, with just a few using the older versions of the books.

While not surprising, it’s interesting to see how clearly page counts map to exercise counts, even across Book 1 and Book 2. Also worth noting (though not clearly seen here): even though textbooks change often, many of the exercises stay the same year after year, just wrapped in a new shiny cover.

Peter Bernheim is CTO of Slader.com. Questions? Comments? Something to add? Email me at peter.bernheim@slader.com

School’d: Predicting student failure - an introduction

School’d is a series about the data we collect at Slader and what we’re learning from it.

Some of this data is pure novelty - fun stuff that we’ve become experts in from spending hours with our site and observing our users’ behavior.

Other learnings seem more significant - not just in terms of how we run our site, but in regards to how students today are learning, and how they’re using the Internet to support their learning.

Hundreds of thousands of students visit Slader.com each week to help them with their homework. They are here by choice, not at the urging of their parents, their schools, or their teachers, and they’re taking a proactive approach to their own learning.

What can we learn from them?

We often say that the data that we collect at Slader allows us to ‘predict student failure’. By this, we mean that by looking at a student’s behavior on Slader, we develop an understanding of problem areas and concepts that challenge them more than others. And, based on data patterns, we can use this history to predict the exercises/concepts which they will, in the future, have trouble with.

Knowing this information can provide a number of opportunities:

  • offer additional resources that might aid this student
  • predict exercises they might have trouble with
  • suggest practice problems/tutorials around the areas with which they are having trouble

One of our core beliefs at Slader is that students seeking homework help online probably don’t know what concepts they’re struggling with - they just know they are struggling. Our solution product aims to offer help exactly where they need it, the homework they’re tasked with completing on a given school night. If we can assess trouble areas and understand where a student is heading, we can help them to brush up on concepts or techniques which they might not even know they are struggling with. The first step in getting help is to identify what you need help with - that’s where we are heading with this data.

As we develop these features and refine our algorithms, I thought that sharing some background behind the data we are using and what it tells us might spark some good discussion.

I’m heading back to Stewart Calculus, one of our most popular books. Looking back at a popular, and often challenging problem from an earliest post, we’re going to look at Page 151, exercise 29.

I pulled a set of users who had trouble with this exercise, both showing a higher view time than their average time spent with a solution, and who spent more time than their peers. A rough breakdown of how these students’ time is spent is shown in Fig. 1 below.

I’ve filtered out views that seemed extra long here - students are a distractable bunch, and most likely that wandered off to Tumblr or Facebook and lost track of time. So, extra long view times are omitted. Once those outliers are pulled out, we get a fairly good chart of median (blue) and mean (red) solution view times. I used this data to understand how best to evaluate whether or not a user was finding a specific solution challenging.

For simplicity, I’ve further narrowed the dataset for this post based on users who spent more than 425 seconds with Pg. 151, Ex. 29 AND who spent more than their average time with the solution. For each of these users, I then pulled their histories for two weeks: Sept. 22 and Oct. 13. This allows a concentrated pool of exercises to look at. For all of the exercises viewed, I then mapped out the exercises which challenged these users both prior to and after viewing exercise 29. This resulted in Fig. 2.

We see a long tail of exercises that challenged just a few users. And a small set of exercises jump out as being challenging to a large group of users at the right edge of the chart.

Here are three examples, one from earlier in the book, one from the same problem set, one from later in the book:
Page 128, Ex. 41
Page 150, Ex. 5
Page 182, Ex. 51

What similarities can we see in these exercises? Why were they all so challenging to this group of students?

Ex. 41 on Page 128 involves understanding the graph of a function and where the function is discontinuous. Ex. 5 on Page 150 involves finding a tangent line to a curve. Ex. 29 on Page 151 finds the limit of a function (aka the line tangent to the curve at a point). And Ex. 51 on Page 182 asks for the horizontal tangent line of a curve. See any patterns? The prompts on Page 150 and 182 are nearly identical. And the concepts involved in all 4 are closely related. These students need some practice with visualizing functions as graphs and understanding the tangent lines along those graphs. With some added practice, we might be able to head off issues with these same concepts down the line.

The end goal of predicting student failure is preventing it - if we can see trouble before it happens, we can proactively help these students succeed.

As educators, what sort of data would be useful to you in predicting student failure? As a student, what might be useful information to help your studies?

Peter Bernheim is CTO of Slader.com. Questions? Comments? Something to add? Email me at peter.bernheim@slader.com

Slader in PandoDaily

Slader: The “cool” homework help platform that your mom doesn’t know about (via Pando Daily)

By Cale Guthrie Weissman On October 11, 2013High school homework is one of those shitty things most people are glad they no longer have to endure. For me, the bane of my existence was always AP Physics. Not only did I not understand the subject, but…