Data Science Interview Guide: Product / Case Question

(This is the second part of the Data Science Interview Guide written under the mentorship of Haiyang.)

I was interviewing for a data science role at an acclaimed start-up. As I was waiting for the interviewer in the interview room, my brain ran through all the possible questions that could come up later.

I was waiting to explain why I was a good fit with the company, to showcase my completed take-home assignment on Jupyter notebook, to nail that question on machine learning algorithms. I was confident. After all, I spent hours preparing for this particular interview. To my relief, the interviewer dished out questions that were mostly unsurprising to me, until he said…

‘Right, so now we’ll test how quick you can think on your feet.’ Oh boy, I thought. ‘How would you measure the customer lifetime value of a user on our platform?’

That was… not expected, but I answered to the best of my ability. And as I was mentored by Mr Sun Haiyang in the NTU Mentorship Programme, I was taught a little something about this kind of question —which is known more generally as a product interview or case question. The following blog post outlines what I have learnt from Haiyang during the programme.

A product interview / case question during a data science interview should not come as a surprise to candidates. In general, there are 6 types of product interview questions.

These questions form the bread and butter of a management consulting interview, but they are also used in data science interviews to test the ability of a candidate to think and act logically given a real-life scenario.Generally, you are given a whiteboard to walk through your response with your interviewer. The interviewer might give you responses as you do so (to give you an indication of whether you’re on the right path).

These questions might seem intimidating at first, as you might not have given thought to that particular question before. But that is exactly the point — the interviewer is giving you a chance to demonstrate your logical thinking process as you crack the novel problem. So take a deep breath, and let’s begin by answering a mock question above: ‘Imagine you are the manager of the Facebook group product. Recently, there is a decrease in user engagement on Facebook groups. What is the root cause?’

In general, the strategy to address the problem is to

1. Understand the problem.
2. Break down the problem into smaller, easily-addressed parts.
3. Address each small part of the problem.

Let’s run through the steps.

Step 1: Understand the problem

This step would involve asking clarifying questions before you start addressing the problem. For this particular question, some salient questions (and the rationale) would be as follows.

Of course, interviewers can decline to answer your question because answering your question might give you too much information or restrict the response unnecessarily. In that case, you will need to make an assumption to the problem.

Let’s imagine that the interviewers’ responses to the clarifying questions are that

• The decrease in usage happened gradually in the past one year.
• The decrease was observed in almost all countries.
• The interviewer declines to answer if any features were introduced.

With these in mind, let’s begin with step 2.

Step 2: Break down the problem into smaller, easily-understood parts.

Let’s recall the question again. The question is about a decrease in user engagement. But what does it mean specifically? Let’s break it down into smaller parts that are Mutually Exclusive and Collectively Exhaustive (MECE). If two parts are mutually exclusive, they cannot exist together. If two parts are collectively exhaustive, the two smaller parts (and nothing else) sums up to the bigger part completely. A textbook MECE example is to break down ‘profit’ into ‘revenue’ and ‘costs’. This is MECE because revenue cannot be considered as cost and vice versa (mutually exclusive) and that revenue and cost sum up to the profit (collectively exhaustive). To illustrate this process, we use an issue tree, as illustrated below.

The concept of user engagement can be thought of the frequency and length of user interaction with the app. We can translate this into (approximately) MECE parts.

User engagement is a factor of

• Number of users on Facebook groups
• Average time a user spends on Facebook group each in each visit
• Average number of times a user visits a Facebook group

With that, we further break down the individual parts into even smaller parts.

• The number of users on Facebook groups is affected by the average number of users on each Facebook group and the number of active Facebook groups
• The average time a user spends on Facebook Groups in each visit is affected by the quantity and quality of content on each Facebook group, the relevance of Facebook groups to users, among other variables.
• The average number of times a user visits a Facebook Group is affected by how frequently Facebook pushes a Facebook Group notification to the user, the relevance of the Facebook group to the users, the average frequency of update of the Facebook groups, the visibility of each Facebook group on the home page.

You see where this is going. With the keyword ‘user engagement’ broken down into smaller parts, we are poised to hypothesize why there is a decrease in user engagement. You can further break down the parts into even smaller parts, which might help you in articulating the response.

Step 3: Address each small part of the problem.

For this part, we will hypothesize how each small part could have contributed to the issue at hand. I will illustrate this with the help of a table below. (During the interview, you are likely to explain your response as you write them on the whiteboard.)

As you can imagine, the interviewer may pose follow-up questions depending on your response. For instance, if your response hints at the decrease in the frequency of Facebook Group notifications (3.1.3), the interviewer might ask you the method of verifying this hypothesis. (Hint: A/B testing is helpful here!). The interviewer might also try to steer you away from a particular direction, so be responsive to the hints given.

After the tirade of follow-up questions, it’s time to pat yourself on the back — you’re done with this segment of the interview!

Being caught off-guard by an interview question is certainly an unpleasant process (I can vouch for it). I hope this article and my learning from Haiyang and Nanyang in the LevelUp journey has helped you a little in understanding the case question, which can arise during a data science interview. Again, I’d like to thank Haiyang for patiently mentoring my data science buddy (Nanyang) and I.

If you’d like to learn more about case questions, the online world is teeming with free resources, such as the YouTube Series Case Interview 101. Another excellent resource which I used to learn case questions is Case Interview Secrets by Victor Cheng.

If you have more questions or any feedback, feel free to comment down below, or simply connect with me on LinkedIn!