Clean And Analyze Employee Exit Surveys part 7 NaN Issue

Screen Link: https://app.dataquest.io/m/348/guided-project%3A-clean-and-analyze-employee-exit-surveys/7/identify-dissatisfied-employees

The output should read:
False 241
True 91
NaN 8

and

False 162
True 149

but as you can see in my screenshot, it’s not coming out that way and I can’t figure out why. Any ideas on how to fix this?

Hey, Beckett.

This very much depends on what you did before this. Please share your notebook so people can take a look.

The reason this is happening is that you’re applying update_vals to columns in dete_resignations, making every value in those columns True. update_vals is only supposed to be applied to tafe_resignations.

3 Likes

Hi Beckett,

Did you fix the problem for tafe_resignations_up? I have a similar problem with non NaN. My answer is also:
False 241
True 91
True 8

When I go to the definition website for pandas.any(), I found the description for skipna:

skipna bool, default True

Exclude NA/null values. If the entire row/column is NA and skipna is True, then the result will be False, as for an empty row/column.
If skipna is False, then NA are treated as True, because these are not equal to zero.

Apparently, if skipna is False, then outputs of NA values will be true. How can we generate:
False 241
True 91
NaN 8

Thanks.

1 Like

I also have the exact same problem. Can’t seem to find a way around this :frowning:

Hi Cardons,

Glad to know that this is not only my problem. To fix this problem, I even try to download and compile the solution jupyter notebook from dataquest. However, I still got the same result as :
False 241
True 91
True 8

Is it because of the version of pandas library or jupter notebook?
Thanks.

People asking questions in this topic, I recommend you create your own topic asking your specific question and share your notebooks.

2 Likes

Thanks for your advice.
I created my post for the NaN question

1 Like

Hi branBeckett,

Were you able to find a solution for this? I just stumbled upon this strange error as well. Additionally, I downloaded the solution and it produced 2 Trues and 1 False. I know it’s been a while since anyone has posted anything, but I also saw no one responded to @bernard.bmehk as well. If anyone has a solution, that’d be great - thanks!

Hello,

So I wasn’t able to find a solution for this. I believe the issue occurred after I updated Pandas to version 1.0.0

There is an experimental new feature added to denote missing values with pd.NA, and you can view the documentation here:
https://pandas.pydata.org/docs/whatsnew/v1.0.0.html

I’m just not sure how to use it properly to replace np.nan. I still need to experiment with it some more.

Let me know if you find anything out!

Thanks for the response!

I checked out the documentation and played around with pd.NA but was also unable to obtain the desired outputs of True, False and NaN.

Interestingly enough, I was able to get through the guide by using the fillna method on the last part of the instructions and it successfully filled the extra ‘True’ (which was supposed to be NaN values) with True values and it worked.

To me, it seems like a bug or glitch that causes NaN to be represented as an extra True but still shows up as NaN within the filled values. Definitely very strange!

Update:

This is due to a bug in pandas version 1.0.1. Please update the pandas version in your local environment if you are experiencing this issue.

Thank you, @ebuschang, for figuring this out.

Best,
Sahil

3 Likes