Apply method faster than series.str method for string manipulation

Q&A DQ Courses
,
#1

Hi all,

Screen Link:
https://app.dataquest.io/m/346/working-with-strings-in-pandas/2/using-apply-to-transform-strings

I used the apply method with a function to select the word at -1 position on the CurrencyUnit and problem and timed it and got back 991 microseconds while the series str method (https://app.dataquest.io/m/346/working-with-strings-in-pandas/3/vectorized-string-methods-overview) gave me 1.73 milliseconds even though series.str was told to be faster than the apply.

Anybody has any insight on this?

My Code: 

#Apply method:
def extract(element):
    raw=str(element)
    splitd=raw.split()
    word=splitd[-1]
    return word

%time merged["CurrencyUnit"]=merged["CurrencyUnit"].apply(extract)


merged["CurrencyUnit"].head()

#Series.str method:
%time merged['CurrencyUnit']=merged['CurrencyUnit'].str.split().str.get(-1)
merged['CurrencyUnit'].head()

What I expected to happen:
I expected series.str to be faster

What actually happened:
Apply method turned out to be faster


![apply method for string manipulation|690x286](upload://dlAnZYdSpkS8S7wFbWMgpaagjfd.png) ![series str method|690x280](upload://fGOhBO4PTafWLvUtCr9hQ1yB1Cw.png)
1 Like
#2

Hi @immanuel.ajay.r

The answer is: vectorization

First let’s define vectorization, basically it’s the process of executing operations on entire arrays. In other words, instead of making a operation line per line (like using .str.split() etc etc), it makes the operation on the entire dataset

This two articles might help you to understand more about it


https://engineering.upside.com/a-beginners-guide-to-optimizing-pandas-code-for-speed-c09ef2c6a4d6?gi=5b8b0f7ac9db
1 Like
#3

Yeah, but series.str is supposed to be using vectorization and hence, supposed to be faster
but not apply()

But i did find some info here https://stackoverflow.com/questions/50744369/how-to-speed-up-pandas-string-function

It’s not clear cut as to why but the discussion hints at what may be causing apply() to run faster than series.str

3 Likes