This tutorial will show you how to apply Pandas drop duplicates to move out duplication rows from a dataframe.
The tutorial will explain what the technique does, explain the syntax, and IT will also reveal you clear examples.
You rear end click on any of the following links, and IT will take you to the fit section in the tutorial.
Mesa of Table of contents:
- An Founding to Pandas Drop Duplicates
- The sentence structure of
drop_duplicates() - Examples: How to drop duplicate rows from a dataframe
- FAQ
If you need something specialized, you can click on one of the links above.
Having aforesaid that, if you really want to know how this technique works, you should probably read the whole tutorial.
An Introduction to Pandas Drop Duplicates
Expressed just, the Pandas drop duplicates method acting removes duplicate rows from a Pandas dataframe.
I'll show you some examples of this in the examples incision, but first, I want to quickly review some fundamentals about Pandas and Pandas dataframes.
This will give you some context, and avail you understand exactly what this technique does, and why we might utilisation it.
Dataframes Fund Python Data
First, let's quickly review what a dataframe is. (Recall: the cast duplicates method operates happening Pandas dataframes.)
A dataframe is a data structure in Python that's available in the Pandas parcel.
We enjoyment dataframes to lay in predestined types of information, and we use Pandas techniques to wangle dataframe data.
Dataframes have Rows and Columns
Let's look more carefully at dataframe anatomical structure. Pandas dataframes shop data in a quarrel-and-column format.
Typically, the columns of a dataframe are variables, and the rows typically record individual data records. E.g., if you had a dataframe with sales data, the individual rows might record the sales entropy for unshared people.
If you've ever used Excel, a Pandas dataframe is really a lot like an Excel spreadsheet, in the sense that they some have this row-and-newspaper column social system.
It's Possible to Have Rows with the Exact Equal Information
Significantly, it's possible for a dataframe to have duplicate rows of information.
So for example, if two rows had the same value for every column, we'd consider those to be duplicate rows.
There's also a different type of parallel, where two rows have the same value for one or more cardinal columns (but not necessarily every column).
Sometimes, Matched Records are Bad
Sometimes, these duplicates are okay.
When you're practical with some types of data, IT might be normal – justified expected – to accept duplicate rows of data.
But at that place are many instances where having duplicate rows of data is bad. Sometimes, duplicate rows can make problems with an analysis or a specific data science technique.
For example, imagine you'Ra working with sales data. If a data organisation accidentally recorded a duplicate record for a particular salesperson, information technology mightiness over-report that person's actual sales performance.
Frankly, at that place are many possible examples in fields like selling, finance, accounting, and else areas, where having same data could be an issue.
In addition, there are certain types of data wrangling trading operations where duplicates can cause big problems.
In especial, duplicated data ofttimes causes problems when information scientists aggregate information.
Double up rows can also be a really big problem when you merge or join multiple datasets together. As a matter of fact, before you do a fall in, you almost ever need to hitch for duplicate records!
With that said, as an analyst operating theater information scientist, you pauperism ways to name and remove double rows of data.
Set down Duplicates Removes Duplicate Rows
There are respective ways to identify twinned rows. In Python, you can use the Pandas unique method. There are also or s ways to identify duplicates by using aggregation and summation.
We'rhenium not going to masking those tools in this teacher.
Here, we'rhenium going to discuss the drop duplicates technique, which enables you to remove duplicate rows from a dataframe, one time you've found them.
There's actually a few different ways to take duplicate rows, and information technology really depends on several parameters in the syntax.
Having said that, let's take a look at the syntax of Pandas drop duplicates, then we can better understand how information technology works.
The syntax of drop_duplicates
Here, I'll excuse how the syntax of the Pandas drop_duplicates() method.
I'll show some examples later in the example section, but here, I just want to fail the sentence structure bit by spell.
dataframe.drop_duplicates syntax
Soh how does the syntax puzzle out?
The first thing to know about the drop_duplicates syntax is that this proficiency is a method, non a serve.
Specifically, it is a dataframe method.
That being the case, the low step to calling the method is to typewrite the call of your Pandas dataframe.
(Obviously, this assumes that you've created a dataframe first. For more information about creating dataframes, read our instructor most Pandas dataframes.)
Satisfactory. So first, you type the name of your dataframe.
Then, you use questionable "dot sentence structure" to call the drop_duplicates method.
Then, in spite of appearanc the parenthesis, you can use a few parameters to control exactly how drop_duplicates works.
Let's take a consider those.
The parameters of drop_duplicates
There are a fewer parameters for the drop_duplicates() function:
-
subset -
keep -
inplace -
ignore_index
Let's discuss what each of those parameters does.
subset (optional)
The subset parameter enables USA to specify a subset of columns in which to look for duplicate data.
By default, drop_duplicates() will look at all variables … meaning that it will search for rows of data where all of the data is the same.
Still, when we use subset, we can specify a list or sequence of tower name calling in which to search for duplicate information. (I'll show you examples of this in example 2 and example 3.)
Also note that this parameter is nonmandatory. If you don't use it, it will front at all columns by default.
keep (optional)
The keep parameter enables United States of America to specify which duplicate row of data to keep.
This parameter is optional.
If you don't use IT, then past default, this will be set to keep back = False. This will cause drop_duplicates to delete wholly of the duplicated rows.
Alternatively, you derriere set keep to 'first' operating theater 'last', in which case IT will proceed the best reduplicate row or last duplicate row, severally.
inplace (optional)
The inplace parameter enables us to specify whether or non drop_duplicates will operate directly on the input dataframe.
Like many other Pandas functions and methods, by default, drop_duplicates creates a new dataframe as the output. That means that by default, drop_duplicates leaves the dataframe you'Ra operating on unchanged. That's because by default, the inplace parameter is set to inplace = False.
You fire deepen this doings by scene inplace = True. If you do this, drop_duplicates will directly qualify your input dataframe.
Simply personify careful! Make a point that the method acting is working exactly as you intend before you directly change your data!
ignore_index (optional)
The ignore_index parameter controls the index of the output, after the duplicates have been removed.
By default, this is set to ignore_index = False. This causes drop_duplicates to stay fresh the same index values for the undeleted rows that rest in the output.
If, however, you set ignore_index = False, drop_duplicates will create a new index for the output, starting at 0 and ending at n – 1.
The output of drop_duplicates
Arsenic I've mentioned, the nonremittal end product of Pandas drop_duplicates is a new dataframe.
That's because past default, the inplace parameter is set to inplace = False.
Alternatively, you can set inplace = Trusty, which leave campaign drop_duplicates to directly modify the dataframe you're operating on.
But as I've mentioned, you need to follow careful with this! Make a point that the method acting is doing what you expect it to!
Examples: How to Strike down Duplicate Rows from a Pandas Dataframe
Now that we've looked at the phrase structure, have's claim a take just about examples of how the Pandas drop duplicates method works.
In these examples, I'll show you several different ways to knock off duplicate rows from a Pandas dataframe.
You send away click on whatever of the following links, and it will issue you to the appropriate example.
Examples:
- Search for rows where all data is the same and keep first wrangle (default option)
- Keep the last duplicate row
- Drop all duplicate rows
- Look to duplicates on one and only variable
- Looking at for duplicates on multiple variables
- Operate your dataframe "in situ"
Before you jump into the examples though, we need to import Pandas and create the dataframe we'll study with.
Significance Pandas
First we need to import Pandas.
meaning pandas as Pd
Hera, we're importing Pandas with the alias 'atomic number 46'.
This is the vernacular convention, so we'll use IT going forward.
Create Dataframe
Next, we'll create a simple dataframe.
sales_data = atomic number 46.DataFrame({"name":["William","William","Emma","Emma","Anika","Anika"] ,"region":["Eastside","East","North","Mae West","East","East"] ,"sales":[50000,50000,52000,52000,65000,72000] ,"expenses":[42000,42000,43000,43000,44000,53000]}) And Lashkar-e-Taiba's print the data out.
print(sales_data)
OUT:
name region gross sales expenses 0 William East 50000 42000 1 William Eastside 50000 42000 2 Emma North 52000 43000 3 Emma West 52000 43000 4 Anika E 65000 44000 5 Anika East 72000 53000
Observe that the data has some types of double records.
The first 2 rows (for William) are exactly the same.
The second two rows (for Emma), have the same epithet, but the neighborhood variable is unusual.
And the final two rows (for Anika) receive the same name and region, simply the information for sales and expenses are different.
I'll show you how to deal with these different cases in the pursuit examples.
Model 1: Search for rows where all data is the same and keep first row (default)
First, we'll just use drop_duplicates() with the default doings.
Here, Pandas shake off duplicates will find rows where all of the information is the Lapp (i.e., the values are the same for every chromatography column). It will keep the first row and delete all of the other duplicates.
Let's take a look.
sales_data.drop_duplicates()
OUT:
name region gross sales expenses 0 William East 50000 42000 2 Emma North 52000 43000 3 Emma West 52000 43000 4 Anika East 65000 44000 5 Anika Easternmost 72000 53000
Explanation
Point out that here, drop duplicates deleted row 1.
In the original dataset, row 0 and row 1 (the rows for William) were identical.
When we use drop_duplicates with the default setting, it only operates on rows where all of the values are the same. And past default, information technology keeps the first row (in this case, row 0).
Model 2: Keep the live on duplicate row
Side by side, we're going to change our code and then that drop_duplicates keeps the last duplicate row.
(Remember in the previous case, aside default option, drop duplicates kept the first double row.)
Let's get a load.
Here, we'll set keep = 'ultimate' to campaign drop_duplicates to keep in the last row:
sales_data.drop_duplicates(keep = 'last')
OUT:
name region sales expenses 1 William East 50000 42000 2 Emma North 52000 43000 3 Emma West 52000 43000 4 Anika East 65000 44000 5 Anika Eastside 72000 53000
Account
In this example, drop duplicates operated along row 0 and row 1 (the rows for William).
Remember: away default, Pandas drop duplicates looks for rows of data where all of the values are the same. Therein dataframe, that applied to rowing 0 and row 1.
But here, instead of keeping the first duplicate row, IT kept the last extra row.
It should atomic number 4 pretty obvious that this was because we bent keep = 'net'.
This literally causes the method to keep the last duplicate, out of every last the duplicates that it finds.
EXAMPLE 3: Shake off wholly duplicate rows
Now, we'll modify the encode so that it deletes all of the duplicates that it finds.
Here, we'll call Pandas drop_duplicates with keep = False.
Remember that by default, drop_duplicates will look to rows where entirely of the values are the same.
Let's get a load:
sales_data.drop_duplicates(keep = False)
OUT:
name region gross revenue expenses 2 Emma North 52000 43000 3 Emma West 52000 43000 4 Anika East 65000 44000 5 Anika East 72000 53000
Explanation
Once again, this code is operating just happening the rows where all of the data are the same … rows 0 and 1 (the data for William).
In these rows, every value is the equal.
But instead of keeping one quarrel and deleting the other duplicates, this code is deleting all of the duplicates.
That's because we set keep = False.
When we ordered dungeon = False, Pandas drop_duplicates will off all rows that are duplicates of other row.
EXAMPLE 4: Look for duplicates happening one varied
Next, let's look away for duplicates happening a specific variable.
In the past examples, drop_duplicates looked for rows where all of the data was the same. The information needed to make up the same for every varied.
Here, we're going to use the subset parameter to nidus on one specific variable.
We'll set subset = ['name'] to look for records where only the value for name is the same.
Here's the code:
sales_data.drop_duplicates(subset = ['name'])
OUT:
name region sales expenses 0 William East 50000 42000 2 Emma North 52000 43000 4 Anika Eastern United States 65000 44000
Explanation
Here, the drop duplicates method looked for same information only in the name inconstant.
When looking at the name variable, there were duplicates for rowing 0 and 1 (William), rows 2 and 3 (Emma), and rows 4 and 5 (Anika).
Afterwards finding these rows that had matching name calling, it deleted the second same (remember, that's the default behavior).
Also notice syntactically how we executed this. When we use the subset parametric quantity, we provide a list of variable names.
In this case, we only wanted to look for duplicates connected the name variable, so we set subset = ['call'] (i.e., a list with only one inconstant name).
Instance 5: Seem for duplicates on multiple variables
Now let's see for duplicates on a subset of multiple variables.
This is in reality very similar to the past example, but here, we'll look at both name and region.
sales_data.drop_duplicates(subset = ['refer','region'])
OUT:
name region sales expenses 0 William East 50000 42000 2 Emma North 52000 43000 3 Emma West 52000 43000 4 Anika East 65000 44000
Explanation
In this example, the drop_duplicates method operated connected the rows for William (rows 0 and 1) as well as the rows for Anika (rows 4 and 5).
Why?
Here, we set subset = ['name','region'].
This caused drop_duplicates to seek for records where name and realm were the aforesaid.
That applied to rows 0 and 1, which had the same appoint and region.
It also applied to rows 4 and 5, which also had the very name and region.
Only it socialist rows 2 and 3 alone. The reason for this is because even though rows 2 and 3 have the same figure (Emma) they accept a different region.
EXAMPLE 6: Operate on your dataframe "in place"
Finally, let's operate on on our dataframe "in situ".
Remember: by default, when we use drop_duplicates, Pandas will make up a new dataframe as the yield, and will leave the stimulation dataframe unchanged.
By exploitation the inplace parameter, we can chance this demeanour.
Create copy dataframe
Instead of operating directly on sales_data, we'll function on a copy of the dataframe.
This is often a cagy matter to do before you execute a data wrangling technique.
Ofttimes, you want to test your write in code to earn sure that it works by rights ahead you operate happening your information flat.
sales_data_copy = Pd.DataFrame({"key out":["William","William","Emma","Emma","Anika","Anika"] ,"region":["East","East","North","West","Orient","East"] ,"sales":[50000,50000,52000,52000,65000,72000] ,"expenses":[42000,42000,43000,43000,44000,53000]}) There are a a couple of ways of copying your information, but here, we're reasonable creating a polar dataframe called sales_data_copy.
If your dataset is particularly large, you might neediness to create a subset or a sampling of rows, since copying a very large dataset mightiness cause storage problems.
Drop-off duplicates with inplace = True
Ok. Now that we have sales_data_copy created, Army of the Pure's operate directly along this dataframe.
Here, we'Ra going to set the inplace parameter to inplace = True.
This will cause drop_duplicates to directly transfer the input dataframe, sales_data_copy.
Let's take out a feel.
sales_data_copy.drop_duplicates(inplace = True up)
And now, Lashkar-e-Toiba's print the dataframe.
print(sales_data_copy)
OUT:
distinguish realm gross revenue expenses 0 William East 50000 42000 2 Emma North 52000 43000 3 Emma Westernmost 52000 43000 4 Anika East 65000 44000 5 Anika East 72000 53000
Account
You might have detected that drop_duplicates did not send off the output directly to the console.
Instead, we needed to mark out sales_data_copy to deal the original dataframe.
In this case, sales_data_copy was actually changed directly. Here, drop_duplicates deleted the duplicate rows (row 1, the last quarrel for William). Furthermore, it operated directly happening sales_data_copy, without creating a new, different dataframe as output.
That's what happens when we circle inplace = True.
But be careful!
When you situated inplace = True, drop duplicates will immediately overwrite your input data, so you need without doubt that your code is operating exactly A you want to.
Ofttimes Asked Questions about Pandas Drop Duplicates
Forthwith that you've learned about the Pandas drop duplicates method acting, and now that you've seen some examples, have's review a average question.
Often asked questions:
- Why isn't drop_duplicates changing my dataframe?
Question 1: Why isn't drop_duplicates changing my dataframe?
In that respect's a adventure that you've run drop_duplicates, only to find that your original input dataframe is unchanged.
This is because by default, Pandas drop duplicates creates a new dataframe arsenic an yield, and keeps the original input dataframe unvarying.
To change this demeanour and directly modify the stimulant dataframe, you need to set inplace = True.
I covered this step by interfere example 6.
Leave your other questions in the comments below
Do you have new questions about the Pandas shake off duplicates method acting?
If you serve, just leave your questions in the comments section downstairs.
If you want to master Pandas, join our course
In this tutorial, I've explained how to wont the drop_duplicates to remove duplicate rows from a Pandas dataframe.
This instructor should help you understand this one technique, but if you really want to master data use in Pandas, there's a circle much to learn.
That said, if you need to victor data wrangling with Pandas, you should join our insurance premium online course, Pandas Mastery.
Pandas Mastery will teach you everything you need to jazz about Pandas, including:
- How to create dataframes
- How to subset your Python information
- Data aggregation with Pandas
- How to wrangle rows and columns
- How to reshape your data
- and much more …
Moreover, IT will help you completely master the syntax within a couple of weeks. You'll discover how to become "fluent" in writing Pandas encrypt to manipulate your data.
Find exterior more Hera:
Hear More than Close to Pandas Subordination
How to Change Fan Belt on a Panda Easy 1200 Yr 2013
Source: https://www.sharpsightlabs.com/blog/pandas-drop-duplicates/