The Remove-DuplicateItems PowerShell script by Michel de Rooij is a powerful script used to clean up duplicate messages within Microsoft Exchange mailboxs, including Microsoft Office 365.
The script has been around for many years and Michel has done a great job at keeping it up to date and optimizing it. While they do provide some instructions that are kept up to date, I found they were lacking when it came to using the script with Microsoft 365 Online.
The script is easy enough to use if you can use Basic auth (just a username and password) but for the most part, you must use Modern auth (with multi-factor authentication) these days, which makes it harder to run from a PowerShell script that doesn't support that natively.
These instructions are here as my notes on one way (there are several valid ways) to get it set up and working with Office 365.
Download the PowerShell script and the supporting files from GitHub https://github.com/michelderooij/Remove-DuplicateItems/ and save them all together in a folder.
You may be able to get away with only the PowerShell script and not the .dll files if you run the following commands:
Register-PackageSource -Provider NuGet -Name nugetRepository -Location https://www.nuget.org/api/v2
Install-Package Exchange.WebServices.Managed.Api
This downloads the files from https://www.nuget.org/packages/Exchange.WebServices.Managed.Api/ and installs them in C:\Program Files\PackageManagement\NuGet\Packages\
While I did do this, I also downloaded the copies from GitHub because I had more success getting those to work. Possibily because I'm missing a command to get the Microsoft.Identity.Client files?
For what it's worth the files from NuGet were identical to the ones in Michel's GitHub. (If I made sure the line endings of the .xml file matched -- CR LF vs just LF)
Load up what used to be Azure AD, but is now the Microsoft Entra admin center
Under Identity > Applications > App registrations
Tap New registration at the top
Name it whatever you want, I called mine Remove Duplicate Items Script
Supported account types: Accounts in this organizational directory only (Single tenant)
You can leave the Redirect URI area blank since we aren't actually making a web app. Tap Register
Take note of the Application (client) ID and the Directory (tenant) ID as we will need both of these later.
We'll have to give the script permissions to access our mailboxes in order to run.
While looking at your newly created app registration, tap API Permissions. You'll see by default it already has permissions to Microsoft Graph's User.Read.
Tap Add a permission
Choose the APIs my organization uses tab and search for Office 365 Exchange Online
We are going to use Application permissions and give it access to full_access_as_app so it can access the full mailbox. Tap Add permissions
You'll need to tap the Grant admin consent for organization in order to actually grant that permission.
Still looking at the App registration created above, tap Certificates & secrets.
Under Client secrets, tap New client secret
Enter any description and set an expiration date for this secret. Remember, this app has full permissions to all your mailboxes on Exchange Online, so realize how important this secret is to keep safe! Copy the secret value that it creates. After you leave this screen, it will not show it to you again (but you can create another one later).
You can use certificates as well with the script, but these instructions don't cover that scenario.
To run the script on Exchange Online, include the TenantId, ClientId, and Secret from above, and set Server to outlook.office365.com.
Secret must be a PowerShell SecureString, so you can use the command $secret = ConvertTo-SecureString -String "the secret value that was given earlier" -AsPlainText -Force
Then, for example: .\Remove-DuplicateItems.ps1 -Identity user@example.com -Server outlook.office365.com -TenantId 00000000-0000-0000-0000-000000000000 -ClientId 00000000-0000-0000-0000-000000000000 -Secret $secret -Report -Verbose -WhatIf
Obviously replace the secret and ids with your own. The -WhatIf flag lets you see what the script would do without it actually removing any messages until you are ready.
When you are done with the script, feel free to delete the app's Client Secret or even the entire App registration if you never plan on running the script again, or don't mind setting it back up when you do. Because remember that anyone with your tenant id, app's client id, and secret has full access to your organization's emails.
Last updated: 2023-07-24