Get Site Crawl Logs | SharePoint Search PowerShell Script

Get Site Crawl Logs

This PnP PowerShell script retrieves SharePoint site crawl logs to help troubleshoot search indexing issues and monitor crawl performance. Essential for diagnosing why content may not be appearing in search results.

Purpose

Site crawl logs provide valuable insights for:

Identifying crawl errors and indexing issues
Monitoring search crawler activity and performance
Troubleshooting missing content in search results
Understanding crawl patterns and frequency
Optimising content for better search visibility

Prerequisites

PnP PowerShell module installed
SharePoint Online Admin permissions
Connection to your SharePoint Online tenant
Site collection administrator rights
Assign youself permissions to the crawl logs here...
https://tenantName-admin.sharepoint.com/_layouts/15/searchadmin/crawllogreadpermission.aspx

PowerShell Script

# Retrieves crawl log entries for a specific site with crawl timestamps

            # Variables
            $clientId = ""
            $outputPath = ""
            
            Connect-PnPOnline -Url "https://tenantName-admin.sharepoint.com" -ClientId $clientId -Interactive
            
            $endDate = Get-Date
            $startDate = $endDate.AddDays(-100)
            
            Write-Host "Retrieving crawl log from $startDate to $endDate..." -ForegroundColor Cyan
            
            # Filter by URL after retrieving
            $crawlLog = Get-PnPSearchCrawlLog -ContentSource Sites -StartDate $startDate -EndDate $endDate -Filter "https://tenantName.sharepoint.com/sites/siteName"
            
            if ($crawlLog) {
                Write-Host "Crawl entries found: $($crawlLog.Count)" -ForegroundColor Green
                
                # Display summary in console
                Write-Host "`nCrawl Status Summary:" -ForegroundColor Yellow
                $crawlLog | Group-Object StatusCode | Select-Object Count, Name | Format-Table -AutoSize
                
                # Display recent entries with timestamps
                Write-Host "`nMost Recent Crawl Entries:" -ForegroundColor Yellow
                $crawlLog | Sort-Object CrawlTime -Descending | Select-Object -First 20 | 
                    Select-Object Url, StatusCode, StatusMessage, CrawlTime | 
                    Format-Table -AutoSize
                
                # Export full details to CSV
                $csvFile = Join-Path $outputPath "crawl_log_$(Get-Date -Format 'yyyyMMdd_HHmmss').csv"
                $crawlLog | Select-Object Url, StatusCode, StatusMessage, CrawlTime, ErrorMessage, Level | 
                    Sort-Object CrawlTime -Descending | 
                    Export-Csv -Path $csvFile -NoTypeInformation
                
                Write-Host "`nFull crawl log exported to: $csvFile" -ForegroundColor Green
                
            } else {
                Write-Host "No crawl entries for this site" -ForegroundColor Yellow
            }
            
            Disconnect-PnPOnline

Usage Notes

Update the site URL to match your SharePoint environment
Ensure you have appropriate search permissions
Crawl logs may have a delay before appearing
Use filters to focus on specific time periods or error types
Review logs regularly for search optimization insights