How Do You Reduce Azure Virtual Machine Scale Set Costs?

VMSS Cost Optimization Priority Flowchart
==========================================

[Start] Assess VMSS Fleet
  |
  +-- Step 1: Are there scale sets with min = max instance count?
  |     |
  |     +-- YES --> Fix autoscale: set distinct min/max based on demand
  |     +-- NO  --> Proceed
  |
  +-- Step 2: Are SKUs right-sized? (Check Advisor telemetry)
  |     |
  |     +-- CPU avg < 20% --> Downsize SKU or adopt B-series burstable
  |     +-- CPU avg 20-60% --> Validate sizing is appropriate
  |     +-- CPU avg > 60% --> Sizing is likely correct
  |
  +-- Step 3: Can workloads tolerate interruption?
  |     |
  |     +-- YES --> Adopt Spot instances with Priority Mix (Flexible only)
  |     +-- NO  --> Use On-Demand with RI/Savings Plan
  |
  +-- Step 4: Are there CI/CD VMSS agent pools?
  |     |
  |     +-- YES --> Enable auto-teardown, tune standby count
  |     +-- NO  --> Proceed
  |
  +-- Step 5: Is this a Windows Server workload?
  |     |
  |     +-- YES --> Apply Azure Hybrid Benefit
  |     +-- NO  --> Proceed
  |
  +-- Step 6: Is demand predictable?
  |     |
  |     +-- YES --> Add schedule-based autoscale profiles
  |     +-- NO  --> Tune metric-based autoscale thresholds
  |
  +-- Step 7: Are there orphaned/empty scale sets?
        |
        +-- YES --> Remove or deallocate (check associated LB, PIP)
        +-- NO  --> Optimization complete

# Check current VMSS SKU
az vmss show --name <vmss-name> --resource-group <rg-name> \
  --query "sku" -o table

# List Advisor cost recommendations for VMSS
az advisor recommendation list \
  --filter "Category eq 'Cost'" \
  --query "[?resourceMetadata.resourceId contains 'virtualMachineScaleSets']" \
  -o table

# Get detailed CPU metrics for right-sizing analysis
az monitor metrics list \
  --resource <vmss-resource-id> \
  --metric "Percentage CPU" \
  --interval PT1H \
  --start-time $(date -u -d '7 days ago' +%Y-%m-%dT%H:%M:%SZ) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \
  --aggregation Average Maximum \
  -o table

Spot Priority Mix Configuration
================================

Total desired capacity: 10 instances

  [Standard] [Standard] [Standard] [Standard] [  Spot  ] [  Spot  ] [  Spot  ] [  Spot  ] [  Spot  ] [  Spot  ]
  Instance 1  Instance 2  Instance 3  Instance 4  Instance 5  Instance 6  Instance 7  Instance 8  Instance 9  Instance 10

  |<-------- 4 Standard (baseline) -------->|<------------- 6 Spot (burst capacity) ------------->|

Configuration:
  baseRegularPriorityCount: 4     # Always on-demand
  regularPriorityPercentageAboveBase: 0  # All additional = Spot

Result:
  - 4 instances guaranteed (no eviction)
  - 6 instances at ~90% discount (evictable)
  - Blended savings: ~54% vs all on-demand

# Poll IMDS for Spot eviction notice (run inside the VM)
# Returns empty if no eviction scheduled, or eviction details

curl -s -H "Metadata:true" \
  "http://169.254.169.254/metadata/scheduledevents?api-version=2020-07-01" \
  | jq '.Events[] | select(.EventType == "Preempt")'

# Recommended: poll every 5 seconds in a background service
# and gracefully drain workloads when eviction is detected

# Create autoscale setting with CPU-based scale-out and scale-in
az monitor autoscale create \
  --resource-group <rg-name> \
  --resource <vmss-resource-id> \
  --resource-type Microsoft.Compute/virtualMachineScaleSets \
  --name "autoscale-cpu" \
  --min-count 2 \
  --max-count 10 \
  --count 2

# Add scale-out rule: add 2 instances when avg CPU > 70% for 10 min
az monitor autoscale rule create \
  --resource-group <rg-name> \
  --autoscale-name "autoscale-cpu" \
  --condition "Percentage CPU > 70 avg 10m" \
  --scale out 2

# Add scale-in rule: remove 1 instance when avg CPU < 30% for 10 min
az monitor autoscale rule create \
  --resource-group <rg-name> \
  --autoscale-name "autoscale-cpu" \
  --condition "Percentage CPU < 30 avg 10m" \
  --scale in 1

# Add cooldown period to prevent flapping
az monitor autoscale rule create \
  --resource-group <rg-name> \
  --autoscale-name "autoscale-cpu" \
  --condition "Percentage CPU > 70 avg 10m" \
  --scale out 2 \
  --cooldown 5

# Check current autoscale settings for a VMSS
az monitor autoscale show \
  --resource-group <rg-name> \
  --name <autoscale-setting-name> \
  --query "{MinCount:profiles[0].capacity.minimum, \
    MaxCount:profiles[0].capacity.maximum, \
    DefaultCount:profiles[0].capacity.default}" -o table

# Fix: set distinct min and max
az monitor autoscale update \
  --resource-group <rg-name> \
  --name <autoscale-setting-name> \
  --min-count 2 \
  --max-count 10 \
  --count 2

# Find all VMSS with min=max across the subscription
az monitor autoscale list --query "[?profiles[0].capacity.minimum == \
  profiles[0].capacity.maximum].{Name:name, Min:profiles[0].capacity.minimum, \
  Max:profiles[0].capacity.maximum, Resource:targetResourceUri}" -o table

Autoscale Tuning Decision Tree
===============================

[Start] Review current autoscale metrics
  |
  +-- Is the scale-out threshold too low (e.g., 50% CPU)?
  |     |
  |     +-- YES --> Raise to 70-80% to avoid premature scaling
  |     +-- NO  --> Is the threshold too high (e.g., 95%)?
  |           |
  |           +-- YES --> Lower to 70-80% to avoid latency spikes
  |           +-- NO  --> Threshold is appropriate
  |
  +-- Is the scale-in threshold too high (e.g., 50% CPU)?
  |     |
  |     +-- YES --> Lower to 20-30% to scale in more aggressively
  |     +-- NO  --> Is the threshold too low (e.g., 5%)?
  |           |
  |           +-- YES --> Raise to 20-30% to avoid running idle instances
  |           +-- NO  --> Threshold is appropriate
  |
  +-- Is the cooldown period too short (< 5 min)?
  |     |
  |     +-- YES --> Increase to 5-10 min to prevent flapping
  |     +-- NO  --> Is it too long (> 15 min)?
  |           |
  |           +-- YES --> Reduce to 5-10 min for better responsiveness
  |           +-- NO  --> Cooldown is appropriate
  |
  +-- Is the metric time grain too short (< 5 min)?
        |
        +-- YES --> Increase to 5-10 min to smooth out spikes
        +-- NO  --> Time grain is appropriate

# Disable autoscale and set fixed capacity for test VMSS
az monitor autoscale update \
  --resource-group <rg-name> \
  --name <autoscale-setting-name> \
  --enabled false

# Set manual capacity to 1 instance
az vmss scale \
  --resource-group <rg-name> \
  --name <vmss-name> \
  --new-capacity 1

Weekly VMSS Capacity Schedule
==============================

         Mon    Tue    Wed    Thu    Fri    Sat    Sun
0:00     2      2      2      2      2      1      1
6:00     2      2      2      2      2      1      1
8:00     6      6      6      6      6      1      1    <-- Business hours start
12:00    8      8      8      8      6      1      1    <-- Peak
18:00    4      4      4      4      2      1      1    <-- Wind down
22:00    2      2      2      2      2      1      1    <-- Night

Savings vs fixed 8 instances:
  Fixed:    8 x 24 x 7 = 1,344 instance-hours/week
  Scheduled:                 ~420 instance-hours/week
  Reduction: ~69%

# Add a recurrence profile for business hours (weekdays 8am-6pm)
az monitor autoscale profile create \
  --resource-group <rg-name> \
  --autoscale-name "autoscale-cpu" \
  --name "business-hours" \
  --min-count 4 \
  --max-count 10 \
  --count 6 \
  --recurrence week Mon Tue Wed Thu Fri \
  --start "08:00" \
  --end "18:00" \
  --timezone "Eastern Standard Time"

# Add a weekend/night profile with minimal capacity
az monitor autoscale profile create \
  --resource-group <rg-name> \
  --autoscale-name "autoscale-cpu" \
  --name "off-hours" \
  --min-count 1 \
  --max-count 3 \
  --count 2 \
  --recurrence week Mon Tue Wed Thu Fri Sat Sun \
  --start "18:00" \
  --end "08:00" \
  --timezone "Eastern Standard Time"

# Enable Azure Hybrid Benefit on an existing VMSS
az vmss update \
  --resource-group <rg-name> \
  --name <vmss-name> \
  --license-type Windows_Server

# Verify AHB is enabled
az vmss show \
  --resource-group <rg-name> \
  --name <vmss-name> \
  --query "virtualMachineProfile.licenseType" -o tsv

# Check all VMSS in a subscription for missing AHB
az vmss list --query "[?virtualMachineProfile.osProfile.windowsConfiguration != null \
  && virtualMachineProfile.licenseType != 'Windows_Server'].{Name:name, \
  ResourceGroup:resourceGroup, LicenseType:virtualMachineProfile.licenseType}" -o table

# Deallocate a VMSS (stops all instances, retains configuration)
az vmss deallocate \
  --resource-group <rg-name> \
  --name <vmss-name>

# Deallocate specific instances within a VMSS
az vmss deallocate \
  --resource-group <rg-name> \
  --name <vmss-name> \
  --instance-ids 0 1 2

# Find VMSS with zero instances
az vmss list --query "[?sku.capacity == `0`].{Name:name, \
  ResourceGroup:resourceGroup, SKU:sku.name, Capacity:sku.capacity}" -o table

# Before deleting, check for associated resources
az vmss show --name <vmss-name> --resource-group <rg-name> \
  --query "{LoadBalancer:virtualMachineProfile.networkProfile.networkInterfaceConfigurations[0].\
  ipConfigurations[0].loadBalancerBackendAddressPools, \
  PublicIP:virtualMachineProfile.networkProfile.networkInterfaceConfigurations[0].\
  ipConfigurations[0].publicIPAddressConfiguration}" -o json

# Delete an empty VMSS and its associated resources
az vmss delete --resource-group <rg-name> --name <vmss-name>

# Weekly VMSS Fleet Health Check - PowerShell Automation Script
# Schedule: Weekly via Azure Automation or GitHub Actions

param(
    [string[]]$SubscriptionIds,
    [int]$CpuThresholdPercent = 20,
    [int]$LookbackDays = 7
)

$report = @()

foreach ($subId in $SubscriptionIds) {
    Set-AzContext -SubscriptionId $subId | Out-Null

    $vmssList = Get-AzVmss

    foreach ($vmss in $vmssList) {
        $rgName = $vmss.ResourceGroupName
        $vmssName = $vmss.Name

        # Check 1: Autoscale configuration
        $autoscale = Get-AzAutoscaleSetting -ResourceGroupName $rgName \
            -ErrorAction SilentlyContinue |
            Where-Object { $_.TargetResourceUri -match $vmssName }

        $minMaxIssue = $false
        if ($autoscale) {
            $profile = $autoscale.Profiles | Select-Object -First 1
            if ($profile.Capacity.Minimum -eq $profile.Capacity.Maximum) {
                $minMaxIssue = $true
            }
        }

        # Check 2: Average CPU utilization
        $endTime = Get-Date
        $startTime = $endTime.AddDays(-$LookbackDays)
        $metrics = Get-AzMetric -ResourceId $vmss.Id \
            -MetricName "Percentage CPU" \
            -StartTime $startTime -EndTime $endTime \
            -TimeGrain 01:00:00 -AggregationType Average \
            -ErrorAction SilentlyContinue

        $avgCpu = ($metrics.Data | Where-Object { $_.Average } |
            Measure-Object -Property Average -Average).Average

        # Check 3: Azure Hybrid Benefit
        $ahbEnabled = $vmss.VirtualMachineProfile.LicenseType -eq "Windows_Server"
        $isWindows = $vmss.VirtualMachineProfile.OsProfile.WindowsConfiguration -ne $null

        # Check 4: Instance count
        $capacity = $vmss.Sku.Capacity
        $isEmpty = $capacity -eq 0

        # Build report entry
        $report += [PSCustomObject]@{
            Subscription    = $subId
            ResourceGroup   = $rgName
            VMSSName        = $vmssName
            SKU             = $vmss.Sku.Name
            Capacity        = $capacity
            AvgCpuPercent   = [math]::Round($avgCpu, 1)
            MinEqualsMax    = $minMaxIssue
            IsWindows       = $isWindows
            AHBEnabled      = $ahbEnabled
            AHBMissing      = ($isWindows -and -not $ahbEnabled)
            IsEmpty         = $isEmpty
            Recommendation  = ""
        }
    }
}

# Generate recommendations
foreach ($entry in $report) {
    $recs = @()

    if ($entry.MinEqualsMax) {
        $recs += "FIX: Autoscale min=max override detected"
    }
    if ($entry.AvgCpuPercent -lt $CpuThresholdPercent -and $entry.Capacity -gt 0) {
        $recs += "RIGHT-SIZE: Avg CPU $($entry.AvgCpuPercent)% - consider smaller SKU"
    }
    if ($entry.AHBMissing) {
        $recs += "ENABLE: Azure Hybrid Benefit missing on Windows VMSS"
    }
    if ($entry.IsEmpty) {
        $recs += "CLEANUP: Empty scale set - check for orphaned resources"
    }

    $entry.Recommendation = ($recs -join "; ")
}

# Output report
$report | Format-Table -AutoSize
$report | Export-Csv -Path "vmss-fleet-health-$(Get-Date -Format yyyy-MM-dd).csv" \
    -NoTypeInformation

# Summary
Write-Output "\n=== VMSS Fleet Health Summary ==="
Write-Output "Total VMSS: $($report.Count)"
Write-Output "Min=Max Issues: $(($report | Where-Object MinEqualsMax).Count)"
Write-Output "Under-utilized (<$CpuThresholdPercent% CPU): $(($report | \
    Where-Object { $_.AvgCpuPercent -lt $CpuThresholdPercent -and \
    $_.Capacity -gt 0 }).Count)"
Write-Output "Missing AHB: $(($report | Where-Object AHBMissing).Count)"
Write-Output "Empty VMSS: $(($report | Where-Object IsEmpty).Count)"