Efficient Sorting Algorithms

CSC 385 - Data Structures and Algorithms

Brian-Thomas Rogers

broge2@uis.edu

University of Illinois Springfield

College of Health, Science, and Technology

Contents

• Divide and Conquer
• Merge Sort
• Quicksort
• Radix Sort
• Analysis Table for All Sorts
• Which to Choose?
• DEFHIINS

Divide and Conquer

Divide and Conquer

• Suppose we have an array of 16 elements
• We need to sort the array using one of the sorts we learned
  • Selection or Insertion \(O(n^2)\)
• Max number of comparisons becomes \(16^2 = 256\)
• This isn’t bad but can we do better

Divide and Conquer

• Idea: sorting smaller arrays requires less time than sorting larger arrays
• Suppose we split the 16 element array into two 8 element arrays and sorted the smaller arrays then combined the sorted arrays
  • Max number of comparisons becomes \(S(n) = (\frac{n}{2})^2 + (\frac{n}{2})^2 + n\) \[ \begin{align*} S(16) &= (\frac{16}{2})^2 + (\frac{16}{2})^2 + 16 \\ &= 8^2 + 8^2 + 16 \\ &= 64 + 64 + 16 = 144 \end{align*} \]
• We get a value less than 256

Divide and Conquer

• If we continue down this line we can divide the two 8 element arrays into four 4 element arrays and have two combines

\[ \begin{align*} S(16) &= (\frac{n}{4})^2 + (\frac{n}{4})^2 + (\frac{n}{4})^2 + (\frac{n}{4})^2 + 2n \\ &= 4(\frac{n}{4})^2 + 2n \\ &= 4(\frac{16}{4})^2 + 2(16) \\ &= 4(4^2) + 32 \\ &= 64 + 32 \\ &= 96 \end{align*} \]

• A value less than 144

Divide and Conquer

• You can keep doing this until you have divided the array to subarrays of 1 element leaving only the combines left
  • \(16(1^2) + 4(16) = 80\)
• This process of splitting the problem, solving small versions of the problem, and recombining the smaller portions to solve the larger problem is a common algorithmic tactic known as Divide and Conquer
• Two of the three algorithms discussed are Divide and Conquer algorithms

Merge Sort

Algorithm

• MergeSort
  1. Recursively Divide array by half until subarrays of size 1 are reached
  2. Merge the halves
• Merging Halves
  1. Create empty temporary array
  2. Process each subarray half from left to right
     1. If left <= right: add left element to temp
     2. else move right element to temp
  3. After reaching the end move the remaining elements from either the left or right subarray into temp
  4. Place elements from temp back into main array
• The process of merging sorts the subarrays

Example

Dividing and merging

Notes on Merge Sort

• Recursive mergesort method does not do any sorting but simply divides the array
• Sorting is done by the merge
• The recursion operates on near equal-sized arrays
  • If the array size is odd then one of the subarrays will have 1 extra element
• The merged subarrays is stored in a temporary which means we require additional storage
• Merge Sort is Stable

Analysis

• Recurrence Relation

\[ \begin{align*} T(1) &= 0 \\ T(n) &= 2T(n/2) + n \end{align*} \]

• Base Case \(T(1) = 0\) is the number of comparisons needed for 1 element which is 0
• \(2T(n/2)\) is the number of recursive calls: one for the left half of the subarray and one for the second half
• \(+ n\) is the number of comparisons needed for the merges
• The BigO is \(O(n log n)\)
• Proof: Merge Sort Analysis

Practice Question

• Given the following two subarrays merge them based on the algorithm

a1 = [14 19 20 27] a2 = [8 10 15 25]

a1	a2	Temp
[14 19 20 27]	[8 10 15 25]	[]
[-14- 19 20 27]	[-8- 10 15 25]	[8]
[-14- 19 20 27]	[8 -10- 15 25]	[8 10]
[-14- 19 20 27]	[8 10 -15- 25]	[8 10 14]
[14 -19- 20 27]	[8 10 -15- 25]	[8 10 14 15]
[14 -19- 20 27]	[8 10 15 -25-]	[8 10 14 15 19]
[14 19 -20- 27]	[8 10 15 -25-]	[8 10 14 15 19 20]
[14 19 20 -27-]	[8 10 15 -25-]	[8 10 14 15 19 20 25]

• Copy rest of a1 into Temp for [8 10 14 15 19 20 25 27]

Quicksort

Algorithm

1. If subarray length is less than 10, use insertion sort
2. Choose a pivot that will be used to partition the elements
3. Move pivot element out of the way by placing it in the next to last position in the array
4. Partition the array such that all elements less than the pivot are to the left of the pivot and all elements >= the pivot are to the right of the pivot
5. Recursively quicksort left partition of the array
6. Recursively quicksort right partition of the array

Why call Insertion Sort?

• During the recursion the subarray that gets sorted becomes increasingly smaller due to repeated partitioning
• Once the array reaches a certain threshold size, quicksort’s efficiency is compromised due to the overhead incurred by the recursive calls
• To avoid this problem, when the subarray reaches the threshold size, it is more efficient to use an algorithm such as insertion sort
• In practice, this threshold is typically in the range of 10 to 20

How to choose a pivot

• The pivot step is crucial to achieving optimum performance
• Ideally pivot is the median of the array so the array would be divided into 2 equal sized subarrays
  • It is too costly to find the median
• It is not a good idea to randomly pick a pivot
  • The pivot might end up being the largest or smallest element of the subarray
  • This would cause one subarray to be empty while the other has the rest of the elements
• So what do we do?

Median of three

• We pick the median of three elements: First, Middle, and Last
• Suppose we have the array [10 1 6 3 4 9 7 8 0]
• We want the median of the first, middle, and last element
  • [10 4 0]
  • The median is 4
  • The partitions become [1 3 0] and [10 6 9 7 8]
  • The two partitions will be recursively sorted with new medians for each

Analysis

• Best Case Scenario
  • Every choice of pivot causes equal sized arrays so the recursive calls halve the array
• Recurrence Relation

\[ \begin{align*} T(1) &= 0 \\ T(n) &= 2T(\frac{n}{2}) + n \end{align*} \]

• We know this to be \(O(n log n)\)

Analysis

• Worst Case Scenario
  • If the division is unequal (bad pivots are picked) then quicksort takes longer
  • In the absolute worst case the array will always be divided into an empty subarray and a subarray of n-1 elements (pivot is not included)
• Recurrence Relation in Worst Case

\[ \begin{align*} T(1) &= 0 \\ T(n) &= T(n - 1) + T(0) + n \\ &= T(n - 1) + n \end{align*} \]

• \(T(0)\) is for the empty subarray so nothing happens
• This recurrence relation resolves to \(O(\frac{n (n + 1)}{2}) = O(n^2)\)

Quicksort Notes

• The average case scenario is \(O(n log n)\)
• Merge sort is always \(O(n log n)\) regardless of order of items in the array
• Even though Quicksort is also of \(O(n log n)\) in the average case in practice it is faster than Merge sort and unlike Merge sort it does not need additional memory
• Also, even though the worst case scenario for Quicksort is \(O(n^2)\) it is rare and the average case, with a good pivot choice, will be dominant
• The Quicksort is unstable.

Practice Question

• What would be the result after the first partitioning of the following array using a quicksort?

a = [0 3 7 8 2 1 4]

Answer

1. First, sort the elements of first, middle, and last positions which are 0, 8, and 4 and get the resulting array [0 3 7 4 2 1 8], pivot is 4
2. Move the pivot to the one before last position and start partitioning:
   [0 3 7 1 2 4 8], pivot is 4
3. Partition array
   • a = [0 3 7 1 2 4 8]; leftindex = 2; rightindex = 4, 7 > 4 and 2 < 4 so swap and increment leftindex and decrement rightindex
   • a = [0 3 2 1 7 4 8]; leftindex = 3; rightindex = 3, 1 < 4 so increment leftindex
   • a = [0 3 2 1 7 4 8]; leftindex = 4; rightindex = 3, leftindex > rightindex so we swap the pivot with the element at leftindex
   • a = [0 3 2 1 4 7 8]; leftindex = 4; rightindex = 3, pivot is in its properly sorted location ending the first partition

Radix Sort

Radix Sort

• Radix sort breaks the \(O(n log n)\) sorting barrier by sorting without comparisons
• The sort assumes that the data being sorted is of type integer or string where each character or digit can be extracted one at a time
• The sort uses what are called buckets which is just an array of lists
• The number of buckets is equal to the number of different digits or characters that are possible

Example Step 1

• Usorted array: [123 398 210 19 528 3 513 129 220 294]
• Place digits in buckets based ones digit

digit	bucket
0	[210 220]
1	[]
2	[]
3	[123 3 513]
4	[294]
5	[]
6	[]
7	[]
8	[398 528]
9	[19 129]

Example Step 2

• Distribute the elements from the buckets going top-to-bottom left-to-right back into the array

digit	bucket
0	[210 220]
1	[]
2	[]
3	[123 3 513]
4	[294]
5	[]
6	[]
7	[]
8	[398 528]
9	[19 129]

• Array after distribution: [210 220 123 3 513 294 398 528 19 129]

Example Step 3

• Array: [210 220 123 3 513 294 398 528 19 129]
• Place into buckets again only do 10’s digit now

digit	bucket
0	[3]
1	[210 513 19]
2	[220 123 528 129]
3	[]
4	[]
5	[]
6	[]
7	[]
8	[]
9	[294 398]

Example Step 4

• Distribute the elements from the buckets going top-to-bottom left-to-right back into the array

digit	bucket
0	[3]
1	[210 513 19]
2	[220 123 528 129]
3	[]
4	[]
5	[]
6	[]
7	[]
8	[]
9	[294 398]

• Array after distribution: [3 210 513 19 220 123 528 129 294 398]

Example Step 5

• Array: [3 210 513 19 220 123 528 129 294 398]
• Place into buckets again only do 100’s digit now

digit	bucket
0	[3 19]
1	[123 129]
2	[210 220 294]
3	[398]
4	[]
5	[513 528]
6	[]
7	[]
8	[]
9	[]

Example Step 6

• Final step
• Distribute the elements from the buckets going top-to-bottom left-to-right back into the array

digit	bucket
0	[3 19]
1	[123 129]
2	[210 220 294]
3	[398]
4	[]
5	[513 528]
6	[]
7	[]
8	[]
9	[]

• Sorted array: [3 19 123 129 210 220 294 398 513 528]

Algorithm

radix sort(collection) {
  maxdigits = maximum number of digits in collection
  digitPosition = 1
  buckets = array of lists of size 10
  for i in 1 to maxdigits {
    for index in 0 to length(collection) - 1 {
      digit = (collection[index] / digitPosition) % 10
      buckets[digit].add(collection[index])
    }
    read buckets back into collection
    clear buckets
    digitPosition *= 10
  }
}

Analysis of Radix Sort

• Radix sort goes over each integer for each digit until maximum number of digits has been reached
• This means we have \(O(n)\) for size of array times \(O(d)\) for maximum digit
• This means radix sort is \(O(d * n)\)
• If the number of digits is fixed we would have \(O(n)\)
• If the size of the items are very small \(d \lt n\) then the outer loop execution becomes neglible
• If the size of the items are very large, think million character strings, then the outer loop dominates the runtime
• While radix sort has best time efficiency it is dependent on the data and may not always be applicable
• Radix sort is stable

Notes on Radix Sort

• The example provided only has positive integer
• Try the same approach with negative numbers and notice something is not quite right
  • You need to reverse the read out operation to go bottom-to-top, left-to-right
• For strings, if you plan on sorting all possible characters that will require 65,536 buckets for unicode characters
• The array should utilize a proper data structure so the process of retrival will cause the lists to be empty at without the need to manually empty them afterward

Analysis Table for All Sorts

Algorithm	Average Case	Best Case	Worst Case
Radix Sort	\(O(n)\)	\(O(n)\)	\(O(n)\)
Merge Sort	\(O(n log n)\)	\(O(n log n)\)	\(O(n log n)\)
Quick Sort	\(O(n log n)\)	\(O(n log n)\)	\(O(n^2)\)
Shell sort	\(O(n^{1.5})\)	\(O(n)\)	\(O(n^2)\) or \(O(n^{1.5})\)
Insertion Sort	\(O(n^2)\)	\(O(n)\)	\(O(n^2)\)
Selection Sort	\(O(n^2)\)	\(O(n^2)\)	\(O(n^2)\)

Which to Choose?

• If the size of the array is small then the quadratic algorithms work fine
• If the size of the array is large
  • If data can fit in memory then use quicksort
  • If external storage is needed use mergesort
• If the type of data is integer or string use radix sort with fixed size \(d\) value

Visualizations

• Toptal Sorting Algorithms
• VisuAlgo Sorting Visualizations

DEFHIINS