Learning ObjectivesBy the end of this chapter, you will be able to:
- Apply computer vision techniques to football video analysis
- Detect and track players and the ball in video footage
- Recognize formations and plays from video data
- Extract spatial and tracking data from video sources
- Build automated scouting and analysis tools
Introduction
The emergence of computer vision has revolutionized football analytics by enabling the extraction of rich spatial data from video sources. While technologies like RFID chips and optical tracking systems provide high-precision tracking data, computer vision offers a more accessible and flexible approach that can be applied to any video footage—from broadcast games to practice film.
This chapter explores the complete pipeline of video analysis in football, from basic image processing to sophisticated deep learning models that can automatically detect players, track movements, recognize formations, and classify plays. These techniques are increasingly critical for teams seeking competitive advantages through comprehensive video analysis.
Computer Vision in Football
Computer vision enables machines to interpret and understand visual information from football video. Applications range from player detection and tracking to automated play classification and biomechanical analysis—tasks that would require hundreds of hours of manual work.The Computer Vision Pipeline for Football
Overview of the Analysis Workflow
A typical computer vision pipeline for football analysis consists of several stages:
- Video Acquisition and Preprocessing: Loading video, frame extraction, resolution normalization
- Field Detection: Identifying the playing field and establishing coordinate systems
- Object Detection: Locating players, the ball, and other objects of interest
- Tracking: Following objects across frames to create trajectories
- Feature Extraction: Computing higher-level features like formations and spacing
- Action Recognition: Classifying plays, formations, and player actions
- Data Integration: Combining video-derived data with other sources
Challenges in Football Video Analysis
Football presents unique challenges for computer vision:
- Occlusion: Players frequently block each other from view
- Camera Movement: Broadcast cameras pan, zoom, and change angles
- Player Similarity: Uniform colors and body shapes make individual identification difficult
- Fast Motion: Quick movements can cause motion blur
- Crowding: 22 players in close proximity create detection challenges
- Varying Conditions: Weather, lighting, and field conditions affect image quality
Image Processing Fundamentals
Loading and Processing Video
#| eval: false
#| echo: true
# Install required packages
install.packages("magick")
install.packages("opencv")
install.packages("reticulate")
#| message: false
#| warning: false
library(magick)
library(tidyverse)
library(reticulate)
# Load a single frame from video
load_frame <- function(video_path, frame_number = 1) {
# Read video and extract frame
vid <- image_read_video(video_path, fps = 30)
frame <- vid[frame_number]
return(frame)
}
# Example: Load and display frame
# frame <- load_frame("game_footage.mp4", frame_number = 100)
# print(frame)
cat("✓ Image processing packages loaded\n")
#| message: false
#| warning: false
import cv2
import numpy as np
import matplotlib.pyplot as plt
from PIL import Image
def load_frame(video_path, frame_number=1):
"""Load a specific frame from video"""
cap = cv2.VideoCapture(video_path)
cap.set(cv2.CAP_PROP_POS_FRAMES, frame_number)
ret, frame = cap.read()
cap.release()
if ret:
# Convert BGR to RGB for display
frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
return frame
return None
def preprocess_frame(frame, target_size=(1280, 720)):
"""Preprocess frame for analysis"""
# Resize to standard dimensions
resized = cv2.resize(frame, target_size)
# Normalize pixel values
normalized = resized / 255.0
return normalized
print("✓ Computer vision packages loaded")
print(f"✓ OpenCV version: {cv2.__version__}")
Basic Image Operations
#| eval: false
#| label: image-operations-r
#| message: false
#| warning: false
# Using magick for image processing
process_football_frame <- function(frame) {
# Convert to different color spaces
frame_gray <- image_convert(frame, colorspace = "Gray")
# Apply edge detection
frame_edges <- image_canny(frame)
# Enhance contrast
frame_enhanced <- image_contrast(frame, sharpen = 1)
# Apply Gaussian blur
frame_blurred <- image_blur(frame, radius = 5, sigma = 2)
return(list(
original = frame,
grayscale = frame_gray,
edges = frame_edges,
enhanced = frame_enhanced,
blurred = frame_blurred
))
}
# Detect field boundaries using color thresholding
detect_field <- function(frame) {
# Convert to HSV color space
frame_hsv <- image_convert(frame, colorspace = "HSV")
# Threshold for green field (will vary by field condition)
# This is a simplified example
field_mask <- image_threshold(frame_hsv, type = "color")
return(field_mask)
}
#| label: image-operations-py
#| message: false
#| warning: false
def apply_image_operations(frame):
"""Demonstrate basic image processing operations"""
# Convert to grayscale
gray = cv2.cvtColor(frame, cv2.COLOR_RGB2GRAY)
# Edge detection using Canny
edges = cv2.Canny(gray, threshold1=50, threshold2=150)
# Gaussian blur
blurred = cv2.GaussianBlur(frame, (5, 5), 0)
# Histogram equalization for contrast enhancement
gray_eq = cv2.equalizeHist(gray)
return {
'original': frame,
'grayscale': gray,
'edges': edges,
'blurred': blurred,
'equalized': gray_eq
}
def detect_field_hsv(frame):
"""Detect football field using HSV color space"""
# Convert to HSV
hsv = cv2.cvtColor(frame, cv2.COLOR_RGB2HSV)
# Define range for green field (adjust based on lighting)
lower_green = np.array([35, 40, 40])
upper_green = np.array([85, 255, 255])
# Create mask
field_mask = cv2.inRange(hsv, lower_green, upper_green)
# Apply morphological operations to clean up mask
kernel = np.ones((5, 5), np.uint8)
field_mask = cv2.morphologyEx(field_mask, cv2.MORPH_CLOSE, kernel)
field_mask = cv2.morphologyEx(field_mask, cv2.MORPH_OPEN, kernel)
return field_mask
# Example usage (with dummy data)
dummy_frame = np.random.randint(0, 255, (720, 1280, 3), dtype=np.uint8)
results = apply_image_operations(dummy_frame)
print("✓ Image operations completed")
print(f" - Original shape: {results['original'].shape}")
print(f" - Grayscale shape: {results['grayscale'].shape}")
print(f" - Edges detected: {np.sum(results['edges'] > 0)} pixels")
Object Detection: Players and Ball
YOLO (You Only Look Once) for Player Detection
YOLO is one of the most popular real-time object detection frameworks, ideal for detecting players and the ball in football video.
#| eval: false
#| echo: true
# Install YOLOv8 via ultralytics
pip install ultralytics torch torchvision
#| label: yolo-detection
#| message: false
#| warning: false
#| eval: false
from ultralytics import YOLO
import torch
class FootballPlayerDetector:
"""Detect players and ball in football footage"""
def __init__(self, model_path='yolov8n.pt'):
"""Initialize YOLO model"""
self.model = YOLO(model_path)
self.device = 'cuda' if torch.cuda.is_available() else 'cpu'
def detect_players(self, frame, conf_threshold=0.5):
"""
Detect players in a single frame
Parameters:
-----------
frame : numpy.ndarray
Input frame (RGB)
conf_threshold : float
Confidence threshold for detections
Returns:
--------
detections : list of dict
Each dict contains: bbox, confidence, class_id, class_name
"""
# Run inference
results = self.model(frame, conf=conf_threshold, device=self.device)
detections = []
for result in results:
boxes = result.boxes
for box in boxes:
# Extract bounding box coordinates
x1, y1, x2, y2 = box.xyxy[0].cpu().numpy()
confidence = box.conf[0].cpu().numpy()
class_id = int(box.cls[0].cpu().numpy())
class_name = result.names[class_id]
# Filter for person class (players)
if class_name == 'person':
detections.append({
'bbox': [x1, y1, x2, y2],
'confidence': float(confidence),
'class_id': class_id,
'class_name': class_name
})
return detections
def annotate_frame(self, frame, detections):
"""Draw bounding boxes on frame"""
annotated = frame.copy()
for det in detections:
x1, y1, x2, y2 = [int(coord) for coord in det['bbox']]
confidence = det['confidence']
# Draw bounding box
cv2.rectangle(annotated, (x1, y1), (x2, y2), (0, 255, 0), 2)
# Add label
label = f"Player {confidence:.2f}"
cv2.putText(annotated, label, (x1, y1 - 10),
cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 2)
return annotated
# Example usage
detector = FootballPlayerDetector()
# Process single frame
# detections = detector.detect_players(frame)
# annotated_frame = detector.annotate_frame(frame, detections)
print("✓ Player detection system initialized")
Custom Detection Model for Football
For better performance on football-specific tasks, fine-tune YOLO on football data:
#| label: custom-football-detector
#| eval: false
#| message: false
#| warning: false
class FootballObjectDetector:
"""
Custom detector for football-specific objects
Classes: player_offense, player_defense, referee, ball, goalpost
"""
def __init__(self, model_path='football_yolov8.pt'):
self.model = YOLO(model_path)
self.classes = {
0: 'player_offense',
1: 'player_defense',
2: 'referee',
3: 'ball',
4: 'goalpost'
}
def detect_all_objects(self, frame, conf_threshold=0.5):
"""Detect all football objects"""
results = self.model(frame, conf=conf_threshold)
detections = {
'players_offense': [],
'players_defense': [],
'referees': [],
'balls': [],
'goalposts': []
}
for result in results:
boxes = result.boxes
for box in boxes:
x1, y1, x2, y2 = box.xyxy[0].cpu().numpy()
confidence = float(box.conf[0].cpu().numpy())
class_id = int(box.cls[0].cpu().numpy())
class_name = self.classes[class_id]
detection = {
'bbox': [float(x1), float(y1), float(x2), float(y2)],
'confidence': confidence,
'center': [float((x1 + x2) / 2), float((y1 + y2) / 2)]
}
if class_name == 'player_offense':
detections['players_offense'].append(detection)
elif class_name == 'player_defense':
detections['players_defense'].append(detection)
elif class_name == 'referee':
detections['referees'].append(detection)
elif class_name == 'ball':
detections['balls'].append(detection)
elif class_name == 'goalpost':
detections['goalposts'].append(detection)
return detections
def train_custom_model(self, data_yaml, epochs=100, imgsz=640):
"""
Fine-tune YOLO on custom football dataset
Parameters:
-----------
data_yaml : str
Path to dataset configuration YAML
epochs : int
Number of training epochs
imgsz : int
Image size for training
"""
model = YOLO('yolov8n.pt') # Start with pretrained model
# Train model
results = model.train(
data=data_yaml,
epochs=epochs,
imgsz=imgsz,
batch=16,
name='football_detector',
patience=20,
save=True,
device='cuda' if torch.cuda.is_available() else 'cpu'
)
return results
print("✓ Custom football detector defined")
Multi-Object Tracking
Tracking Players Across Frames
Once players are detected, tracking maintains their identities across frames:
#| eval: false
#| echo: true
# Install tracking libraries
pip install filterpy scipy lap
#| label: sort-tracker
#| message: false
#| warning: false
#| eval: false
from filterpy.kalman import KalmanFilter
from scipy.optimize import linear_sum_assignment
import numpy as np
class KalmanBoxTracker:
"""Kalman Filter for tracking bounding boxes"""
count = 0
def __init__(self, bbox):
"""Initialize tracker with initial bounding box"""
# Define constant velocity model
self.kf = KalmanFilter(dim_x=7, dim_z=4)
# State transition matrix
self.kf.F = np.array([
[1, 0, 0, 0, 1, 0, 0],
[0, 1, 0, 0, 0, 1, 0],
[0, 0, 1, 0, 0, 0, 1],
[0, 0, 0, 1, 0, 0, 0],
[0, 0, 0, 0, 1, 0, 0],
[0, 0, 0, 0, 0, 1, 0],
[0, 0, 0, 0, 0, 0, 1]
])
# Measurement matrix
self.kf.H = np.array([
[1, 0, 0, 0, 0, 0, 0],
[0, 1, 0, 0, 0, 0, 0],
[0, 0, 1, 0, 0, 0, 0],
[0, 0, 0, 1, 0, 0, 0]
])
# Measurement noise
self.kf.R *= 10.0
# Process noise
self.kf.Q[-1, -1] *= 0.01
self.kf.Q[4:, 4:] *= 0.01
# Initial state
self.kf.x[:4] = self._convert_bbox_to_z(bbox)
self.time_since_update = 0
self.id = KalmanBoxTracker.count
KalmanBoxTracker.count += 1
self.history = []
self.hits = 0
self.hit_streak = 0
self.age = 0
def _convert_bbox_to_z(self, bbox):
"""Convert [x1, y1, x2, y2] to [x, y, s, r]"""
w = bbox[2] - bbox[0]
h = bbox[3] - bbox[1]
x = bbox[0] + w / 2.0
y = bbox[1] + h / 2.0
s = w * h
r = w / float(h)
return np.array([x, y, s, r]).reshape((4, 1))
def _convert_x_to_bbox(self, x):
"""Convert [x, y, s, r] to [x1, y1, x2, y2]"""
w = np.sqrt(x[2] * x[3])
h = x[2] / w
return np.array([
x[0] - w / 2.0,
x[1] - h / 2.0,
x[0] + w / 2.0,
x[1] + h / 2.0
]).reshape((1, 4))
def update(self, bbox):
"""Update tracker with new detection"""
self.time_since_update = 0
self.history = []
self.hits += 1
self.hit_streak += 1
self.kf.update(self._convert_bbox_to_z(bbox))
def predict(self):
"""Predict next state"""
if (self.kf.x[6] + self.kf.x[2]) <= 0:
self.kf.x[6] *= 0.0
self.kf.predict()
self.age += 1
if self.time_since_update > 0:
self.hit_streak = 0
self.time_since_update += 1
self.history.append(self._convert_x_to_bbox(self.kf.x))
return self.history[-1]
def get_state(self):
"""Return current bounding box estimate"""
return self._convert_x_to_bbox(self.kf.x)
class SORTTracker:
"""SORT: Simple Online and Realtime Tracking"""
def __init__(self, max_age=30, min_hits=3, iou_threshold=0.3):
"""
Parameters:
-----------
max_age : int
Maximum frames to keep track alive without detections
min_hits : int
Minimum hits before returning track
iou_threshold : float
IOU threshold for matching detections to tracks
"""
self.max_age = max_age
self.min_hits = min_hits
self.iou_threshold = iou_threshold
self.trackers = []
self.frame_count = 0
def _iou(self, bb_test, bb_gt):
"""Compute IOU between two boxes"""
xx1 = np.maximum(bb_test[0], bb_gt[0])
yy1 = np.maximum(bb_test[1], bb_gt[1])
xx2 = np.minimum(bb_test[2], bb_gt[2])
yy2 = np.minimum(bb_test[3], bb_gt[3])
w = np.maximum(0., xx2 - xx1)
h = np.maximum(0., yy2 - yy1)
wh = w * h
o = wh / ((bb_test[2] - bb_test[0]) * (bb_test[3] - bb_test[1])
+ (bb_gt[2] - bb_gt[0]) * (bb_gt[3] - bb_gt[1]) - wh)
return o
def _associate_detections_to_trackers(self, detections, trackers):
"""Match detections to existing trackers"""
if len(trackers) == 0:
return np.empty((0, 2), dtype=int), \
np.arange(len(detections)), \
np.empty((0, 5), dtype=int)
iou_matrix = np.zeros((len(detections), len(trackers)), dtype=np.float32)
for d, det in enumerate(detections):
for t, trk in enumerate(trackers):
iou_matrix[d, t] = self._iou(det, trk)
# Hungarian algorithm for optimal assignment
row_ind, col_ind = linear_sum_assignment(-iou_matrix)
matched_indices = np.column_stack((row_ind, col_ind))
unmatched_detections = []
for d in range(len(detections)):
if d not in matched_indices[:, 0]:
unmatched_detections.append(d)
unmatched_trackers = []
for t in range(len(trackers)):
if t not in matched_indices[:, 1]:
unmatched_trackers.append(t)
# Filter out matched with low IOU
matches = []
for m in matched_indices:
if iou_matrix[m[0], m[1]] < self.iou_threshold:
unmatched_detections.append(m[0])
unmatched_trackers.append(m[1])
else:
matches.append(m.reshape(1, 2))
if len(matches) == 0:
matches = np.empty((0, 2), dtype=int)
else:
matches = np.concatenate(matches, axis=0)
return matches, np.array(unmatched_detections), np.array(unmatched_trackers)
def update(self, detections):
"""
Update tracker with new detections
Parameters:
-----------
detections : np.ndarray
Array of detections in format [[x1, y1, x2, y2, score], ...]
Returns:
--------
tracks : np.ndarray
Array of active tracks [[x1, y1, x2, y2, track_id], ...]
"""
self.frame_count += 1
# Get predicted locations from existing trackers
trks = np.zeros((len(self.trackers), 5))
to_del = []
for t, trk in enumerate(trks):
pos = self.trackers[t].predict()[0]
trk[:] = [pos[0], pos[1], pos[2], pos[3], 0]
if np.any(np.isnan(pos)):
to_del.append(t)
trks = np.ma.compress_rows(np.ma.masked_invalid(trks))
for t in reversed(to_del):
self.trackers.pop(t)
# Associate detections to trackers
matched, unmatched_dets, unmatched_trks = \
self._associate_detections_to_trackers(detections, trks)
# Update matched trackers
for m in matched:
self.trackers[m[1]].update(detections[m[0], :])
# Create new trackers for unmatched detections
for i in unmatched_dets:
trk = KalmanBoxTracker(detections[i, :])
self.trackers.append(trk)
# Return current tracks
ret = []
for trk in self.trackers:
if (trk.time_since_update < 1) and \
(trk.hit_streak >= self.min_hits or self.frame_count <= self.min_hits):
d = trk.get_state()[0]
ret.append(np.concatenate((d, [trk.id])).reshape(1, -1))
if len(ret) > 0:
return np.concatenate(ret)
return np.empty((0, 5))
print("✓ SORT tracker implemented")
Complete Tracking Pipeline
#| label: tracking-pipeline
#| eval: false
#| message: false
#| warning: false
class FootballTracker:
"""Complete pipeline for detecting and tracking players"""
def __init__(self, detector_path='yolov8n.pt'):
self.detector = FootballPlayerDetector(detector_path)
self.tracker = SORTTracker(max_age=30, min_hits=3, iou_threshold=0.3)
self.track_history = {}
def process_video(self, video_path, output_path=None):
"""
Process entire video and track all players
Parameters:
-----------
video_path : str
Path to input video
output_path : str, optional
Path to save annotated video
Returns:
--------
tracking_data : dict
Complete tracking data for all frames
"""
cap = cv2.VideoCapture(video_path)
# Get video properties
fps = int(cap.get(cv2.CAP_PROP_FPS))
width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
# Setup video writer if output requested
if output_path:
fourcc = cv2.VideoWriter_fourcc(*'mp4v')
out = cv2.VideoWriter(output_path, fourcc, fps, (width, height))
tracking_data = []
frame_num = 0
while cap.isOpened():
ret, frame = cap.read()
if not ret:
break
# Convert BGR to RGB
frame_rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
# Detect players
detections = self.detector.detect_players(frame_rgb, conf_threshold=0.5)
# Convert to numpy array for tracker
det_array = np.array([[d['bbox'][0], d['bbox'][1],
d['bbox'][2], d['bbox'][3],
d['confidence']] for d in detections])
# Update tracker
if len(det_array) > 0:
tracks = self.tracker.update(det_array)
else:
tracks = self.tracker.update(np.empty((0, 5)))
# Store tracking data
frame_data = {
'frame': frame_num,
'tracks': []
}
# Draw tracks and collect data
for track in tracks:
x1, y1, x2, y2, track_id = track
track_id = int(track_id)
# Store track data
frame_data['tracks'].append({
'track_id': track_id,
'bbox': [float(x1), float(y1), float(x2), float(y2)],
'center': [float((x1 + x2) / 2), float((y1 + y2) / 2)]
})
# Update track history
if track_id not in self.track_history:
self.track_history[track_id] = []
self.track_history[track_id].append({
'frame': frame_num,
'center': [float((x1 + x2) / 2), float((y1 + y2) / 2)]
})
# Draw on frame
cv2.rectangle(frame, (int(x1), int(y1)), (int(x2), int(y2)),
(0, 255, 0), 2)
cv2.putText(frame, f"ID: {track_id}", (int(x1), int(y1) - 10),
cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 2)
# Draw trajectory
if len(self.track_history[track_id]) > 1:
points = np.array([h['center'] for h in self.track_history[track_id][-30:]])
for i in range(1, len(points)):
cv2.line(frame,
tuple(points[i-1].astype(int)),
tuple(points[i].astype(int)),
(255, 0, 0), 2)
tracking_data.append(frame_data)
# Write frame if output requested
if output_path:
out.write(frame)
frame_num += 1
# Progress update
if frame_num % 100 == 0:
print(f"Processed {frame_num}/{total_frames} frames")
cap.release()
if output_path:
out.release()
return tracking_data
# Example usage
# tracker = FootballTracker()
# tracking_data = tracker.process_video('game.mp4', 'tracked_output.mp4')
print("✓ Complete tracking pipeline ready")
Formation Recognition
Detecting Offensive and Defensive Formations
#| label: formation-recognition
#| message: false
#| warning: false
import pandas as pd
from sklearn.cluster import DBSCAN
from scipy.spatial import ConvexHull
class FormationRecognizer:
"""Recognize football formations from player positions"""
def __init__(self):
self.formation_templates = self._load_formation_templates()
def _load_formation_templates(self):
"""Define common formation templates"""
templates = {
'offense': {
'I-Formation': {
'description': '1 RB directly behind QB',
'positions': {
'OL': 5, # Offensive line
'TE': 1, # Tight end
'WR': 2, # Wide receivers
'RB': 1, # Running back
'QB': 1 # Quarterback
}
},
'Shotgun': {
'description': 'QB 5+ yards behind center',
'positions': {
'OL': 5,
'TE': 1,
'WR': 3,
'RB': 0,
'QB': 1
}
},
'Spread': {
'description': '3+ wide receivers',
'positions': {
'OL': 5,
'TE': 0,
'WR': 4,
'RB': 0,
'QB': 1
}
}
},
'defense': {
'4-3': {
'description': '4 DL, 3 LB',
'positions': {
'DL': 4, # Defensive line
'LB': 3, # Linebackers
'CB': 2, # Cornerbacks
'S': 2 # Safeties
}
},
'3-4': {
'description': '3 DL, 4 LB',
'positions': {
'DL': 3,
'LB': 4,
'CB': 2,
'S': 2
}
},
'Nickel': {
'description': '5 DBs',
'positions': {
'DL': 4,
'LB': 2,
'CB': 3,
'S': 2
}
}
}
}
return templates
def extract_spatial_features(self, player_positions):
"""
Extract spatial features from player positions
Parameters:
-----------
player_positions : list of tuples
List of (x, y) coordinates for each player
Returns:
--------
features : dict
Spatial features describing the formation
"""
if len(player_positions) < 3:
return None
positions = np.array(player_positions)
# Calculate spread metrics
x_coords = positions[:, 0]
y_coords = positions[:, 1]
features = {
'width': np.max(x_coords) - np.min(x_coords),
'depth': np.max(y_coords) - np.min(y_coords),
'center_x': np.mean(x_coords),
'center_y': np.mean(y_coords),
'density': len(positions) / ((np.max(x_coords) - np.min(x_coords)) *
(np.max(y_coords) - np.min(y_coords))),
'num_players': len(positions)
}
# Calculate convex hull area
if len(positions) >= 3:
try:
hull = ConvexHull(positions)
features['hull_area'] = hull.volume # 'volume' is area in 2D
except:
features['hull_area'] = 0
# Cluster analysis to find position groups
clustering = DBSCAN(eps=10, min_samples=2).fit(positions)
features['num_clusters'] = len(set(clustering.labels_)) - (1 if -1 in clustering.labels_ else 0)
# Calculate horizontal spread (receivers)
y_threshold = np.percentile(y_coords, 75)
wide_positions = positions[y_coords > y_threshold]
if len(wide_positions) > 0:
features['receiver_spread'] = np.max(wide_positions[:, 0]) - np.min(wide_positions[:, 0])
else:
features['receiver_spread'] = 0
return features
def classify_formation(self, player_positions, team='offense'):
"""
Classify formation based on player positions
Parameters:
-----------
player_positions : list of tuples
List of (x, y) coordinates
team : str
'offense' or 'defense'
Returns:
--------
formation : str
Predicted formation name
confidence : float
Confidence score
"""
features = self.extract_spatial_features(player_positions)
if features is None:
return None, 0.0
# Simple rule-based classification
if team == 'offense':
# Check for spread formation
if features['receiver_spread'] > 40:
return 'Spread', 0.8
# Check for shotgun
elif features['depth'] > 15:
return 'Shotgun', 0.75
else:
return 'I-Formation', 0.7
else: # defense
if features['num_clusters'] >= 4:
return 'Nickel', 0.7
elif features['density'] > 0.5:
return '4-3', 0.75
else:
return '3-4', 0.7
def analyze_formation_presnap(self, offense_positions, defense_positions):
"""
Analyze both offensive and defensive formations pre-snap
Parameters:
-----------
offense_positions : list of tuples
Offensive player positions
defense_positions : list of tuples
Defensive player positions
Returns:
--------
analysis : dict
Complete formation analysis
"""
off_formation, off_conf = self.classify_formation(offense_positions, 'offense')
def_formation, def_conf = self.classify_formation(defense_positions, 'defense')
off_features = self.extract_spatial_features(offense_positions)
def_features = self.extract_spatial_features(defense_positions)
analysis = {
'offense': {
'formation': off_formation,
'confidence': off_conf,
'features': off_features
},
'defense': {
'formation': def_formation,
'confidence': def_conf,
'features': def_features
},
'matchup': {
'offensive_width': off_features['width'] if off_features else 0,
'defensive_width': def_features['width'] if def_features else 0,
'defensive_density': def_features['density'] if def_features else 0
}
}
return analysis
# Example with synthetic data
recognizer = FormationRecognizer()
# Simulate offensive positions (spread formation)
offense_pos = [
(50, 10), # QB
(45, 5), (48, 5), (50, 5), (52, 5), (55, 5), # OL
(30, 15), (70, 15), # Wide receivers
(35, 12), (65, 12) # Slot receivers
]
# Simulate defensive positions
defense_pos = [
(45, -5), (48, -5), (52, -5), (55, -5), # DL
(47, -10), (53, -10), # LB
(30, -15), (70, -15), # CB
(40, -20), (60, -20) # S
]
analysis = recognizer.analyze_formation_presnap(offense_pos, defense_pos)
print("✓ Formation recognition complete")
print(f" - Offensive formation: {analysis['offense']['formation']} ({analysis['offense']['confidence']:.2f})")
print(f" - Defensive formation: {analysis['defense']['formation']} ({analysis['defense']['confidence']:.2f})")
Deep Learning for Formation Classification
#| label: formation-cnn
#| eval: false
#| message: false
#| warning: false
import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import Dataset, DataLoader
class FormationCNN(nn.Module):
"""CNN for classifying formations from image data"""
def __init__(self, num_classes=10):
super(FormationCNN, self).__init__()
self.features = nn.Sequential(
# Conv block 1
nn.Conv2d(3, 64, kernel_size=3, padding=1),
nn.BatchNorm2d(64),
nn.ReLU(inplace=True),
nn.MaxPool2d(kernel_size=2, stride=2),
# Conv block 2
nn.Conv2d(64, 128, kernel_size=3, padding=1),
nn.BatchNorm2d(128),
nn.ReLU(inplace=True),
nn.MaxPool2d(kernel_size=2, stride=2),
# Conv block 3
nn.Conv2d(128, 256, kernel_size=3, padding=1),
nn.BatchNorm2d(256),
nn.ReLU(inplace=True),
nn.Conv2d(256, 256, kernel_size=3, padding=1),
nn.BatchNorm2d(256),
nn.ReLU(inplace=True),
nn.MaxPool2d(kernel_size=2, stride=2),
# Conv block 4
nn.Conv2d(256, 512, kernel_size=3, padding=1),
nn.BatchNorm2d(512),
nn.ReLU(inplace=True),
nn.Conv2d(512, 512, kernel_size=3, padding=1),
nn.BatchNorm2d(512),
nn.ReLU(inplace=True),
nn.MaxPool2d(kernel_size=2, stride=2)
)
self.classifier = nn.Sequential(
nn.Dropout(0.5),
nn.Linear(512 * 7 * 7, 4096),
nn.ReLU(inplace=True),
nn.Dropout(0.5),
nn.Linear(4096, 1024),
nn.ReLU(inplace=True),
nn.Linear(1024, num_classes)
)
def forward(self, x):
x = self.features(x)
x = x.view(x.size(0), -1)
x = self.classifier(x)
return x
class FormationDataset(Dataset):
"""Dataset for formation classification"""
def __init__(self, image_paths, labels, transform=None):
self.image_paths = image_paths
self.labels = labels
self.transform = transform
def __len__(self):
return len(self.image_paths)
def __getitem__(self, idx):
# Load image
image = cv2.imread(self.image_paths[idx])
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
if self.transform:
image = self.transform(image)
label = self.labels[idx]
return image, label
def train_formation_classifier(train_loader, val_loader, num_epochs=50):
"""Train formation classification model"""
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
# Initialize model
model = FormationCNN(num_classes=10).to(device)
# Loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)
scheduler = optim.lr_scheduler.ReduceLROnPlateau(optimizer, 'min', patience=5)
best_val_loss = float('inf')
for epoch in range(num_epochs):
# Training phase
model.train()
train_loss = 0.0
train_correct = 0
train_total = 0
for images, labels in train_loader:
images, labels = images.to(device), labels.to(device)
# Forward pass
outputs = model(images)
loss = criterion(outputs, labels)
# Backward pass
optimizer.zero_grad()
loss.backward()
optimizer.step()
train_loss += loss.item()
_, predicted = outputs.max(1)
train_total += labels.size(0)
train_correct += predicted.eq(labels).sum().item()
# Validation phase
model.eval()
val_loss = 0.0
val_correct = 0
val_total = 0
with torch.no_grad():
for images, labels in val_loader:
images, labels = images.to(device), labels.to(device)
outputs = model(images)
loss = criterion(outputs, labels)
val_loss += loss.item()
_, predicted = outputs.max(1)
val_total += labels.size(0)
val_correct += predicted.eq(labels).sum().item()
# Calculate metrics
train_loss = train_loss / len(train_loader)
val_loss = val_loss / len(val_loader)
train_acc = 100. * train_correct / train_total
val_acc = 100. * val_correct / val_total
print(f'Epoch {epoch+1}/{num_epochs}:')
print(f' Train Loss: {train_loss:.4f}, Train Acc: {train_acc:.2f}%')
print(f' Val Loss: {val_loss:.4f}, Val Acc: {val_acc:.2f}%')
# Learning rate scheduling
scheduler.step(val_loss)
# Save best model
if val_loss < best_val_loss:
best_val_loss = val_loss
torch.save(model.state_dict(), 'best_formation_model.pt')
return model
print("✓ Formation CNN classifier defined")
Play Classification from Video
Action Recognition with 3D CNNs
#| label: play-classification
#| eval: false
#| message: false
#| warning: false
import torch.nn.functional as F
class PlayClassifier3DCNN(nn.Module):
"""3D CNN for play classification from video clips"""
def __init__(self, num_classes=20, dropout=0.5):
"""
Parameters:
-----------
num_classes : int
Number of play types to classify
"""
super(PlayClassifier3DCNN, self).__init__()
# 3D Convolutional layers
self.conv1 = nn.Conv3d(3, 64, kernel_size=(3, 3, 3), padding=(1, 1, 1))
self.pool1 = nn.MaxPool3d(kernel_size=(1, 2, 2), stride=(1, 2, 2))
self.conv2 = nn.Conv3d(64, 128, kernel_size=(3, 3, 3), padding=(1, 1, 1))
self.pool2 = nn.MaxPool3d(kernel_size=(2, 2, 2), stride=(2, 2, 2))
self.conv3a = nn.Conv3d(128, 256, kernel_size=(3, 3, 3), padding=(1, 1, 1))
self.conv3b = nn.Conv3d(256, 256, kernel_size=(3, 3, 3), padding=(1, 1, 1))
self.pool3 = nn.MaxPool3d(kernel_size=(2, 2, 2), stride=(2, 2, 2))
self.conv4a = nn.Conv3d(256, 512, kernel_size=(3, 3, 3), padding=(1, 1, 1))
self.conv4b = nn.Conv3d(512, 512, kernel_size=(3, 3, 3), padding=(1, 1, 1))
self.pool4 = nn.MaxPool3d(kernel_size=(2, 2, 2), stride=(2, 2, 2))
# Fully connected layers
self.fc1 = nn.Linear(512 * 2 * 7 * 7, 4096)
self.fc2 = nn.Linear(4096, 2048)
self.fc3 = nn.Linear(2048, num_classes)
self.dropout = nn.Dropout(p=dropout)
# Batch normalization
self.bn1 = nn.BatchNorm3d(64)
self.bn2 = nn.BatchNorm3d(128)
self.bn3a = nn.BatchNorm3d(256)
self.bn3b = nn.BatchNorm3d(256)
self.bn4a = nn.BatchNorm3d(512)
self.bn4b = nn.BatchNorm3d(512)
def forward(self, x):
# x shape: (batch, channels, frames, height, width)
# Conv block 1
x = F.relu(self.bn1(self.conv1(x)))
x = self.pool1(x)
# Conv block 2
x = F.relu(self.bn2(self.conv2(x)))
x = self.pool2(x)
# Conv block 3
x = F.relu(self.bn3a(self.conv3a(x)))
x = F.relu(self.bn3b(self.conv3b(x)))
x = self.pool3(x)
# Conv block 4
x = F.relu(self.bn4a(self.conv4a(x)))
x = F.relu(self.bn4b(self.conv4b(x)))
x = self.pool4(x)
# Flatten
x = x.view(x.size(0), -1)
# Fully connected layers
x = F.relu(self.fc1(x))
x = self.dropout(x)
x = F.relu(self.fc2(x))
x = self.dropout(x)
x = self.fc3(x)
return x
class PlayClassificationPipeline:
"""Complete pipeline for play classification"""
def __init__(self, model_path=None):
self.device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
self.model = PlayClassifier3DCNN(num_classes=20).to(self.device)
if model_path:
self.model.load_state_dict(torch.load(model_path))
self.play_types = [
'run_inside', 'run_outside', 'run_draw',
'pass_short', 'pass_medium', 'pass_deep',
'screen', 'play_action',
'zone_read', 'rpo',
'blitz', 'zone_coverage', 'man_coverage',
'prevent', 'goal_line',
'field_goal', 'punt', 'kickoff',
'extra_point', 'two_point_conversion'
]
def preprocess_clip(self, video_clip, num_frames=16):
"""
Preprocess video clip for model input
Parameters:
-----------
video_clip : np.ndarray
Video clip of shape (frames, height, width, channels)
num_frames : int
Number of frames to sample
Returns:
--------
tensor : torch.Tensor
Preprocessed tensor of shape (1, channels, frames, height, width)
"""
# Sample frames uniformly
indices = np.linspace(0, len(video_clip) - 1, num_frames, dtype=int)
sampled_frames = video_clip[indices]
# Resize frames
resized_frames = []
for frame in sampled_frames:
resized = cv2.resize(frame, (224, 224))
resized_frames.append(resized)
# Stack and convert to tensor
clip_array = np.stack(resized_frames, axis=0) # (frames, height, width, channels)
clip_array = clip_array.transpose(3, 0, 1, 2) # (channels, frames, height, width)
clip_array = clip_array / 255.0 # Normalize
# Convert to tensor and add batch dimension
tensor = torch.FloatTensor(clip_array).unsqueeze(0)
return tensor
def classify_play(self, video_clip):
"""
Classify play type from video clip
Parameters:
-----------
video_clip : np.ndarray
Video clip array
Returns:
--------
prediction : dict
Predicted play type and confidence scores
"""
self.model.eval()
# Preprocess
input_tensor = self.preprocess_clip(video_clip).to(self.device)
# Forward pass
with torch.no_grad():
outputs = self.model(input_tensor)
probabilities = F.softmax(outputs, dim=1)[0]
# Get top predictions
top5_prob, top5_idx = torch.topk(probabilities, 5)
prediction = {
'top_prediction': self.play_types[top5_idx[0]],
'confidence': float(top5_prob[0]),
'top_5': [
{
'play_type': self.play_types[idx],
'confidence': float(prob)
}
for idx, prob in zip(top5_idx, top5_prob)
]
}
return prediction
def extract_play_clips(self, video_path, snap_times, clip_duration=5):
"""
Extract play clips from full game video
Parameters:
-----------
video_path : str
Path to game video
snap_times : list of float
Times (in seconds) when ball is snapped
clip_duration : float
Duration of each clip in seconds
Returns:
--------
clips : list of np.ndarray
List of video clips
"""
cap = cv2.VideoCapture(video_path)
fps = cap.get(cv2.CAP_PROP_FPS)
clips = []
for snap_time in snap_times:
# Calculate frame range
start_frame = int((snap_time - 1) * fps) # 1 second before snap
end_frame = int((snap_time + clip_duration - 1) * fps)
# Extract clip
cap.set(cv2.CAP_PROP_POS_FRAMES, start_frame)
frames = []
for _ in range(end_frame - start_frame):
ret, frame = cap.read()
if not ret:
break
frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
frames.append(frame)
if len(frames) > 0:
clips.append(np.array(frames))
cap.release()
return clips
print("✓ Play classification pipeline ready")
Pose Estimation and Biomechanics
Player Pose Estimation
#| eval: false
#| echo: true
# Install pose estimation library
pip install mediapipe
#| label: pose-estimation
#| eval: false
#| message: false
#| warning: false
import mediapipe as mp
class FootballPoseEstimator:
"""Estimate player poses for biomechanical analysis"""
def __init__(self):
self.mp_pose = mp.solutions.pose
self.pose = self.mp_pose.Pose(
static_image_mode=False,
model_complexity=2,
enable_segmentation=False,
min_detection_confidence=0.5,
min_tracking_confidence=0.5
)
self.mp_drawing = mp.solutions.drawing_utils
def estimate_pose(self, frame):
"""
Estimate pose landmarks for frame
Parameters:
-----------
frame : np.ndarray
Input frame (RGB)
Returns:
--------
landmarks : list
Pose landmarks with (x, y, z, visibility) for each keypoint
"""
# Process frame
results = self.pose.process(frame)
if results.pose_landmarks:
landmarks = []
for landmark in results.pose_landmarks.landmark:
landmarks.append({
'x': landmark.x,
'y': landmark.y,
'z': landmark.z,
'visibility': landmark.visibility
})
return landmarks
return None
def draw_pose(self, frame, landmarks_result):
"""Draw pose landmarks on frame"""
annotated = frame.copy()
if landmarks_result:
self.mp_drawing.draw_landmarks(
annotated,
landmarks_result,
self.mp_pose.POSE_CONNECTIONS,
landmark_drawing_spec=self.mp_drawing.DrawingSpec(
color=(0, 255, 0), thickness=2, circle_radius=2
),
connection_drawing_spec=self.mp_drawing.DrawingSpec(
color=(255, 0, 0), thickness=2
)
)
return annotated
def analyze_throwing_mechanics(self, landmarks_sequence):
"""
Analyze quarterback throwing mechanics
Parameters:
-----------
landmarks_sequence : list of landmarks
Sequence of pose landmarks during throw
Returns:
--------
analysis : dict
Biomechanical analysis of throwing motion
"""
if not landmarks_sequence or len(landmarks_sequence) < 10:
return None
analysis = {
'shoulder_rotation': [],
'elbow_angle': [],
'hip_rotation': [],
'follow_through': None
}
for landmarks in landmarks_sequence:
# Calculate shoulder rotation
left_shoulder = landmarks[self.mp_pose.PoseLandmark.LEFT_SHOULDER.value]
right_shoulder = landmarks[self.mp_pose.PoseLandmark.RIGHT_SHOULDER.value]
shoulder_angle = np.arctan2(
right_shoulder['y'] - left_shoulder['y'],
right_shoulder['x'] - left_shoulder['x']
)
analysis['shoulder_rotation'].append(np.degrees(shoulder_angle))
# Calculate elbow angle
shoulder = landmarks[self.mp_pose.PoseLandmark.RIGHT_SHOULDER.value]
elbow = landmarks[self.mp_pose.PoseLandmark.RIGHT_ELBOW.value]
wrist = landmarks[self.mp_pose.PoseLandmark.RIGHT_WRIST.value]
elbow_angle = self._calculate_angle(
(shoulder['x'], shoulder['y']),
(elbow['x'], elbow['y']),
(wrist['x'], wrist['y'])
)
analysis['elbow_angle'].append(elbow_angle)
# Analyze follow-through
final_wrist = landmarks_sequence[-1][self.mp_pose.PoseLandmark.RIGHT_WRIST.value]
initial_wrist = landmarks_sequence[0][self.mp_pose.PoseLandmark.RIGHT_WRIST.value]
follow_through_distance = np.sqrt(
(final_wrist['x'] - initial_wrist['x'])**2 +
(final_wrist['y'] - initial_wrist['y'])**2
)
analysis['follow_through'] = follow_through_distance
return analysis
def _calculate_angle(self, p1, p2, p3):
"""Calculate angle between three points"""
v1 = np.array([p1[0] - p2[0], p1[1] - p2[1]])
v2 = np.array([p3[0] - p2[0], p3[1] - p2[1]])
cos_angle = np.dot(v1, v2) / (np.linalg.norm(v1) * np.linalg.norm(v2))
angle = np.arccos(np.clip(cos_angle, -1.0, 1.0))
return np.degrees(angle)
def detect_contact_events(self, landmarks_sequence, threshold=0.3):
"""
Detect potential contact/collision events
Parameters:
-----------
landmarks_sequence : list
Sequence of pose landmarks
threshold : float
Motion threshold for contact detection
Returns:
--------
events : list
List of detected contact events with frame indices
"""
events = []
for i in range(1, len(landmarks_sequence)):
prev_landmarks = landmarks_sequence[i-1]
curr_landmarks = landmarks_sequence[i]
# Calculate motion magnitude
motion = 0
for j in range(len(curr_landmarks)):
if curr_landmarks[j] and prev_landmarks[j]:
dx = curr_landmarks[j]['x'] - prev_landmarks[j]['x']
dy = curr_landmarks[j]['y'] - prev_landmarks[j]['y']
motion += np.sqrt(dx**2 + dy**2)
motion /= len(curr_landmarks)
# Detect sudden motion changes (potential contact)
if motion > threshold:
events.append({
'frame': i,
'motion_magnitude': motion,
'type': 'potential_contact'
})
return events
# Example usage
pose_estimator = FootballPoseEstimator()
print("✓ Pose estimation system ready")
print(" - Can analyze throwing mechanics")
print(" - Can detect contact events")
print(" - Supports real-time tracking")
Real-Time Processing Considerations
Optimization Strategies
#| label: realtime-processing
#| message: false
#| warning: false
class RealtimeVideoProcessor:
"""Optimized processor for real-time video analysis"""
def __init__(self, detector_model, tracker_model=None):
self.detector = detector_model
self.tracker = tracker_model
# Performance settings
self.skip_frames = 2 # Process every Nth frame
self.detection_interval = 10 # Run detection every N frames
self.frame_buffer = []
self.frame_count = 0
# Threading for parallel processing
self.processing_queue = []
def set_performance_mode(self, mode='balanced'):
"""
Set performance/accuracy trade-off
Parameters:
-----------
mode : str
'fast' - prioritize speed
'balanced' - balance speed and accuracy
'accurate' - prioritize accuracy
"""
if mode == 'fast':
self.skip_frames = 3
self.detection_interval = 15
self.detector.conf_threshold = 0.6
elif mode == 'balanced':
self.skip_frames = 2
self.detection_interval = 10
self.detector.conf_threshold = 0.5
elif mode == 'accurate':
self.skip_frames = 1
self.detection_interval = 5
self.detector.conf_threshold = 0.4
def process_frame_optimized(self, frame):
"""
Process frame with optimization strategies
Parameters:
-----------
frame : np.ndarray
Input frame
Returns:
--------
results : dict
Processing results
"""
self.frame_count += 1
# Skip frames for performance
if self.frame_count % self.skip_frames != 0:
return None
results = {}
# Run detection only periodically
if self.frame_count % self.detection_interval == 0:
detections = self.detector.detect_players(frame)
results['detections'] = detections
# Update tracker with new detections
if self.tracker:
self.tracker.update(detections)
# Always run tracking (lighter operation)
if self.tracker:
tracks = self.tracker.get_current_tracks()
results['tracks'] = tracks
return results
def resize_for_inference(self, frame, target_size=(640, 640)):
"""Resize frame to optimal inference size"""
return cv2.resize(frame, target_size)
def use_model_optimization(self, model_path, optimization='tensorrt'):
"""
Apply model optimization techniques
Parameters:
-----------
model_path : str
Path to model
optimization : str
'tensorrt', 'onnx', 'openvino', or 'tflite'
"""
if optimization == 'tensorrt':
# TensorRT optimization for NVIDIA GPUs
print("Applying TensorRT optimization...")
# Implementation would go here
pass
elif optimization == 'onnx':
# ONNX optimization for cross-platform
print("Converting to ONNX format...")
# Implementation would go here
pass
elif optimization == 'openvino':
# Intel OpenVINO optimization
print("Applying OpenVINO optimization...")
# Implementation would go here
pass
def benchmark_performance(self, test_video, num_frames=100):
"""
Benchmark processing performance
Parameters:
-----------
test_video : str
Path to test video
num_frames : int
Number of frames to process for benchmark
Returns:
--------
metrics : dict
Performance metrics
"""
import time
cap = cv2.VideoCapture(test_video)
start_time = time.time()
frames_processed = 0
for _ in range(num_frames):
ret, frame = cap.read()
if not ret:
break
self.process_frame_optimized(frame)
frames_processed += 1
end_time = time.time()
cap.release()
elapsed_time = end_time - start_time
fps = frames_processed / elapsed_time
metrics = {
'frames_processed': frames_processed,
'elapsed_time': elapsed_time,
'fps': fps,
'ms_per_frame': (elapsed_time / frames_processed) * 1000
}
return metrics
# Performance tips
print("✓ Real-time processing optimizations:")
print(" 1. Use GPU acceleration (CUDA/TensorRT)")
print(" 2. Reduce input resolution (640x640 or 416x416)")
print(" 3. Skip frames (process every 2-3 frames)")
print(" 4. Run detection periodically, tracking continuously")
print(" 5. Use lightweight models (YOLOv8n instead of YOLOv8x)")
print(" 6. Batch processing when possible")
print(" 7. Multi-threading for I/O operations")
Building Automated Scouting Systems
Complete End-to-End Pipeline
#| label: automated-scouting
#| eval: false
#| message: false
#| warning: false
class AutomatedScoutingSystem:
"""Complete system for automated video scouting"""
def __init__(self, config):
"""
Initialize scouting system
Parameters:
-----------
config : dict
Configuration with model paths and parameters
"""
# Initialize components
self.detector = FootballObjectDetector(config['detector_model'])
self.tracker = FootballTracker(config['detector_model'])
self.formation_recognizer = FormationRecognizer()
self.play_classifier = PlayClassificationPipeline(config.get('play_model'))
self.pose_estimator = FootballPoseEstimator()
# Database for storing results
self.scouting_database = []
def analyze_game(self, video_path, game_metadata):
"""
Analyze complete game video
Parameters:
-----------
video_path : str
Path to game video
game_metadata : dict
Game information (teams, date, etc.)
Returns:
--------
game_analysis : dict
Complete game analysis
"""
print(f"Analyzing game: {game_metadata.get('teams', 'Unknown')}")
game_analysis = {
'metadata': game_metadata,
'plays': [],
'player_stats': {},
'formation_stats': {},
'highlights': []
}
# Detect play segments
play_segments = self._detect_play_segments(video_path)
# Analyze each play
for i, segment in enumerate(play_segments):
print(f"Analyzing play {i+1}/{len(play_segments)}")
play_analysis = self._analyze_play(video_path, segment)
game_analysis['plays'].append(play_analysis)
# Update stats
self._update_stats(game_analysis, play_analysis)
# Generate summary
game_analysis['summary'] = self._generate_summary(game_analysis)
return game_analysis
def _detect_play_segments(self, video_path):
"""
Detect individual play segments in game video
Uses scene detection and motion analysis to identify plays
"""
cap = cv2.VideoCapture(video_path)
fps = cap.get(cv2.CAP_PROP_FPS)
segments = []
in_play = False
play_start = 0
frame_num = 0
prev_frame = None
while cap.isOpened():
ret, frame = cap.read()
if not ret:
break
# Detect field
field_mask = detect_field_hsv(frame)
field_ratio = np.sum(field_mask > 0) / field_mask.size
# Calculate motion
motion = 0
if prev_frame is not None:
diff = cv2.absdiff(frame, prev_frame)
motion = np.mean(diff)
# Detect play start/end
if not in_play and field_ratio > 0.5 and motion > 20:
in_play = True
play_start = frame_num
elif in_play and (field_ratio < 0.3 or motion < 5):
in_play = False
segments.append({
'start_frame': play_start,
'end_frame': frame_num,
'start_time': play_start / fps,
'end_time': frame_num / fps
})
prev_frame = frame.copy()
frame_num += 1
cap.release()
return segments
def _analyze_play(self, video_path, segment):
"""Analyze individual play"""
# Extract play clip
clip = self._extract_clip(video_path, segment['start_frame'],
segment['end_frame'])
# Pre-snap analysis
presnap_frame = clip[0]
player_positions = self._extract_player_positions(presnap_frame)
formation_analysis = self.formation_recognizer.analyze_formation_presnap(
player_positions['offense'],
player_positions['defense']
)
# Classify play type
play_classification = self.play_classifier.classify_play(clip)
# Track players during play
tracking_data = self._track_play(clip)
# Analyze key moments
key_moments = self._identify_key_moments(clip, tracking_data)
play_analysis = {
'segment': segment,
'formation': formation_analysis,
'play_type': play_classification,
'tracking': tracking_data,
'key_moments': key_moments,
'metrics': self._calculate_play_metrics(tracking_data)
}
return play_analysis
def _extract_clip(self, video_path, start_frame, end_frame):
"""Extract clip from video"""
cap = cv2.VideoCapture(video_path)
cap.set(cv2.CAP_PROP_POS_FRAMES, start_frame)
frames = []
for _ in range(end_frame - start_frame):
ret, frame = cap.read()
if not ret:
break
frames.append(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB))
cap.release()
return np.array(frames)
def _extract_player_positions(self, frame):
"""Extract player positions from frame"""
detections = self.detector.detect_all_objects(frame)
positions = {
'offense': [(d['center'][0], d['center'][1])
for d in detections['players_offense']],
'defense': [(d['center'][0], d['center'][1])
for d in detections['players_defense']]
}
return positions
def _track_play(self, clip):
"""Track all players throughout play"""
tracking_data = []
for frame in clip:
detections = self.detector.detect_all_objects(frame)
tracking_data.append(detections)
return tracking_data
def _identify_key_moments(self, clip, tracking_data):
"""Identify key moments in play"""
key_moments = []
# Detect snap
snap_frame = self._detect_snap(clip)
if snap_frame:
key_moments.append({'type': 'snap', 'frame': snap_frame})
# Detect pass/handoff
ball_transfer = self._detect_ball_transfer(tracking_data)
if ball_transfer:
key_moments.append({'type': 'ball_transfer', 'frame': ball_transfer})
# Detect tackle/completion
end_event = self._detect_play_end(tracking_data)
if end_event:
key_moments.append(end_event)
return key_moments
def _detect_snap(self, clip):
"""Detect snap frame using motion analysis"""
if len(clip) < 2:
return None
max_motion = 0
snap_frame = None
for i in range(1, min(30, len(clip))): # Check first 30 frames
diff = cv2.absdiff(clip[i], clip[i-1])
motion = np.mean(diff)
if motion > max_motion:
max_motion = motion
snap_frame = i
return snap_frame
def _detect_ball_transfer(self, tracking_data):
"""Detect pass or handoff"""
# Simplified implementation
return None
def _detect_play_end(self, tracking_data):
"""Detect tackle or incomplete pass"""
# Simplified implementation
return {'type': 'tackle', 'frame': len(tracking_data) - 1}
def _calculate_play_metrics(self, tracking_data):
"""Calculate metrics from tracking data"""
metrics = {
'duration': len(tracking_data),
'players_tracked': 0,
'average_speed': 0,
'max_speed': 0
}
return metrics
def _update_stats(self, game_analysis, play_analysis):
"""Update cumulative statistics"""
formation = play_analysis['formation']['offense']['formation']
if formation not in game_analysis['formation_stats']:
game_analysis['formation_stats'][formation] = 0
game_analysis['formation_stats'][formation] += 1
def _generate_summary(self, game_analysis):
"""Generate game summary"""
summary = {
'total_plays': len(game_analysis['plays']),
'most_common_formation': max(
game_analysis['formation_stats'].items(),
key=lambda x: x[1]
)[0] if game_analysis['formation_stats'] else None,
'highlights_count': len(game_analysis['highlights'])
}
return summary
def generate_scouting_report(self, game_analysis, output_path):
"""Generate detailed scouting report"""
report = {
'game_info': game_analysis['metadata'],
'executive_summary': game_analysis['summary'],
'formation_breakdown': game_analysis['formation_stats'],
'key_plays': game_analysis['highlights'],
'recommendations': self._generate_recommendations(game_analysis)
}
# Save report
import json
with open(output_path, 'w') as f:
json.dump(report, f, indent=2)
return report
def _generate_recommendations(self, game_analysis):
"""Generate scouting recommendations"""
recommendations = []
# Analyze tendencies
if game_analysis['formation_stats']:
most_used = max(game_analysis['formation_stats'].items(),
key=lambda x: x[1])
recommendations.append(
f"Team heavily uses {most_used[0]} formation ({most_used[1]} times)"
)
return recommendations
# Example configuration
config = {
'detector_model': 'football_yolov8.pt',
'play_model': 'play_classifier.pt'
}
# scouting_system = AutomatedScoutingSystem(config)
# game_analysis = scouting_system.analyze_game('game.mp4', {'teams': 'Team A vs Team B'})
print("✓ Automated scouting system ready")
print(" - Detects play segments automatically")
print(" - Analyzes formations and play types")
print(" - Tracks all players")
print(" - Generates comprehensive reports")
Visualizations
Drawing Detections and Tracking
#| label: fig-detection-viz-r
#| eval: false
#| fig-cap: "Player detection and tracking visualization"
#| fig-width: 12
#| fig-height: 8
library(ggplot2)
library(gganimate)
# Simulate tracking data
set.seed(123)
frames <- 100
players <- 11
tracking_data <- data.frame(
frame = rep(1:frames, each = players),
player_id = rep(1:players, frames),
x = rnorm(frames * players, 50, 20),
y = rnorm(frames * players, 30, 10)
)
# Create visualization
p <- ggplot(tracking_data, aes(x = x, y = y, color = factor(player_id))) +
geom_point(size = 3) +
geom_path(aes(group = player_id), alpha = 0.3) +
scale_color_discrete(name = "Player ID") +
coord_fixed(ratio = 1, xlim = c(0, 100), ylim = c(0, 60)) +
labs(
title = "Player Tracking from Video",
subtitle = "Frame: {frame_time}",
x = "X Position (yards)",
y = "Y Position (yards)"
) +
theme_minimal() +
theme(
plot.title = element_text(face = "bold", size = 14),
legend.position = "right"
) +
transition_time(frame)
# This would create an animation
# animate(p, nframes = 100, fps = 10)
📊 Visualization Output
The code above generates a visualization. To see the output, run this code in your R or Python environment. The resulting plot will help illustrate the concepts discussed in this section.
#| label: fig-detection-viz-py
#| fig-cap: "Player detection and tracking visualization - Python"
#| fig-width: 12
#| fig-height: 8
import matplotlib.pyplot as plt
import matplotlib.patches as patches
# Simulate frame with detections
fig, axes = plt.subplots(1, 2, figsize=(14, 6))
# Left: Raw detections
ax1 = axes[0]
frame_dummy = np.random.randint(0, 255, (480, 640, 3), dtype=np.uint8)
ax1.imshow(frame_dummy)
# Draw bounding boxes
detections_demo = [
{'bbox': [100, 150, 150, 250], 'id': 1},
{'bbox': [200, 160, 250, 260], 'id': 2},
{'bbox': [300, 140, 350, 240], 'id': 3},
{'bbox': [400, 155, 450, 255], 'id': 4}
]
for det in detections_demo:
x1, y1, x2, y2 = det['bbox']
rect = patches.Rectangle((x1, y1), x2-x1, y2-y1,
linewidth=2, edgecolor='green',
facecolor='none')
ax1.add_patch(rect)
ax1.text(x1, y1-5, f"ID: {det['id']}",
color='green', fontweight='bold')
ax1.set_title('Player Detection', fontsize=12, fontweight='bold')
ax1.axis('off')
# Right: Tracking trajectories
ax2 = axes[1]
ax2.set_xlim(0, 100)
ax2.set_ylim(0, 53.3)
ax2.set_aspect('equal')
# Simulate trajectories
np.random.seed(42)
for player_id in range(1, 6):
x_traj = np.linspace(20, 80, 50) + np.random.randn(50) * 2
y_traj = 26.65 + np.random.randn(50) * 10
ax2.plot(x_traj, y_traj, alpha=0.6, linewidth=2,
label=f'Player {player_id}')
ax2.scatter(x_traj[-1], y_traj[-1], s=100, zorder=5)
# Draw field markings
for yard in range(0, 101, 10):
ax2.axvline(yard, color='white', alpha=0.3, linewidth=0.5)
ax2.axhline(0, color='white', linewidth=2)
ax2.axhline(53.3, color='white', linewidth=2)
ax2.set_title('Player Tracking Trajectories', fontsize=12, fontweight='bold')
ax2.set_xlabel('Field Position (yards)', fontsize=10)
ax2.set_ylabel('Width (yards)', fontsize=10)
ax2.legend(loc='upper right', fontsize=8)
ax2.set_facecolor('#2d5016')
ax2.grid(True, alpha=0.2)
plt.tight_layout()
plt.show()
📊 Visualization Output
The code above generates a visualization. To see the output, run this code in your R or Python environment. The resulting plot will help illustrate the concepts discussed in this section.
Formation and Heat Map Visualization
#| label: fig-formation-heatmap
#| fig-cap: "Formation detection and heat map from video"
#| fig-width: 14
#| fig-height: 6
from scipy.stats import gaussian_kde
fig, axes = plt.subplots(1, 2, figsize=(14, 6))
# Left: Formation diagram
ax1 = axes[0]
ax1.set_xlim(-10, 110)
ax1.set_ylim(-5, 58.3)
ax1.set_aspect('equal')
# Draw field
ax1.axhline(0, color='white', linewidth=2)
ax1.axhline(53.3, color='white', linewidth=2)
for yard in [0, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100]:
ax1.axvline(yard, color='white', alpha=0.3, linewidth=1)
# Draw offensive formation (spread)
offense_positions = np.array([
[50, 26.65], # QB (center)
[48, 26.65], # Center
[46, 26.65], [44, 26.65], # Guards
[42, 26.65], [40, 26.65], # Tackles
[35, 5], [35, 48], # Wide receivers
[45, 15], [45, 38], # Slot receivers
[55, 26.65] # RB
])
# Draw defensive formation (nickel)
defense_positions = np.array([
[45, 26.65], [47, 26.65], [49, 26.65], [51, 26.65], # DL
[42, 20], [42, 33], # LB
[35, 8], [35, 25], [35, 45], # DB
[30, 15], [30, 38] # Safeties
])
ax1.scatter(offense_positions[:, 0], offense_positions[:, 1],
s=200, c='blue', marker='o', edgecolors='white',
linewidth=2, label='Offense', zorder=5)
ax1.scatter(defense_positions[:, 0], defense_positions[:, 1],
s=200, c='red', marker='s', edgecolors='white',
linewidth=2, label='Defense', zorder=5)
ax1.set_title('Detected Formation\nOffense: Spread | Defense: Nickel',
fontsize=12, fontweight='bold')
ax1.set_xlabel('Field Position (yards)', fontsize=10)
ax1.set_ylabel('Width (yards)', fontsize=10)
ax1.legend(loc='upper left', fontsize=10)
ax1.set_facecolor('#2d5016')
ax1.grid(True, alpha=0.2, color='white')
# Right: Heat map from video
ax2 = axes[1]
# Generate sample heat map data
np.random.seed(42)
all_positions = np.vstack([
np.random.multivariate_normal([45, 26], [[20, 0], [0, 50]], 500),
np.random.multivariate_normal([65, 26], [[30, 0], [0, 30]], 300),
np.random.multivariate_normal([25, 26], [[15, 0], [0, 40]], 200)
])
# Create heat map
x = all_positions[:, 0]
y = all_positions[:, 1]
# Create grid
xi = np.linspace(0, 100, 100)
yi = np.linspace(0, 53.3, 53)
xi, yi = np.meshgrid(xi, yi)
# Calculate density
positions = np.vstack([x, y])
kernel = gaussian_kde(positions)
zi = kernel(np.vstack([xi.flatten(), yi.flatten()]))
zi = zi.reshape(xi.shape)
# Plot heat map
im = ax2.contourf(xi, yi, zi, levels=15, cmap='YlOrRd', alpha=0.7)
ax2.contour(xi, yi, zi, levels=15, colors='black', alpha=0.2, linewidths=0.5)
# Field markings
for yard in range(0, 101, 10):
ax2.axvline(yard, color='white', alpha=0.3, linewidth=0.5)
ax2.axhline(0, color='white', linewidth=2)
ax2.axhline(53.3, color='white', linewidth=2)
ax2.set_title('Player Position Heat Map\nExtracted from Video',
fontsize=12, fontweight='bold')
ax2.set_xlabel('Field Position (yards)', fontsize=10)
ax2.set_ylabel('Width (yards)', fontsize=10)
ax2.set_xlim(0, 100)
ax2.set_ylim(0, 53.3)
ax2.set_facecolor('#2d5016')
# Add colorbar
cbar = plt.colorbar(im, ax=ax2)
cbar.set_label('Player Density', fontsize=9)
plt.tight_layout()
plt.show()
📊 Visualization Output
The code above generates a visualization. To see the output, run this code in your R or Python environment. The resulting plot will help illustrate the concepts discussed in this section.
Summary
Computer vision and video analysis represent a transformative frontier in football analytics. The ability to extract rich spatial and temporal data from video opens up entirely new possibilities for player evaluation, tactical analysis, and automated scouting.
In this chapter, we covered:
- Image Processing Fundamentals: Loading, preprocessing, and basic operations on football video
- Object Detection: Using YOLO and custom models to detect players, the ball, and field elements
- Multi-Object Tracking: Maintaining player identities across frames using SORT and Kalman filters
- Formation Recognition: Spatial analysis and classification of offensive and defensive formations
- Play Classification: Deep learning approaches for automated play-type recognition
- Pose Estimation: Biomechanical analysis of player movements and techniques
- Real-Time Processing: Optimization strategies for live video analysis
- Automated Scouting: Building complete end-to-end systems for video analysis
The techniques presented here are actively used by professional teams and are rapidly evolving as deep learning methods improve. While the implementation details can be complex, the fundamental concepts—detection, tracking, classification, and analysis—remain consistent across applications.
Exercises
Conceptual Questions
-
Detection vs Tracking: Explain the difference between object detection and object tracking. Why is tracking necessary in addition to detection?
-
Formation Recognition: What spatial features would you extract to distinguish between an I-Formation and a Shotgun formation?
-
Real-Time Constraints: Discuss three optimization strategies for achieving real-time video processing (30 fps) and the trade-offs involved.
Coding Exercises
Exercise 1: Player Detection System
Build a basic player detection system: a) Load a football video frame b) Apply a pre-trained YOLO model to detect all persons (players) c) Filter detections to only include those on the field (using field detection) d) Visualize the detections with bounding boxes **Bonus**: Calculate the average confidence score and detection count per frame across a 10-second clip.Exercise 2: Formation Classifier
Create a formation classifier: a) Generate synthetic player position data for 3 different formations b) Extract spatial features (width, depth, clustering, etc.) for each formation c) Train a simple classifier (logistic regression or decision tree) on the features d) Evaluate the classifier's accuracy on a test set **Hint**: Use sklearn for the classification model.Exercise 3: Tracking Evaluation
Implement basic tracking evaluation: a) Create synthetic tracking data with known ground truth b) Implement a simple IOU-based tracker c) Calculate tracking metrics: precision, recall, and MOTA (Multiple Object Tracking Accuracy) d) Compare performance with different IOU thresholds **Formula**: MOTA = 1 - (FP + FN + ID_switches) / GT_detectionsExercise 4: Automated Scouting Prototype
Build a simple automated scouting tool: a) Load a short game clip (30 seconds) b) Detect players in each frame c) Identify when the ball is snapped (using motion detection) d) Extract and save the pre-snap frame for each play e) Generate a simple report with: - Number of plays detected - Average number of players detected per play - Screenshots of each pre-snap formation **Extension**: Add formation classification to automatically label each play's formation.Further Reading
Academic Papers
-
Bridgeman, L., et al. (2019). "A Deep Learning Approach for Automated Formation Recognition in American Football." MIT Sloan Sports Analytics Conference.
-
Mehrasa, N., et al. (2018). "Deep Learning of Player Trajectory Representations for Team Activity Analysis." MIT Sloan Sports Analytics Conference.
-
Quiroga, R., et al. (2020). "Player Tracking Data in Professional Football: New Metrics and Analysis." Journal of Sports Analytics, 6(2), 115-132.
Technical Resources
-
Redmon, J., & Farhadi, A. (2018). "YOLOv3: An Incremental Improvement." arXiv preprint arXiv:1804.02767.
-
Bewley, A., et al. (2016). "Simple Online and Realtime Tracking." IEEE International Conference on Image Processing.
-
Cao, Z., et al. (2019). "OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields." IEEE Transactions on Pattern Analysis and Machine Intelligence.
Software and Libraries
- Ultralytics YOLOv8: https://github.com/ultralytics/ultralytics
- OpenCV: https://opencv.org/
- MediaPipe: https://google.github.io/mediapipe/
- PyTorch: https://pytorch.org/
- TensorFlow: https://www.tensorflow.org/
Industry Applications
- NFL Next Gen Stats computer vision pipeline
- Hudl automated tagging and analysis
- Second Spectrum tracking technology
- Stats Perform optical tracking systems
References
:::