“Hyperparameter tuning for MATLAB classification models”

by Posted on

Hyperparameter tuning is critical for optimizing the performance of MATLAB classification models. Below is a structured guide to hyperparameter tuning for common classifiers like SVM, decision trees, ensemble methods, and neural networks.

1. Common Hyperparameters by Model

Model Key Hyperparameters
SVM KernelFunction, KernelScale, BoxConstraint (regularization), PolynomialOrder
Decision Tree MinLeafSize, MaxNumSplits, SplitCriterion (Gini, entropy)
Random Forest NumLearningCycles, MinLeafSize, MaxNumSplits, NumVariablesToSample
k-Nearest Neighbors NumNeighbors, DistanceMetric, Standardize
Neural Networks Layer sizes, Activation functions, Learning rate, Regularization (L1/L2)
Logistic Regression Regularization type (ridge/lasso), Lambda (strength)

2. Manual Tuning Workflow

Step 1: Train a Baseline Model

matlab
Copy
% Example: SVM with default parameters
mdl = fitcsvm(X_train, Y_train, 'KernelFunction', 'linear');

Step 2: Evaluate Performance

matlab
Copy
% Predict and compute metrics
Y_pred = predict(mdl, X_val);
accuracy = sum(Y_pred == Y_val) / numel(Y_val);
confusionmat(Y_val, Y_pred);

Step 3: Adjust Hyperparameters Iteratively

matlab
Copy
% Example: Tuning SVM kernel scale and box constraint
mdl_tuned = fitcsvm(X_train, Y_train, ...
    'KernelFunction', 'rbf', ...
    'KernelScale', 'auto', ...  % or specify a value
    'BoxConstraint', 1.5);

3. Automated Hyperparameter Optimization

Use MATLAB’s hyperparameter optimization tools to automate the search process.

Method 1: bayesopt (Bayesian Optimization)

matlab
Copy
% Define hyperparameters
params = hyperparameters('fitcsvm', X_train, Y_train);
params(1).Range = [1e-3, 1e3];  % BoxConstraint
params(2).Range = [1e-3, 1e3];  % KernelScale

% Optimize
results = bayesopt(@(params)svm_loss(params, X_train, Y_train, X_val, Y_val), params);

% Helper function for loss calculation
function loss = svm_loss(params, X_train, Y_train, X_val, Y_val)
    mdl = fitcsvm(X_train, Y_train, ...
        'KernelFunction', 'rbf', ...
        'BoxConstraint', params.BoxConstraint, ...
        'KernelScale', params.KernelScale);
    Y_pred = predict(mdl, X_val);
    loss = 1 - sum(Y_pred == Y_val)/numel(Y_val); % Misclassification rate
end

Method 2: Grid Search

matlab
Copy
% Example: Grid search for k-NN
numNeighbors = [3, 5, 7, 10];
accuracies = zeros(length(numNeighbors), 1);

for i = 1:length(numNeighbors)
    mdl = fitcknn(X_train, Y_train, 'NumNeighbors', numNeighbors(i));
    Y_pred = predict(mdl, X_val);
    accuracies(i) = sum(Y_pred == Y_val)/numel(Y_val);
end

[best_acc, best_idx] = max(accuracies);
best_k = numNeighbors(best_idx);

4. Cross-Validation for Robust Tuning

Use crossval or cvpartition to avoid overfitting:

matlab
Copy
% Create a cross-validated model
cv_mdl = crossval(mdl, 'KFold', 5);

% Compute cross-validation loss
cv_loss = kfoldLoss(cv_mdl);

5. Handling Imbalanced Classes

For imbalanced datasets:

  • Use class weighting ('Weights' in fitcsvm, fitctree, etc.).
  • Resample data with SMOTE or undersampling.
matlab
Copy
% Example: Class weights in SVM
class_weights = [0.6, 0.4]; % Weight for class 0 and 1
mdl = fitcsvm(X_train, Y_train, 'Weights', class_weights);

6. Visualization Tools

  • ROC Curve: Evaluate classifier performance.
    matlab
    Copy
    [X,Y,T,AUC] = perfcurve(Y_val, scores(:,2), 1);
plot(X,Y);
  • Hyperparameter Impact: Plot hyperparameters vs. accuracy.
    matlab
    Copy
    plot(numNeighbors, accuracies);
xlabel('Number of Neighbors');
ylabel('Validation Accuracy');

7. Example: Full Tuning Workflow for SVM

matlab
Copy
% Load data
load fisheriris
X = meas(:,1:2);
Y = categorical(species);

% Split data into training/validation
cv = cvpartition(Y, 'HoldOut', 0.3);
X_train = X(training(cv), :);
Y_train = Y(training(cv));
X_val = X(test(cv), :);
Y_val = Y(test(cv));

% Bayesian optimization for SVM
params = hyperparameters('fitcsvm', X_train, Y_train);
params(1).Range = [1e-3, 1e3]; % BoxConstraint
params(2).Range = [1e-3, 1e3]; % KernelScale

results = bayesopt(@(params)svm_loss(params, X_train, Y_train, X_val, Y_val), params);

% Train final model with best parameters
best_mdl = fitcsvm(X_train, Y_train, ...
    'KernelFunction', 'rbf', ...
    'BoxConstraint', results.XAtMinObjective.BoxConstraint, ...
    'KernelScale', results.XAtMinObjective.KernelScale);

% Evaluate
Y_pred = predict(best_mdl, X_val);
accuracy = sum(Y_pred == Y_val)/numel(Y_val);

Key Considerations

  1. Trade-off Between Bias and Variance:
    • High model complexity (e.g., small MinLeafSize in trees) → Low bias, high variance.
    • Regularization (e.g., BoxConstraint in SVM) controls overfitting.
  2. Computational Cost:
    • Bayesian optimization is efficient but requires the Statistics and Machine Learning Toolbox.
    • Parallelize tuning with 'UseParallel', true in bayesopt.
  3. Validation Strategy:
    • Always use a hold-out validation set or cross-validation to avoid overfitting.

MATLAB Functions for Tuning

  • bayesopt: Bayesian optimization.
  • fitcsvm, fitctree, fitcensemble: Model-specific tuning.
  • crossval, cvpartition: Cross-validation utilities.
  • perfcurve: Performance metrics (ROC, AUC).

By systematically tuning hyperparameters, you can maximize the performance of your MATLAB classification models.

Published by howdy