Fault Logging and Assessment for Responsive EV Repairs

The FLARE project aims to automate fault detection in EV charging stations using Semantic Segmentation and Vision-Transformer-based Image Classification. By leveraging real-world data, we help identify charger malfunctions faster and improve maintenance response times.

Model Repo Website Repo See Demo

Problem

Broken EV chargers (both external and internal issues) make charging unreliable for EV owners and slows down EV adoption.

Solution

AI-driven fault detection using semantic segmentation and binary classifier to speed up the repair process.

Goal

Increases EV charger reliability, strengthens charging infrastructure, and accelerates EV adoption.

Data Collection & Labeling

The dataset consists of images of EV chargers collected from both manual and online sources, ensuring diverse conditions to improve model generalization.

Manual collection involved capturing chargers in various operational states—healthy, broken screen, damaged cord, or damaged plug—from different angles, with a focus on front-facing perspectives to simulate future user photos. Charger locations were identified using PlugShare, prioritizing stations reported as faulty or undergoing maintenance to maximize the presence of malfunctioning units.

To further expand the dataset, additional images were sourced online from Google Photos, Reddit, EV forums, and blog posts where users discussed charger failures. These sources provided real-world failure cases, increasing dataset diversity. All online images were manually reviewed to ensure relevance and consistency with manually collected samples.

Examples of Collected Data:

Segments.ai Labeling Process

Using the website, members manually labeled different parts of the chargers into:

Screen
Body
Cord
Plug
Void Background

After labeling a sizeable amount of charger images, we used these labeled ground truth images to train our segmentation model & binary classification model.

Post Labeling Dataset:

Images

Healthy

Broken

Labeled

Methodology

Our project enhances EV charger reliability by leveraging artificial intelligence to automatically detect and classify charger issues. To achieve this, we use two primary AI methods: Semantic Segmentation and Binary Classification.

Step 1: Semantic Segmentation

We first segment the different parts of an EV charger—such as the screen, body, cable, and plug—before analyzing their condition.

How? We first tested this technique on a sidewalk dataset before eventually upgrading to a more advanced segmentation model, MIT-B3 SegFormer, to our labeled EV charger dataset.
Why? This helps us focus on specific parts rather than the whole charger, making the classification process easier and more accurate.

Step 2: Binary Classification

Once the charger components are segmented, we train a model to classify whether each part is functioning correctly or broken.

How? We collected and labeled images of EV chargers as either "healthy" or "broken" (e.g., damaged screen, faulty plug, disconnected cable, and out of service chargers).
What model? We use Vision Transformers (ViT)—a type of AI that analyzes images similarly to how people process visual information.
Process: Images are cleaned and standardized, then split into training and testing sets. The model learns from the training data and is tested on new images.

Semantic Segmentation Model Development

Our approach to EV charger segmentation began with an initial feasibility study using a sidewalk dataset from Hugging Face to test transformer-based semantic segmentation models. We started with the MiT-B0 SegFormer, a lightweight segmentation model with only 3.7 million parameters, to evaluate whether transformers could effectively delineate structured objects in real-world images. The goal was to determine whether this segmentation approach could successfully isolate key charger components such as the screen, body, cable, and plug.

The figure above showcases an example of SegFormer performing on sidewalk data.
Link

After validating the approach on the sidewalk dataset, we applied the same MiT-B0 SegFormer model to our custom dataset of EV chargers. The dataset was pre-labeled, containing pixel-wise annotations for each charger component. Initial results demonstrated that while MiT-B0 could segment charger features, it struggled with finer details, particularly in occluded and low-resolution images.

Hugging Face repository for our initial solidified SegFormer B0 model.

The figure above represents the SegFormer Architecture.
Link

For dataset training, the collected images were randomly shuffled and split into 80% training and 20% testing subsets using train_test_split(). The model leveraged a learning rate of 0.00006, a batch size of 2, and 50 training epochs. Data augmentation was applied using ColorJitter, which adjusted brightness, contrast, saturation, and hue to enhance model generalization.

The images and their corresponding labels were preprocessed using Hugging Face’s SegformerImageProcessor, which resized and normalized the inputs before training. Training was conducted using the Hugging Face Trainer API, and the model was initialized with SegformerForSemanticSegmentation.from_pretrained("nvidia/mit-b1"). Performance was monitored via Intersection over Union (IoU) and accuracy metrics, computed using the evaluate library. The model’s predictions were interpolated using bilinear scaling, and performance was assessed per category to ensure accurate segmentation across all charger components.

Recognizing the need for improved segmentation performance, we transitioned to leveraging more powerful compute resources. One of our team members gained access to an NVIDIA A100 GPU, which enabled us to experiment with more complex models in the SegFormer family. We systematically evaluated MiT-B1 through MiT-B5, analyzing their respective parameter sizes, training times, and segmentation accuracy. After extensive testing, we found that the MiT-B3 SegFormer model struck the best balance between computational efficiency and performance, significantly improving segmentation quality without incurring excessive inference costs.

The best-performing model (MIT-B3) was automatically saved and uploaded to the Hugging Face Model Hub for future deployment. To evaluate segmentation accuracy, we applied a custom color palette to differentiate segmented components: red for screens, green for plugs, yellow for the body, blue for cords, and purple for the background.

Hugging Face repository for our final solidified SegFormer B3 model.

The figure above showcases the performances of B0 to B5 models on an existing dataset named ADE20K.
Link

Binary Classification: Model Training & Testing

After segmenting charger components, we trained a Vision Transformer (ViT) model for fault classification. Using vit-base-patch16-224-in21k, we categorized images into two classes: healthy (label = 0) and broken (label = 1). Dataset preparation involved standardization to RGB format, renaming for consistency, and labeling stored in a metadata.csv file.

Vision Transformers operate differently from traditional convolutional architectures by using self-attention mechanisms to process image patches. This approach enables the model to capture long-range dependencies within an image. Below is a graphical representation of how Vision Transformers work on the ImageNet dataset, demonstrating superior performance compared to conventional CNN-based models.

The figure above represents the Vision Transformer Architecture.
Link

The dataset was split into an 80-20 train-test ratio. Images were transformed using PyTorch's torchvision with preprocessing steps such as RandomResizedCrop, Normalization, and ToTensor. The ViT model was initialized using AutoModelForImageClassification.from_pretrained, with a data collator managing batch and padding operations.

Training was conducted via the Hugging Face Trainer API with the following hyperparameters:

Batch size: 2 per device
Learning rate: 5e-5
Epochs: 10
Gradient accumulation: 2 steps
Evaluation strategy: Step-based
Best model selection: Based on accuracy

The training process handled gradient updates, loss calculations, and backpropagation. Upon completion, the best-performing model was automatically saved and uploaded to the Hugging Face Model Hub for future deployment and fine-tuning.

Results & Impact

Our study demonstrated the effectiveness of artificial intelligence in automating EV charger fault detection. By using computer vision models, we successfully identified damaged chargers and categorized faults, enabling a more efficient and structured reporting system. This innovation improves maintenance response times, enhances charger reliability, and ultimately contributes to a more robust EV charging network.

Through rigorous testing, our segmentation and classification models accurately detected charger issues and performed well under diverse conditions. The results highlight the feasibility of integrating AI-powered fault detection into real-world EV infrastructure, reducing reliance on manual reporting and increasing accuracy in maintenance logs.

Segmentation Results

We tested different levels of NVIDIA’s MiT SegFormer models (MiT-B0 through MiT-B3) for EV charger segmentation. Our final MiT-B3 model achieved:

Mean Accuracy: 87.1% across all classes
Overall Accuracy: 93.5%
Mean IoU: 0.804

The model effectively distinguished between different charger components, despite challenges such as low-light environments and background clutter. However, segmentation accuracy can be further improved with additional data augmentation and fine-tuning on diverse datasets.

The figure above showcases the performances of B0, B2, and B3 models.

Classification Results

We used Vision Transformers (ViT) to classify charger faults, training separate models for screen damage, plug condition, cable defects, and out-of-service status. Our models performed well, particularly in high-contrast images, achieving:

High confidence in distinguishing healthy vs. broken charger components.
Reliable performance in well-lit environments.
Challenges in subtle damage detection, requiring additional labeled data.

This classification approach ensures granular fault detection, minimizing false positives and providing precise maintenance reports.

The figure above showcases the performances of our classification models on different charger components.

Discussion & Future Work

The study confirms the practicality of AI-driven fault detection for EV charging infrastructure. Compared to manual reporting (e.g., PlugShare user submissions), our models provide structured, real-time, and standardized fault reports. The segmentation model's high IoU score suggests potential for real-world deployment, provided additional calibrations are made.

Future improvements include:

Expanding dataset diversity: More data from various lighting conditions, weather environments, and charger manufacturers.
Active learning integration: Allow the model to refine its accuracy continuously by learning from new user-reported faults.
Mobile app integration: Enhancing real-world applicability by linking AI fault detection with user-friendly reporting tools.
Incoporating face and license plate blurring: To ensure the privacy of private information in user-submitted pictures, model will blur faces and license plates before data is collected.

By refining these models further, we aim to optimize EV charger maintenance, reduce charger downtime, and contribute to a seamless EV charging experience.

Prototype

Explore our interactive app prototype and product flowchart below:

FLARE product flowchart:

Team

Here are the FLARE team members who developed the project:

Ethan Deng

Jason Gu

Daniel Kong

Irving Zhao

Mentors

We would like to extend the most sincere gratitude to our wonderful SDG&E mentors who guided the FLARE team for the past two quarters:

FLARE