duplicate labels removed

[alicera]

Search before asking

• I have searched the YOLOv5 issues and discussions and found no similar questions.

Question

train: WARNING ⚠️ /work/images/train/coco2017.jpg: 1 duplicate labels removed

what does it mean?

Additional

No response

[glenn-jocher]

@alicera it means that your labels.txt file had two rows that were identical.

[varun-sappa]

I am still getting duplicate labels removed even though all are unique

[glenn-jocher]

@varun-sappa duplicate labels are typically removed during the training process to ensure that each label is unique and doesn't affect the model's performance. However, if you are still encountering the "duplicate labels removed" warning despite having unique labels, there could be a few possibilities to consider.

Firstly, ensure that there are no trailing spaces or other invisible characters in your labels.txt file. Sometimes, these characters can make labels appear identical even though they are technically different.

Secondly, check if there are any duplicate rows in your labels.txt file. Even if the labels themselves are unique, having identical rows can trigger the warning. Make sure each label is listed only once in the file.

Finally, if the issue persists, it could be a bug or an edge case specific to your dataset. In such cases, it would be helpful to provide more details about your dataset, along with a sample of your labels file if possible. Sharing these details will allow the community to better understand and assist you in resolving the issue.

Please let us know if you have any further questions or concerns.

[AloshkaD]

@glenn-jocher I've encountered the same issue with the warning message for duplicate annotations . Here's my code for creating the yolo segmentation files

def save_annotations_as_yolo_format(image_path, polygons, materials, types_, output_folder, tile_id, plot_normalized=False, image_polygons=None):
    yolo_file_path = os.path.join(output_folder, f"{tile_id}.txt")
    print("Saved annot", yolo_file_path)
    with open(yolo_file_path, 'w') as f:
        for polygon, material, type_ in zip(polygons, materials, types_):
            class_type = combine_strings(material, type_)
            material_type = class_id(class_type)

            normalized_polygon = []

            min_x = min(p[0] for p in polygon)
            max_x = max(p[0] for p in polygon)
            min_y = min(p[1] for p in polygon)
            max_y = max(p[1] for p in polygon)

            for point in polygon:
                normalized_x = (point[0] - min_x) / (max_x - min_x)
                normalized_y = (point[1] - min_y) / (max_y - min_y)
                normalized_polygon.append((normalized_x, normalized_y))

            formatted_normalized_polygon = format_polygon(normalized_polygon)
            #print('normalized_polygon',normalized_polygon)
            #print('formatted_normalized_polygon',formatted_normalized_polygon)
            f.write(f"{material_type} {formatted_normalized_polygon}\n")
            

and here's a sample output file

199 0.220973 0.000000 0.239669 0.002237 0.835735 0.725550 1.000000 1.000000 0.821371 1.000000 0.806133 0.998190 0.000000 0.000000 0.220973 0.000000
199 0.241407 0.000000 0.717216 0.606315 1.000000 0.997675 0.984511 1.000000 0.796916 1.000000 0.785156 0.998483 0.000000 0.002535 0.015988 0.000000 0.241407 0.000000

The file will trigger a warning for one duplicate, and the one below will trigger the warning for 7 removed

199 0.131608 0.000000 1.000000 0.869366 1.000000 1.000000 0.000000 0.000000 0.131608 0.000000
199 0.000000 0.612761 0.086143 0.552502 0.047212 0.020283 0.069342 0.000000 0.366297 0.494387 0.702481 0.770362 1.000000 0.933027 0.973900 0.943918 0.711228 0.820088 0.375366 1.000000 0.363775 0.974274 0.426032 0.899308 0.404281 0.838184 0.153428 0.602020 0.009681 0.666830 0.000000 0.612761
199 0.049568 1.000000 0.000000 0.977625 0.379472 0.518443 0.433879 0.015492 0.475246 0.000000 1.000000 0.309024 0.844882 0.371292 0.650428 0.679985 0.802811 0.804316 0.673624 0.802598 0.049568 1.000000
199 0.058544 1.000000 0.031052 0.966085 0.000000 0.277360 0.091591 0.305143 0.466699 0.117978 0.540390 0.000000 0.661030 0.601286 1.000000 0.745980 0.977984 0.793081 0.600377 0.684726 0.058544 1.000000
199 0.065334 0.000000 1.000000 0.936742 1.000000 1.000000 0.066411 0.070269 0.000000 0.000000 0.065334 0.000000
199 0.358318 0.393598 0.030872 0.071750 0.007039 0.062872 0.000000 0.020040 0.020024 0.000000 0.160316 0.138692 0.187824 0.146477 0.245612 0.204715 0.267102 0.242768 0.999584 0.973645 1.000000 0.987173 1.000000 0.997363 0.996581 1.000000 0.448487 0.466589 0.358318 0.393598
199 0.000000 0.937483 0.586922 0.335384 0.874710 0.084324 0.902308 0.038472 0.991552 0.000000 1.000000 0.059337 0.826987 0.187163 0.693817 0.324301 0.656508 0.401337 0.507243 0.541094 0.437591 0.568807 0.000000 1.000000 0.000000 0.937483
199 0.000000 0.000000 0.460324 0.199745 0.630402 0.310242 0.838582 0.568321 0.921608 0.728409 0.924032 0.808761 0.991831 0.920146 1.000000 0.999255 0.993491 1.000000 0.930130 0.960226 0.876606 0.823793 0.659827 0.429317 0.576522 0.344866 0.484739 0.263227 0.200145 0.134099 0.000000 0.064237 0.000000 0.000000

[AloshkaD]

I updated the normalization function to use the image corners and it worked.

old

def normalize_polygon_to_yolov8_format(polygon):
    # Find minimum and maximum coordinates of the polygon
    min_x = min(p[0] for p in polygon)
    max_x = max(p[0] for p in polygon)
    min_y = min(p[1] for p in polygon)
    max_y = max(p[1] for p in polygon)

    # Normalize each point of the polygon
    normalized_polygon = [(point[0] - min_x) / (max_x - min_x) for point in polygon]
    normalized_polygon += [(point[1] - min_y) / (max_y - min_y) for point in polygon]

    return normalized_polygon

new

def normalize_polygon_to_yolov8_format(polygon, image_width, image_height):
    normalized_polygon = []
    for point in polygon:
        normalized_x = point[0] / image_width
        normalized_y = point[1] / image_height
        normalized_polygon.extend([normalized_x, normalized_y])
    return normalized_polygon

[glenn-jocher]

@AloshkaD great to hear that updating the normalization function resolved your issue! 🎉 Your approach in the new code correctly normalizes each point relative to the entire image dimensions (image_width and image_height), which is key for consistent YOLO training data. This method ensures that all points are accurately represented on a scale from 0 to 1 relative to the image size, helping the model learn better. Keep up the good work, and if you have any more questions, feel free to ask!