Cleaning data from deleted competitions

[the1mag]

Hi, we are looking to clean up our local instance of codabench. I see that if I delete a competition and then delete all orphan files, the submissions related files are successfully removed from the django admin as well as from s3, but the competition bundle, reference_data.zip, input_data.zip, scoring_program.zip, ingestion_program.zip still remain in django and in s3. Additionally in the django admin as well the Columns, Leaderboards, Submission scores, and Tasks created from that deleted competition still remain. I also noticed that logos from deleted competitions are still stored in s3. Are there existing methods to prune these associated items from the deleted competition so they aren’t in the django database and s3 anymore?
Thank you

[Didayolo]

This is a very good point that may need some discussion and investigation.

About deleting orphan files, you have used the option from the resources interface https://www.codabench.org/datasets/ ?

[Didayolo]

Hi @the1mag,

Reading again this issue, I realized I got confused when I answered.

And actually we do have a feature to remove orphan files (data without competition, etc.).

It is in "Analytics" page:

Can you please try it and tell us if it actually resolves your problem?

Thanks!

[the1mag]

Hi,

In my above issue I was already using the orphan file deletion feature on the Analytics page. I ended up adding some custom code to delete the additional leftover resources.

[Didayolo]

Hi,

In my above issue I was already using the orphan file deletion feature on the Analytics page. I ended up adding some custom code to delete the additional leftover resources.

Thank you for the update. Could you please share the code you used, to help us solving the problem on the platform?

[the1mag]

Hi,
I had previously tested with our local instance what files/resources remain in s3 and django admin following deletion of a competition and running the delete_orphan_files task on the Analytics page. I found that the competition bundle, reference_data.zip, input_data.zip, scoring_program.zip, ingestion_program.zip still remain in django and in s3. Additionally in the django admin the Columns, Leaderboards, Submission scores, and Tasks created from that deleted competition still remain. I also noticed that logo images from deleted competitions are still stored in s3. I'm not sure for your use case if you want to delete all of those files as I know some bundles or data objects can be reused, but for our use case we did not want any remaining files that weren't connected to a competition. Because of that we also added code to handle deletion of data files that were manually uploaded to the site and not connected to a competition.

I'm not sure if this was the most optimal way, but following the existing code in the delete_orphan_files() task in tasks.py I added this code:

#Delete task objects that have no associated phases, indicating parent competition was deleted
orphaned_tasks = Task.objects.filter(phases__isnull=True)  
task_count = orphaned_tasks.count()  
orphaned_tasks.delete()  
total_orphans_deleted += task_count  
logger.info(f"Deleted {task_count} orphaned tasks") 
  
#Delete Data objects created by competitions that were deleted (deletes from django and s3) 
orphaned_data = Data.objects.filter(  
    competition__isnull=True  
).filter(  
    Q(task_ingestion_programs__isnull=True) &  
    Q(task_input_datas__isnull=True) &  
    Q(task_reference_datas__isnull=True) &  
    Q(task_scoring_programs__isnull=True) &  
    Q(phase_public_data__isnull=True) &  
    Q(phase_starting_kit__isnull=True) &  
    Q(submission__isnull=True) &  
    Q(competition_bundles__isnull=True) &  
    Q(competition_dump_file__isnull=True)  
)  
data_count = orphaned_data.count()  
for data in orphaned_data:  
    data.delete() 
total_orphans_deleted += data_count  
logger.info(f"Deleted {data_count} orphaned data objects")  

#Delete competition bundle Data objects created by deleted competitions (deletes from django and s3)
orphaned_bundles = Data.objects.filter(  
    type=Data.COMPETITION_BUNDLE  
).filter(  
    Q(competition_bundles__isnull=True) |  # Status object exists
    Q(competition_bundles__resulting_competition__isnull=True)  
)
bundle_count = orphaned_bundles.count()
for bundle in orphaned_bundles:
    bundle.delete()  
total_orphans_deleted += bundle_count
logger.info(f"Deleted {bundle_count} orphaned competition bundles") 

#Delete Leaderboards with no phase from django database
#Note: deleting leaderboard will then delete associated columns
orphaned_leaderboards = Leaderboard.objects.filter(phases__isnull=True)  
leaderboard_count = orphaned_leaderboards.count()  
orphaned_leaderboards.delete()  
total_orphans_deleted += leaderboard_count
logger.info(f"Deleted {leaderboard_count} orphaned leaderboards")  

#Delete logo files from deleted competitions
#Get all logo files in storage  
logo_files_in_storage = [obj.key for obj in PublicStorage.bucket.objects.filter(Prefix="logos/")]  
#Get all referenced logos from database  
referenced_logos = set()  
for comp in Competition.objects.all():  
    if comp.logo:  
        referenced_logos.add(comp.logo.name)  
    if comp.logo_icon:  
        referenced_logos.add(comp.logo_icon.name)  
#Find orphaned logos  
orphaned_logos = set(logo_files_in_storage) - referenced_logos  
logo_count = len(orphaned_logos)  
#Delete orphaned logos 
for logo in orphaned_logos:  
    PublicStorage.bucket.delete_objects(Delete={'Objects': [{'Key': logo}]})
total_orphans_deleted += logo_count  
logger.info(f"Deleted {logo_count} orphaned logos")    


logger.info("Delete orphaned database resources completed.")
return {"total_deleted": total_orphans_deleted}