organizing video files using dandi organize

[Saksham20]

This is a continuation of dandi/helpdesk#30

It would be more disk space efficient to store behavioral video files outside of an nwbfile since the compression is not at par with the codecs for mp4 etc. If there is a standard to organize video files and then link those to the nwb file as external_file argument, it would help.

If a user has a video file stored locally and an nwbfile that links to that local file on disc, uploading the nwb on DANDI would break the link. So we should come up with a design for organizing the nwb associated video files on DANDI and update the link stored in the nwbfile.

dandi organize for video files specifically should:

1. Rename the video files prior to uploading on DANDI.
2. Update the nwbfile in real time and change the external file link based on the new name of the video file.

@satra @bendichter I have created a design draft for us to collaborate on
https://docs.google.com/document/d/1oclxWFRNbakjYBlrze4DiFWkh_v0yVhmGjNar09skx4/edit?usp=sharing

[Saksham20]

As an example, I have created a script that takes in a folder containing nwb files and accompanying video files (linked with external_file tag in the ImageSeries container.
Here is an example folder that contains 6 video files (mp4). And here is the output with the created nwbfiles (with changed external_file arg)

Step 1: create the nwb files with video files as external args:

import shutil

import pynwb.image
from pynwb import NWBHDF5IO, NWBFile
from pathlib import Path


## create the nwbfiles:
from datetime import datetime

from dateutil.tz import tzlocal
from pynwb.device import Device
from pynwb.file import Subject
from pynwb.ophys import ImageSeries

down_loc = '/Users/sakshamsharda/Downloads' #path to browser downloads folder

video_base_path = Path(down_loc)/"dandi_video"/"video_files" # after unzipping gdrive downloaded file
dandi_base_path = video_base_path.parent/"DANDI"/"000999"


def create_nwb(name, no):
    subject_id = f'mouse{no}'
    session_id=f'sessionid{no}'
    subject = Subject(subject_id=subject_id, species="Mus musculus", sex="M",
                      description='lab mouse ')
    device = Device(f'imaging_device_{no}')
    nwbfile = NWBFile(f'{name}{no}', 'desc: contains movie for dandi .mp4 storage as external',
                      datetime.now(tzlocal()),
                      experimenter='Dr. Bilbo Baggins',
                      session_id=session_id,
                      subject=subject,
                      devices=[device])

    image_series = ImageSeries(name=f'MouseWhiskers{no}',
                               format='external',
                               external_file=[str(video_base_path/fr"video{no}.mp4"),
                                              str(video_base_path/fr"video{no+1}.mp4")],
                               starting_frame=[0], starting_time=0.0, rate=150.0)
    nwbfile.add_acquisition(image_series)
    nwbfile_path = video_base_path/f'{name}_{no}.nwb'
    with NWBHDF5IO(str(nwbfile_path), 'w') as io:
        io.write(nwbfile)

## organize:

def dandi_organize(nwb_object, symlink=True):
    """
    1. Renames the file_name based on the defined naming method
    2. Creates a symlink (with the same name) to the actual file on disk
    """
    nwbfile = nwb_object.parent
    subject_id = nwbfile.subject.subject_id
    session_id = nwbfile.session_id

    #create nwb symlinks:
    dandi_nwb_base = dandi_base_path/f'sub-{subject_id}'
    dandi_nwb_base.mkdir(parents=True, exist_ok=True)
    dandi_nwb_name = dandi_nwb_base/f'sub-{subject_id}_ses-{session_id}_image.nwb'
    if symlink:
        dandi_nwb_name.symlink_to(Path(nwb_object.container_source))
    else:
        shutil.copy(nwb_object.container_source, dandi_nwb_name)
        io = NWBHDF5IO(str(dandi_nwb_name), mode='r+', load_namespaces=True)
        nwbfile = io.read()
        nwb_object = [i for i in nwbfile.children if i.name==nwb_object.name][0]

    new_video_path = dandi_nwb_base/dandi_nwb_name.stem
    new_video_path.mkdir(parents=True, exist_ok=True)
        
    for idx, ext_file in enumerate(nwb_object.external_file):
        video_name = new_video_path/Path(f'{nwb_object.object_id}_external_file_{idx}.{ext_file.split(".")[-1]}')
        if symlink:
            video_name.symlink_to(Path(ext_file))
        else:
            shutil.copy(ext_file, video_name)
        nwb_object.external_file[idx] = video_name


def get_image_series(container):
    image_series_containers = dict()
    for name, ob in container.items():
        if isinstance(ob, pynwb.image.ImageSeries) and ob.external_file is not None:
            image_series_containers[name] = ob
    return image_series_containers


# main script:
# 1. Create teh nwb files:
for i in [0,2,4]:
    create_nwb('dandi_video_test', i)

# 2. Organize:
for nwb_path in Path(video_base_path).glob('**/*.nwb'):
    with NWBHDF5IO(str(nwb_path), mode='r+', load_namespaces=True) as io:
        nwbfile = io.read()
        nwbfile_containers = [nwbfile.acquisition, nwbfile.processing]
        for data_interface in nwbfile_containers:
            for name, image_series in get_image_series(data_interface).items():
                if image_series.external_file[0].split('.')[-1] == "mp4":
                    dandi_organize(image_series, symlink=False)
        io.write(nwbfile)

Essentially it performs 2 functions:

1. Renames the video files and places them in a sub-folder with the same name as the nwbfile. Its renames the file with the uuid of the parent ImageSeries container + 'external_file_<video_no>.mp4 :
   <nwbfile_name>/{UUID}_external_file_0.mp4
2. Updates the external_file arg in all the nwb files.

[bendichter]

If you need to go multiple levels deep in a relative path, replace "/" with "__"

[Saksham20]

If you need to go multiple levels deep in a relative path, replace "/" with "__"

yes I agree, using double underscores is a better separator.

However, currently, the way the files are named do not require separators in the name: <objectUUID>_external_file_<no>.mp4 is how the video file is renamed; its placed in a folder with the same name as the parent nwbfile and location is next to the nwbfile.

[bendichter]

Yes I was just thinking how we would support this if it came up in an extension

On Fri, Nov 12, 2021 at 8:39 PM Saksham Sharda ***@***.***> wrote: If you need to go multiple levels deep in a relative path, replace "/" with "__" yes I agree, using double underscores is a better separator. However, currently, the way the files are named do not require separators in the name: <objectUUID>_external_file_<no>.mp4 is how the video file is renamed; its placed in a folder with the same name as the parent nwbfile and location is next to the nwbfile. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#785 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGOEERUYULJ7MMZK4A25ZDULXFW5ANCNFSM5FMFJSBA> .

-- Ben Dichter, PhD Data Science Consultant personal website <http://bendichter.com/>