Reduce cache initialization and operation times

[kvillnv]

Hi,
I am working on Nvidia Shield, with and external USB 3 storage, the storage is set as "This device's storage", the apps data is moved to the storage.
I am downloading DASH streams, using DownloadService and DownloadManager.

There is performance issue because of the quantity of files generated by exoplayer Downloaders.

After having downloaded a few GB of data, I now have thousands of .exo files in my download folder. Since then, everytime I boot my device when the external device is mounted, I can see the process /system/bin/sdcard is taking up to 40% of cpu for a while. It's probably indexing all these files.

Same thing when I start my app, the first time I will play a media: the same android indexation process will start and affect my device perfomance, my media will only play after a few seconds.
Once it is indexed every other media will play instantly.

Moreover, exoplayer's actionFile already has its own indexation. So this heavy system indexation process does not really help exoplayer.

Handling thousands of files on the storage is really tough for the system.
Is there any way to optimize that?

Once a media is downloaded, do you have a way to merge or archive all it's .exo files in just 1 file?
Or maybe there is another solution, please advise.
Thanks

[erdemguven]

Which directory do you use for downloading? If you use the dir returned by Context.getExternalFilesDir(), I think it shouldn't be indexed by the system.

[kvillnv]

Yes, I am using Context.getExternalFilesDir(null), just like in the demo.

Now I am thinking about creating a new folder for each media, this should help the file system. But I will lose the ability to make multiple downloads, because it forces me to instantiate a new download manager everytime.
It seems the download manager is made to work in only one folder for all simultaneous downloads.
But do you think there is a way to make the manager work with simultaneous downloads in separated folders?

[erdemguven]

If the issue is the system is scanning too many files, even if you put each download to a different folder, the total number of files, won't be changed. If the issue is a single folder has to many files, than it might help.

You're right DownloadManager/Cache works with a single folder. If you want to use multiple folders then you need to create multiple DownloadManagers and Caches. Also for playback you need to use the right Cache instance.

Could you try putting an empty file which is named ".nomedia" in to the parent folder of the cache folder. Not in to the cache folder as SimpleCache would delete it when the app starts.

One more thing, please provide the information requested in the issue template as much as possible.

[kvillnv]

@erdemguven ,

If we cannot reduce the number of files and if creating multiple folders won't help the system, I believe it could at least help the SimpleCache and the player launch faster.

Yesterday I did try the .nomedia file, but in the cache folder and in order to avoid to have it deleted before android indexation, I created it each time I start y service and each time I initiate my cache, but I was still not sure the file would be deleted before android indexation process. I did it that way because I was not sure creating it in the parent folder would affect the subfolder.
Anyway, It did not help the sdcard process, maybe it only affects the android.media process.
I'm going to try it in the parent folder, to make sure it is never deleted, and I will let you know what happens.

Otherwise:
About using multiple folders, instead of creating multiple downloadManagers, I was wonderding if I edit DownloadService to launch each DownloadAction with a new DownloadConstructorHelper (new cache instance and folder), Do you think it would work this way or something else will break?
It means I would still have only one DownloadManager with one action file and subfolders in my cache folder.

Thanks for your help.

Issue Template:

Issue description

Android file system performance issue because of too much .exo file in the DownloadManager's cache folder. the process "/system/bin/sdcard" consumes a lot of ressources at each android boot and each cache initialization (after force killing and restarting the app for example)

Reproduction steps

Describe how the issue can be reproduced, ideally using the ExoPlayer demo app.

Link to test content

The content is downloaded on the local storage. it's Dash Streams with 4 second segments and Drm encryption.

Version of ExoPlayer being used

2.8.0

Device(s) and version(s) of Android being used

Nvidia Shield, Android 7.0

[erdemguven]

Perhaps you can easy try if multi folder improves sdcard issue by manually moving files to different folders.

[kvillnv]

And about SimpleCache, is there a way to generate an index for each media? so that the SimpleCache instance won't have to seek in all the cache

Are the methods getKeys() and getCachedSpans(key) what I need?

For my use case, is that SimpleCache instance appropriated?
SimpleCache(downloadDirectory, NoOpCacheEvictor())

about the .nomedia file: I retested in the parent folder, it does not help the sdcard process

[kvillnv]

Multiple folders is the solution to this bug.
The android system/bin/sdcard process stopped taking all my device's resources, and the performance has improved when having a lot of data.
Now that I store one media per folder, when I want to play a video from one of my folders, the Cache instance and the player initialization are instantaneous, even just after boot and force kill/relaunch.

@erdemguven I didn't need to create multiple DownloadManagers, I wanted to keep the DownloadService to stay as functional as before.
The best solution I found is to generate a new DownloadConstructorHelper for each download, it does not change a lot the initial code, it would probably be great to have this in the next versions.
Having multiple folders implied to also have multiple cache instances.

Details:
I have modified the DownloadManager and DownloadManager.Task : Now instead of passing a DownloadConstructorHelper into the DownloadManager, I pass my Parent Folder as a File object, and in the Task I generate a new child folder and generate a cache instance by calling a static synchronized getCache() method and then I build a specific DownloadConstructorHelper.
My getCache() method is in My Service companion object, I also use it for playback: it checks and stores cache instances in a HashMap. This way, from the id of a media I can check if a cache is already initialized in its folder (for example if I download and watch at the same time).
The DownloadService still works the same way, I still have one actionFile.

[erdemguven]

Thanks for investigating this. I'll look into this and decide to use either a solution similar to what you have done or modifying SimpleCache to place each media under a separate folder.

Sorry, I just noticed, I forgot to recommend another way to reduce the number of exo files. You can increase maxCacheFileSize when you create CacheDataSource. By default it's 2MB so downloaded media is divided to exo files max 2MB size. Passing a bigger number will make it create fever exo files. Problems with a bigger exo file size are, increased risk of losing more data in case the app crashes and you won't be able to read the currently downloaded exo file until it reaches to the max size or the end of stream.

[kvillnv]

@erdemguven I followed your recommendation and changed the maxCacheFileSize to 20MB for testing, but it seems it cannot help because of my source's segment size:
-My source has short segments of 2.5MB. When I set the maxCacheFileSize to 20MB the downloader does not build files bigger than 2.5MB.
-It only works in the other way: If I set the SIZE to 1MB, all my downloaded files will be 1MB instead of 2.5
-And I also tested with lower bitrate media, all my downloaded files are never bigger than 500KB.

So the current code can divide chunks bigger than maxCacheFileSize, but it cannot combine chunks smaller than it.

Is there anything else I can do to force building bigger files?

I also noticed something else: in my download folder I always have all my chunks of 2MB or so and I also have as much files of 100KB~200KB what are they? is it audio? (they also are .exo)

[erdemguven]

Unfortunately, there is no way to combine segments in to a single file.
Small exo files are probably the audio track. Also if your video segments are 2.5MB, there must be two exo file per segment when maxCacheFileSize is 2MB: 1 2MB file and 1 0.5MB file. So increasing the maxCacheFileSize to > 3MB still helps slightly.

[kvillnv]

Alright, thanks for your help @erdemguven.
In a next release, it would be great to make editable the maxCacheSize of the buildCacheDataSource of the DownloaderConstructorHelper.
And also use one folder per media.

[workspace]

@kvillnv Thank you for your tips. By the way, it is able to apply maxCacheSize to DownloaderConstructorHelper by inject custom CacheDataSinkFactory.