Possible solutions to optimize cpu load for HLS

[zhanghuicuc]

Hi,
As mentioned in #1227 and #1412, Loader:HLS thread cosumes a lot of CPU resources. We have used ARM Streamline to analyze the CPU activity and got the result as follows:

We can see that core 1 and 2 can be more than 60% busy sometimes. On some low performance cpus like Mstar6A938, this can lead to a large amount of dropped frames especially when playing 4K high bitrate streams. The same issue happens on both 1.5.x and 2.x.

Possible solutions to optimize cpu load for HLS may be a) Lower the thread priority of Loader:HLS and b) Load and extract ts packets concurrently.

a)

changeThreadPrio.diff

In this way, OMX may have more resources to do decode related work. The dropped frame numbers can be reduced by 60% according to our 3 hours long time test results. And we have not found any side effects so far.

b)
loadAndparseConcurrently.diff

Loading several ts packets each time can reduce the number of IO operations. By doing so, we need to load and extract ts packets concurrently to prevent the player from getting stuck in buffering state. The cpu activity now is as follows:
(loading 50 packets each time)

(loading 100 packets each time)

We can see that cpu load has been reduces a lot and dropped frame numbers is also reduced.

Maybe switching to fragmented mp4 or DASH is the best solution, but there still exists a lot of HLS-TS streams we need to deal with. I'd appreciate any thoughts on these possible solutions and I'm happy to prepare a PR for any potential fix.

Thanks!

[AquilesCanta]

Would you kindly provide us with the streams you are using to test resource consumption? It would be good for us to know the characteristics of the streams.

[zhanghuicuc]

I simply used Bento4 mp42hls to convert tears_h264_high_uhd_30000.mp4 in https://storage.googleapis.com/wvmedia/clear/h264/tears/tears_uhd.mpd to a HLS-TS stream. The m3u8s are as follows:
master.m3u8
stream.m3u8

The stream is on our internal server right now and I'm afraid it can't be accessed from outside.
I've uploaded some of the media segments to google drive
https://drive.google.com/file/d/0B2c0s1M2EOS7aXFRRDFxTG42QTg/view?usp=sharing

[dbedev]

Think these are some valid points by zhanghuicu on exoplayer's HLS capabillity. Also the proposed enhancements to the exoplayer way of (HLS) operating look very interesting.

Do think that at the root of these HLS cpu peaks there is a more generic choice in the way the default exoplayer data loading operates. It works in a peak pattern that tries to fetch data as quickly as possible with reltatively large intervals (15s). And thus you get these kind of cpu utilization peaks at given intervals for Ts based HLS.

see #2083

As far as I understand this loading approach is geared towards energy efficiency for battery operated devices (also taking into account also the energy for the network tranfser) . For a none battery powered tv / media box solution like the Mstar6A938 one probably is more concerned just about the keeping the cpu clock low and having as little framedrops as possible. As a result one would instead want to spread the loading over time to create a better visual performance.

The cpu utilization side effect of the default loader choice grows with the bitrate as as the loader works in the time domain only. So for a 30mbit stream by default it's trying to load and parse 15 seconds of data (56mb) as quick as possible. Ironically this also means that the higher you're connection speed to the server, the more peak utilization the player will put on one of the cpu's. Which in turn can cause scheduling issues on the system the player runs on.

I think that adapting the loader priority and doing concurrent Ts loading would definitly improve experience for an externally powered device as both methods will effectively counter the default loader cpu utilization side effects as the bitrate increases. Not sure though how the splitting of extracting , lower prio will affect the energy consumption though. Maybe the exoplayer team can do some tests for this based on a nexus device. Also not sure how much the splitting will affect the adaptive selection/bandwith measurement.

Seeing the increasing use of utilizing exoplayer in AndroidTv style devices, maybe it is an idea to develop different load controls for none battery operated devices or make the default one adapt to higher bitrates as are more common for (high quality) tv oriented delivery.

Do think that the current BUFFER_PACKET_COUNT setting (5)in TsExtractor.java , is definitely too small. Seems that this should be a relatively simple improvement, is there any reasoning to why this is set to 5 ?

[ojw28]

Agreed there are some valid points of investigation, and thanks a lot for the analyses. A few thoughts/comments:

• It's worth noting that the rendering thread already has a higher priority than the loading threads (aside: this was broken until 2.4.0, where it was fixed in ecb62cc, so if you're using an older version then updating may be an easy way to see some improvement). Although I agree there may be a valid argument for also lowering priority of the loading threads too.
• If we can better batch up work in order to reduce load then that seems like an easy win. I'm not sure about the proposed patch, however, which would appear to generate a lot of garbage that will be really bad for devices running Dalvik. It also adds an additional memory copy. It's unclear what the cost trade-off is there, and we already have more copy steps through the TS extractor than in other extractors. Does simply increasing BUFFER_PACKET_COUNT (and not doing anything else) yield a performance increase? If so then that looks like something we should just go and do now. If someone could measure the improvement of making just that change, that would be great. @zhanghuicuc - Is that something you could easily get data for, given your existing setup?
• I think fragmented MP4 is the proper long term solution to this problem (whether sticking with HLS or moving to DASH).
• The initial post mentions there's a lot of existing TS content, that there's existing 4K content, and that there exist low powered devices. Whilst all are definitely true, I'm not sure the combination (which is when this is most problematic) is particularly common as a percentage of ExoPlayer usage, and it's unclear whether it ever will be as major content providers shift to using fragmented MP4 instead. This weighs somewhat on what we can reasonable do here (e.g. it doesn't make sense for us to make changes to improve this case that would negatively impact other more predominant uses).

Thanks!

[ojw28]

It would also be a good idea to profile the extractor to see if there are any hotspots that we can optimize. If we're able to make it cheaper, that's going to be a better solution than spreading the cost (e.g. by lowering thread priority) :).

[ojw28]

For looking at increasing BUFFER_PACKET_COUNT, it would be good to measure performance with the current value of 5 compared to a proposed new value of 100, with no other changes. It looks like reads stop being satisfied in full when you get over about 80, at least using the default network stack, so there's little to be gained from going beyond 100 I think. If we get confirmation it makes a significant improvement, we'll make that change.

[zhanghuicuc]

a)
Simply increasing BUFFER_PACKET_COUNT to 100 (and not doing anything else) can not make a significant improvement. The test result is as follows, and the first image in my initial post is the test result when BUFFER_PACKET_COUNT = 5:

However, if we set BUFFER_PACKET_COUNT to a much larger value, like 250, the test result is as follows:

And the side effect is that the player comes into buffering state more often, as I've mentioned before.

It looks like reads stop being satisfied in full when you get over about 80, at least using the default network stack, so there's little to be gained from going beyond 100 I think.

I don't understand the meaning of "reads stop being satisfied in full " and where does this "80" come from?

b)
I will try to remove the additional memory copy in my proposed patch, maybe using a ring buffer. And I'll try to profile the extractor to see if there are any hotspots.

Thanks for your valuable comments!

[ojw28]

• Could you explain how we're supposed to be interpreting the images you've attached. I'm not really sure what I'm looking at :). Ideally, if you could provide numbers instead (e.g. dropped frames per minute), that would be a lot easier to interpret/compare.
• To answer the question about "reads being satisfied in full": The input.read calls are not guaranteed to read the full amount of data requested. The return value is the amount actually read, which may be less (but is at least 1 byte), just like how InputStreams work. If the idea of increasing BUFFER_PACKET_COUNT is to make fewer, larger reads from the input (and hence from the underlying network stack), this will only have a positive effect up to the point where we're requesting as much data as the underlying network stack can actually fulfill in a single call. Increasing beyond that point means we request even larger reads, but the amount actually read will stop increasing, and so there's no actual benefit to doing this. For DefaultHttpDataSource the point where it stops making a difference is around 80, at least on a test device I have.

[dbedev]

Not sure if I managed to get the message across properly before.

The current exoplayer implementation requires a single cpu core to parse up to 15 seconds of container data around ~15s intervals. It will do this at the "maximum" speed at which the data source can provide it and it assumes this will not cause a task scheduling conflict with other related systems tasks on the system.

If there is a clash with another task that happens to be the audio/video OMX decoder (spawned by the driver/os layer) one will end up with a frame drop eventually. So the issue seems to be about peak cpu loads not cpu load in general.

I would argue that this issue is not just related to limited cpu processing power of a device, but more about assuming a relation of cpu processing power vs the device's network IO throughput. With the current loader approach the quicker the network IO the more computation power is required from the device in order to avoid a potential scheduling issue. So if not dealt with, this issue will only worsen for existing devices over time as connections speeds to servers will probably improve, but the device cpu processing power will not.

Ts streams are the obvious candidate to hit it first as they are the most expensive to parse, but possibly encrypted containers or streaming over https(heavy cipher suite) for different formats would eventually hit similar behavior when increasing their bitrate.

1. Setting a higher priority on the playback tasks should avoid most of the scheduling clashes between the exoplayer playback task and loading task. However I have encountered multiple devices where some of the OMX decoding work is actually done on separate task on the OS, so the OMX tasks would still be in conflict with the loader. So the proposed solution of reducing the priority looks like a more effective way to counter this.

2. If possible improving the Ts parsing speed will improve things in general (it makes ts more like the others). Effectively it will however just move the maximum bitrate figure at which this issue occurs. Looking at the beautiful graphs there is no real overall cpu load issue as there is a big amount of free time on all of the cpu's.

3. Setting BUFFER_PACKET_COUNT of 5 looks rather inefficient based on basic network/cpu operating knowledge. Increasing it might actually not improve much in this scenario or at some point even worsen the situation due to increasing the effective peak IO throughput.

4. The plots also seems show a peak of cpu usage at the start of loading of an assumed 15s segment. Could this be to previously unprocessed but already loaded data in some cache causing a peak in IO throughput ? Or is there some computational heavy part at the start of loading ?

In general for our project we did not find a more effective approach worth investigating than tuning the exoplayer parameters to our intended use until we managed a reasonable performance. Which off coarse goes directly against the exoplayer's purpose of providing abstraction.

We came to a value of 20 for BUFFER_PACKET_COUNT in our expoplayer 1.5.x based experiment. Together with avoiding the android 4.x OS cipherInputstream mess. However in our experiment we use HLS streams that typically do not go beyond 15mbit (with optional AES envelope encryption) and we have a fairly powerful cpu at hand.

We did not need complete drop less playback performance at that moment, near fluent was good enough. Drop less would become more important once we can have something like dynamic HDMI output rate switching. Is there any form of support for this in Exoplayer v2 in combination with recent android versions?

[ojw28]

• We're on the same page; I understand your points :). And yes, we'll take a look at the thread priorities.
• The point of my first bullet above is that it's really hard for us to look at screenshots of two graphs and derive meaningful data from them. So it would be much better if you could provide numerical data instead. For example "we see 5 dropped frames per minute with priority X vs 50 dropped frames per minute with priority Y" directly measures user impact and the numbers are trivial to compare :).
• I think there's a lot of low hanging optimization fruit in TsExtractor (and possibly in other extractors too). I just pushed some changes that tackle some of the immediately obvious bits. It would be interesting to know whether you see a measurable difference.

[zhanghuicuc]

OK, I'll do some tests and let you know once I've got any result.