Handbrake utilizing 50% or less i7 CPU when encoding low-res x264?

TJCS

Senior member
Nov 3, 2009
861
0
71
I was trying some 2-pass x264 mp4 encoding with avi/mkv files using latest handbrake 0.9.4, and was trying to figure out why handbrake was only using 50% or less of my i7 CPU (sometimes as low as 34%).

After few trials, I found out that this only happens only when I encode in low-res (320x128). When i encode in higher resolution (ie: 1280x534) it will shoot up to 100%.


Question:

Shouldn't Handbrake utilize the CPU to work at 100% regardless what setting I choose for encoding? Is there is some setting that I am missing out?

Notes:
• I am encoding with the i7 machine that I listed in my sig
• I currently have HT off
• I tried Ripbot using the similar setting and get same 50% encoding results for low-res encoding.
• I move the encoding file to a faster partition to ensure it's not a disk read problem.
 
 

TJCS

Senior member
Nov 3, 2009
861
0
71
Seems like something is not sending data fast enough to the CPU. My CPU utilization increased to 75% after I changed:

• Mixdown from Dolby Pro Logic II to Stereo
• Samplerate from 48k Hz to 44.1k Hz

Is my onboard RealtekHD causing a bottleneck? This doesn't make sense to me because this bottleneck behavior only happens at low-res encoding, high-res encoding CPU is working @100%.

Update: Encoding Tests @ 320x128, Video Bitrate@192kbps:

• x264, Audio bitrate 96k @48Hz, Dolby Pro Logic II: 170~200s fps, CPU @ 50% load
• x264, Audio Bitrate:96k@44.1Hz, Stereo: 150~170s fps, CPU@ 75% load
• Mpeg4, Audio Bitrate: 96k@44.1 Hz, Stereo: 160~170s fps, CPU @ 50% load
 
Last edited:

0roo0roo

No Lifer
Sep 21, 2002
64,795
84
91
two pass high profile seems to really suffer when reencoding an avi on an x4.
its like 25% across 4 cores for the first pass, gets better on the second pass and hits 100% but the first takes forever.
not sure i trust the constant rate quality though alternative, i'll look into it.
 
Last edited:

erwos

Diamond Member
Apr 7, 2005
4,778
0
76
I noticed you wrote "partition". Unless you're reading and writing to two different physical hard drives, you're going to be IO bound far more by using a single drive than two separate drives.
 

0roo0roo

No Lifer
Sep 21, 2002
64,795
84
91
not sure thats true, hardly any cpu is going to be encoding so fast its io bound.
 

vshah

Lifer
Sep 20, 2003
19,003
24
81
disk issues wouldn't explain why he sees more CPU usage while working with higher res files.
 

0roo0roo

No Lifer
Sep 21, 2002
64,795
84
91
foobar encodes 4 separate files, now that flies. now if only handbrake had the option of spawning 4 separate encodes.
 

vertigo_2_20

Junior Member
Apr 20, 2010
4
0
0
TJCS said:
Seems like something is not sending data fast enough to the CPU. My CPU utilization increased to 75% after I changed:

• Mixdown from Dolby Pro Logic II to Stereo
• Samplerate from 48k Hz to 44.1k Hz

I'm more inclined to believe that changing these settings simply made the encode/transcode require more work, not necessarily more efficient.

TJCS said:
• I currently have HT off

According to benchmarks I've looked at and research I've done, HT significantly improves encoding speeds. Unless you're also a big gamer, in which case HT can often have negative effects, you might want to enable it.

TJCS said:
• I tried Ripbot using the similar setting and get same 50% encoding results for low-res encoding.

This tells me it's probably something with the codec (264) and not the encoder (HB).

0roo0roo said:
not sure i trust the constant rate quality though alternative, i'll look into it.

From what I've read and from my own testing, CQ seems like a huge improvement over single/double pass CBR encoding.

vshah said:
disk issues wouldn't explain why he sees more CPU usage while working with higher res files.

I just noticed this problem, which brought me to this thread. I've established it's definitely not a hard drive bottleneck, as I'm getting the same fps whether the source file I'm transcoding is on a USB hard drive, where the USB bandwidth is the bottleneck, or on my SSD. I'm fairly certain my RAM isn't the bottleneck (6GB @ 1532MHz, 8-8-8-24-1T). Yet strangely enough, when the output resolution is low, HB/264 refuses to max out the CPU.

I used the same source file twice, same settings, only one run I left the output resolution at 1280x548 and the other I lowered it to 640x272. The first run it held about 85-90%+ CPU usage, whereas the second run it only maintained ~50-65%. So that tells me without a doubt what the problem is, I just can't for the life of me think of why. It's obvious the lower resolution takes less processing power (exponentially actually), so one would think it would maintain the same CPU usage and just finish that much quicker. For whatever reason, that's not the case.

0roo0roo said:
now if only handbrake had the option of spawning 4 separate encodes.

While you can always simply launch multiple instances of HB to accomplish this, it would be nice to have it automatically do it, not only for simplicity but for less clutter, having one GUI run multiple CLIs, or better yet, integrate the CLI into a tab in the GUI. IMO, the less windows open, and therefore the less clutter, the better.
 

0roo0roo

No Lifer
Sep 21, 2002
64,795
84
91
From what I've read and from my own testing, CQ seems like a huge improvement over single/double pass CBR encoding.
While you can always simply launch multiple instances of HB to accomplish this, it would be nice to have it automatically do it, not only for simplicity but for less clutter, having one GUI run multiple CLIs, or better yet, integrate the CLI into a tab in the GUI. IMO, the less windows open, and therefore the less clutter, the better.

theres no way io is a bottleneck since any encoder worth a damn is reading chunks to work with at a time, if io bottlenecked us we'd be encoding as fast as the file would read, which would blow our minds;)

constant quality is better than 1 pass, but it is still making guesses which makes it unreliable for a certain file size. if you cbr to a certain quality then use that files average bitrate as the starting point for a 2 pass abr encode, the abr 2 pass should be superior since it can adjust once it takes account of the whole files needs, whereas the constant quality can't see that far ahead.

not sure multiple instances work, i tried and it died, guess i'll try it again later.
 

vertigo_2_20

Junior Member
Apr 20, 2010
4
0
0
0roo0roo said:
theres no way io is a bottleneck since any encoder worth a damn is reading chunks to work with at a time, if io bottlenecked us we'd be encoding as fast as the file would read, which would blow our minds;)

Right, as I proved by trying it from my SSD. So the question is, what is the bottleneck, or more accurately, what is the reason it won't use all available resources at lower resolutions. Obviously if it can handle higher resolutions, it's not going to magically experience a bottleneck that wasn't there for that when doing easier work, so it's something else. My guess is it's something with the design of the codec. Whether or not they know about it, who knows. I'm gonna try one of the x264 builds and see if that has the same problem or if it's just H264.

0roo0roo said:
constant quality is better than 1 pass, but it is still making guesses which makes it unreliable for a certain file size. if you cbr to a certain quality then use that files average bitrate as the starting point for a 2 pass abr encode, the abr 2 pass should be superior since it can adjust once it takes account of the whole files needs, whereas the constant quality can't see that far ahead.

I'm not sure you understand constant quality. It makes no guesses. It simply goes through and looks at each frame individually, determining based on the complexity of that particular frame and the difference between it and the frame before and after it (I may not be 100% accurate here, but the general idea is sound) how many bits to expend on encoding it. This allows it to spend only as much data as necessary on frames that require very little, while spending as much as necessary on more demanding frames/scenes that demand it to maintain a high quality. This way, you avoid having to do 2 passes, so it's quicker, and you get a set quality that will be equivalent to a 2 pass, if not better. So there is no need to "see that far ahead," as it deals with each frame and scene on a case by case basis.

Yes, this does result in an unpredictable file size, but generally I have noticed with the efficiency of 264, they are usually quite small, except for more demanding videos, in which case I don't mind them being larger, as that means it's better quality. And as cheap as hard drive space is these days, what's another 25-50MB per movie if it means quality. Not to mention a lot of times with lower quality videos (cartoons, black and white, et cetera) it'll end up actually being smaller than if you just chose a random target size.

Basically, CQ is the same thing as 2-pass, only it takes just one pass (and is therefore much, much quicker and simpler) and allows for the resulting file size to be smaller or larger depending on the needs of the particular video, not just a randomly selected number that's always going to be a compromise. It's the same thing as VBR (vs CBR) for mp3's, which is a very old concept, and it's surprising it took so long to carry over into the realm of video.

0roo0roo said:
not sure multiple instances work, i tried and it died, guess i'll try it again later.

I'm fairly certain it can be done, but I think you have to use the CLI directly, the GUI isn't capable of it. I've never cared to do this, so I've never bothered to learn how. The HB mods claim there's little to no benefit, and while I mostly agree, I've read that 264 does have some diminishing returns when using more cores (e.g. 100% speed for 1 core, 190% for 2, 275% for 3, 360% for 4, or something to that effect). According to that, it would be faster to run multiple instances, but I don't think it's worth the extra effort, the extra clutter, and the extra hard disk access. If you're reading and/or writing numerous files at once, you're stressing the hard drive a lot more, though I'm not sure if it's enough to be concerned about, since it's not really that demanding. Ultimately, for the small potential gain of running multiple instances, I don't think it's worth it.

I also just noticed they have an option in 0.9.4 to "Enable in-GUI encode status (experimental)." I'll have to try that out.

Edit: I just tried the in-GUI status and I'm not exactly impressed. It still launches the CLI window, just doesn't show anything in it and it shows the status in the very bottom of HB instead. Pretty useless IMO. Now if they had it display in the title bar, so you could see the % complete on the taskbar, that'd be something.
 
Last edited:

0roo0roo

No Lifer
Sep 21, 2002
64,795
84
91
No, i still don't think cqr is good as two pass at the same abr, cqr uses a crude assessment of quality at a instant to make its decision, which is good enough if file size is no concern, but its inability to take account fo the entire files bitrate needs means it cannot adjust a best fit to the bitrate that it did use. its basically one pass vbr vs 2 pass vbr.

supposedly some filters or settings are not multithreaded, or poorly so.

i've tried installing a second instance of handbrake, if i run the second the gui works, hitting encode start causes failure.
#%@
 

0roo0roo

No Lifer
Sep 21, 2002
64,795
84
91
don't use the in gui experimental, its got other issues, if you pause and start the queue it breaks, it doesn't give you feedback either when you pause the encode box with a "p"
 

vertigo_2_20

Junior Member
Apr 20, 2010
4
0
0
Thanks for the heads up with the in-GUI status. I was actually going to keep it on just for the heck of it, but I disabled it now.

AFAIK the only filter that's not multithreaded is deinterlace, which I generally have turned off anyways.

I'm still getting the feeling the only reason you're preferring 2-pass is because you can set the final file size. What I'm saying is that if you don't care at all about that, CQ is going to give you better quality at larger file sizes (if needed), because it will use whatever bitrate is required for each part of the video to achieve the quality you've requested, or it will give equal quality at the same bitrate, because it will behave in the same way as 2-pass, only with one pass.

Basically, if you want a file to be 500MB, and you determine what quality factor (say 25) will result in a file that's 500MB (by devoting more or less bits to each frame/scene as it goes), it will be performing the same task as a 2-pass, only you manually did the first pass to determine what quality factor was required for your target size. So by using CQ, you're just saving some time skipping the analysis pass, albeit sometimes using more (but sometimes less) space for the output file.

I agree, it has the disadvantage of an unpredictable size, but that is its only disadvantage, whereas the advantages are that it's faster and for a less demanding encode it might only be, say, 400MB, whereas if you'd done 2-pass, you would have told it to be 500MB, using a higher bitrate than you needed. Granted, you could just assume it's a less demanding video and manually adjust, but that's just more complexity, and it's hard to know and therefore come up with an accurate decision. I'd rather let the codec/encoder figure that out, as that's what they're designed to do, instead of try to do it myself, when I'm clearly nowhere near them in the capability to do so. And if it ends up creating a larger file size than I anticipated, it's probably because I anticipated the complexity of the video wrong, and 264 did its job and still gave me a resultant video of the quality I desired, unlike it would have if I just said to make it a certain size, forcing it to compromise.

So is wanting to know exactly what the end size is going to be your sole reason for preferring 2-pass? If not, please elaborate on why you do, because if I'm missing some detail somewhere, I very much want to know what it is.

BTW, I tried downloading and installing the most recent x264 build, but I'm not even sure if it did anything, and after further research, it appears HB has the codec compiled into the program itself, so even updating it on the system won't affect which version it uses. I might still post on HB's and x264's forums, though, and see if anybody has an answer to the mysterious reduced efficiency at lower resolutions.

As for running two simultaneous encodes, what exactly happens? Does HB just die, and if so is it just the cmd window or the entire program? One thing I read on another forum earlier that you might want to try is having two copies of HB under different names (i.e. HB & HB1).
 

0roo0roo

No Lifer
Sep 21, 2002
64,795
84
91
no, i'm not saying that about the size, i don't care about it either, i'm just saying at a theoretical same size the 2 pass knows more about the entire file so it could theoretically adjust the allocation in a more intelligent way.

two encodes..guis work fine, second spawned command line box dies instantly. two different directory installs fails to fix issue. but i haven't tested copying the install into a 3rd manually without reinstall.
 

vertigo_2_20

Junior Member
Apr 20, 2010
4
0
0
Well, from what I can tell, the only reason it would need to know more about the entire file is to control the resulting size of it. Otherwise, adjusting on the fly based on complexity is no different than 2-pass. With 2-pass, the first pass determines the complexity, then basically does a bunch of calculating to determine where to expend the bits based on that complexity in order to achieve that file size. If the size doesn't matter, there's no reason to not simply determine the complexity and expend the bits as it goes, hence CQ. To me, it's more than that, it's also the fact it will figure out how complex the video is and adjust the file size up or down as needed based on that, whereas I cannot do nearly as good a job as it. As I said, I could be missing something, but it seems to me CQ is equal if not better while being faster and simpler to use. I wonder if anyone has done any extensive subjective testing comparing the two methods.

Try renaming the HB exe's. Maybe being in separate directories isn't enough. The cmd windows they launch might reference back to the original exe, and if there's two with the same name, it might cause an issue. Other than that the only thing I can think of is if one's running, using ~100% of the processor, and you try launching another, maybe instead of playing nice and sharing the cpu half and half they're colliding and causing instability between themselves.

On a side note, I recently was reading about bigadv, which as far as I can tell is basically a very intense folding project, so much so it has to be run in linux (natively or virtually in Windows), since Windows isn't efficient enough for it. Makes me wonder if running HB in linux in VirtualBox would speed up encodes.
 

0roo0roo

No Lifer
Sep 21, 2002
64,795
84
91
well i don't think its that clear, the way it chooses "quality" is limited in range, and is why you still have a quality slider. i think a vbr 2 pass would have more lee way to allocate bits for highs and lows once knowing what it has to work with after the first pass. what you say might be true for a very high setting of cqr, but i don't see how it could be equivalent at lower levels where there has to be some constraint.

and yea i'll try renaming the exe later. damned annoying.
 
Last edited:

0roo0roo

No Lifer
Sep 21, 2002
64,795
84
91
ok i tried using the handbrake copied manually to another directory,still doesn't work.

what does work however is to add jobs to the queue on the second handbrake instance, then under queue theres a generate batch script option. i ran that and it works. dunno if running batches from the same handbrake directory works though. but it does show that with some filters like deinterlace there needs to be a way of spawning multiple jobs at a time because the second running batch is going at the same encode rate as a single job alone, both using all cores, meaning that filter held things way way back.

ok, double bat file from same install directory works.
no log files though.
not sure about shutdown/standby after encode either.
 
Last edited: