rand_harvestq high cpu usage when /dev/urandom is used

Discussion:

Ali Abdallah

2018-08-20 11:27:33 UTC

Hello,

I was just sorting randomly some jpg image files using:

ls *.jpg | sort -R --random-source=/dev/urandom

The above command never exited. Later I noticed that
one of my CPU is always running 100%. top -S reveals that it is
rand_harvestq kernel service.

Is this is a bug? This occurs on 12-ALPHA1 and 11.2

Also, I read on
https://lists.freebsd.org/pipermail/freebsd-current/2013-November/046683.html
someone saying that sysctls to turn off harvesting is documented in
random(6) <https://www.freebsd.org/cgi/man.cgi?query=random&sektion=6&manpath=freebsd-release-ports>.
I had a look
at that document, but wasn't clear to me how to turn it off. I tried
to play with the mask,
but nothing.

Cheers,

Ali

Mark Millard via freebsd-hackers

2018-08-20 11:54:05 UTC

Permalink

Post by Ali Abdallah
. . .
Also, I read on
https://lists.freebsd.org/pipermail/freebsd-current/2013-November/046683.html
someone saying that sysctls to turn off harvesting is documented in
random(6)

It says random(4), so:

<https://www.freebsd.org/cgi/man.cgi?query=random&sektion=4&manpath=freebsd-release-ports>.

Post by Ali Abdallah
. . .

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

Danilo Egêa Gondolfo

2018-08-20 15:23:46 UTC

Permalink

On Mon, Aug 20, 2018 at 8:54 AM Mark Millard via freebsd-hackers <

Post by Ali Abdallah

Post by Ali Abdallah
. . .
Also, I read on

https://lists.freebsd.org/pipermail/freebsd-current/2013-November/046683.html

Post by Ali Abdallah
someone saying that sysctls to turn off harvesting is documented in
random(6)

<
https://www.freebsd.org/cgi/man.cgi?query=random&sektion=4&manpath=freebsd-release-ports

Post by Ali Abdallah
.
. . .

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers

I can confirm this is happening.

It's easily reproducible: dd if=/dev/urandom of=/dev/null
rand_harvestq eat 100% of one CPU for minutes, in my case the system became
unresponsive (it freezes for few seconds).

I'm running FreeBSD 12.0-ALPHA2 #14 r337973M

--
Danilo.
@daniloegea
http://people.freebsd.org/~danilo/

Danilo Egêa Gondolfo

2018-08-20 15:29:24 UTC

Permalink

On Mon, Aug 20, 2018 at 8:54 AM Mark Millard via freebsd-hackers <

Post by Ali Abdallah

Post by Ali Abdallah
. . .
Also, I read on

https://lists.freebsd.org/pipermail/freebsd-current/2013-November/046683.html

Post by Ali Abdallah
someone saying that sysctls to turn off harvesting is documented in
random(6)

<
https://www.freebsd.org/cgi/man.cgi?query=random&sektion=4&manpath=freebsd-release-ports

Post by Ali Abdallah
.
. . .

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers

I can confirm this is happening.

It's easily reproducible: dd if=/dev/urandom of=/dev/null
rand_harvestq eat 100% of one CPU for minutes, in my case the system became
unresponsive (at least the graphical interface, it freezes for few seconds).

I'm running FreeBSD 12.0-ALPHA2 #14 r337973M

(sorry if I sent it twice)

RW via freebsd-hackers

2018-08-20 17:43:37 UTC

Permalink

On Mon, 20 Aug 2018 13:27:33 +0200

Post by Ali Abdallah
Hello,
ls *.jpg | sort -R --random-source=/dev/urandom

urandom is a sim-link to random, so --random-source=/dev/urandom
does nothing useful

Post by Ali Abdallah
The above command never exited. Later I noticed that
one of my CPU is always running 100%. top -S reveals that it is
rand_harvestq kernel service.
Is this is a bug? This occurs on 12-ALPHA1 and 11.2

It's a bit excessive

Post by Ali Abdallah
Also, I read on
https://lists.freebsd.org/pipermail/freebsd-current/2013-November/046683.html
someone saying that sysctls to turn off harvesting is documented in
random(6)
<https://www.freebsd.org/cgi/man.cgi?query=random&sektion=6&manpath=freebsd-release-ports>.
I had a look at that document, but wasn't clear to me how to turn it
off. I tried to play with the mask,
but nothing.

what happens if you set the mask to zero?

RW via freebsd-hackers

2018-08-20 18:35:09 UTC

Permalink

On Mon, 20 Aug 2018 18:43:37 +0100

Post by RW via freebsd-hackers

what happens if you set the mask to zero?

I just tried that and it didn't make any difference, which is strange.

RW via freebsd-hackers

2018-08-22 00:19:01 UTC

Permalink

On Mon, 20 Aug 2018 18:43:37 +0100

Post by RW via freebsd-hackers
On Mon, 20 Aug 2018 13:27:33 +0200

Post by Ali Abdallah
Hello,
ls *.jpg | sort -R --random-source=/dev/urandom

urandom is a sim-link to random, so --random-source=/dev/urandom
does nothing useful

It's a bit excessive

I think I see what is going on. If you have a hardware entropy source
then when you read N bytes out of /dev/random, random_sources_feed()
tries to put at least that amount into each of the entropy pools (32
for fortuna). So if you are reading at 100MB/s, you are trying to feed
3.2GB/s into the pools. Overwriting a slow drive from /dev/random seems
to be enough to waste a CPU core my PC.

Fortuna is only allowed to resend after 100ms, and anything more than
1kB/reseed (pools*keysize) is a waste of CPU cycles. IMO
random_sources_feed() should limit itself to RANDOM_KEYSIZE bytes per
call for each pool/source combination - even that's overkill.

Conrad Meyer

2018-08-22 00:56:40 UTC

Permalink

On Tue, Aug 21, 2018 at 5:19 PM, RW via freebsd-hackers

Post by RW via freebsd-hackers
I think I see what is going on. If you have a hardware entropy source
then when you read N bytes out of /dev/random, random_sources_feed()
tries to put at least that amount into each of the entropy pools (32
for fortuna). So if you are reading at 100MB/s, you are trying to feed
3.2GB/s into the pools. Overwriting a slow drive from /dev/random seems
to be enough to waste a CPU core my PC.

Yep, I came to a similar conclusion[1]. I think you're off by a
factor of two, though — it's even worse than that! It tries to reseed
64x as many bytes from the configured random sources as data read out
of the random device.

Post by RW via freebsd-hackers
Fortuna is only allowed to resend after 100ms, and anything more than
1kB/reseed (pools*keysize) is a waste of CPU cycles. IMO
random_sources_feed() should limit itself to RANDOM_KEYSIZE bytes per
call for each pool/source combination - even that's overkill.

I am less familiar on what Fortuna permits, but yeah, clearly what we
have now is excessive.

Best,
Conrad

[1]: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=230808#c1

Conrad Meyer

2018-08-20 18:57:29 UTC

Permalink

Hi Ali,

Post by Ali Abdallah
Hello,
ls *.jpg | sort -R --random-source=/dev/urandom
The above command never exited. Later I noticed that
one of my CPU is always running 100%. top -S reveals that it is
rand_harvestq kernel service.
Is this is a bug? This occurs on 12-ALPHA1 and 11.2

There is probably at least one bug in sort(1). sort has special
behavior if the --random-source matches its default (/dev/random), but
otherwise doesn't understand device files or pipes very well. Since
urandom isn't exactly the same path as /dev/random, sort fails pretty
hard.

sort attempts to seed its internal RNG by MD5ing the provided random
source path. For its default path, /dev/random, it grabs at most
MAX_DEFAULT_RANDOM_SEED_DATA_SIZE (or 1024) bytes. This is hugely
excessive and MD5ing it is totally unnecessary, but still mostly
harmless.

For non-default files, it just passes the path to MD5File, which will
read() until EOF. Since /dev/urandom will never return EOF, sort
--random-source=/dev/urandom will get stuck in MD5File forever. This
is totally stupid.

I'm not sure why rand_harvestq would take 100% of a CPU core even as a
result of excessive consumption of /dev/urandom, but it is certainly
possible that sort(1) is consuming 100% of a CPU core reading from
urandom and MD5ing the result.

All the best,
Conrad