Limits to seeding /dev/random | random(4)

[Dirk-Willem van Gulik]

When feeding /dev/random from hardware USB devices like Bill Woodcock’s design in PCB incarnation:

https://13-37.org/de/shop/infinite-noise-trng/

Are there any caveats with regard to volume or speed of doing so ? Or is it always a plus ?

Actual code at https://github.com/dirkx/infnoise/blob/master/software/libinfnoise.c line 122:

if ((devRandomFD = open("/dev/random",O_WRONLY)) <0)
.. error handling

if (write(devRandomFD, bytes, length) != length)
.. error handling

And is there any case where length would not return the length written — it seems that the driver traps/ignores EINT, EAGAIN and short writes ?

Or should one check the entropy available in /dev/random (how?) and hold off feeding it until it is low enough (this is what the infinite-trng seems to do on linux).

With kind regards,

Dw

[Ian Lepore]

Post by Dirk-Willem van Gulik
When feeding /dev/random from hardware USB devices like Bill
https://13-37.org/de/shop/infinite-noise-trng/
Are there any caveats with regard to volume or speed of doing so ? Or
is it always a plus ? 
Actual code at https://github.com/dirkx/infnoise/blob/master/software
if ((devRandomFD = open("/dev/random",O_WRONLY)) <0)
.. error handling
if (write(devRandomFD, bytes, length) != length) 
.. error handling
And is there any case where length would not return the length
written — it seems that the driver traps/ignores EINT, EAGAIN and
short writes ? 
Or should one check the entropy available in /dev/random (how?) and
hold off feeding it until it is low enough (this is what the
infinite-trng seems to do on linux).
With kind regards,
Dw

There is no way to check the entropy available in /dev/random because
the whole concept doesn't apply. Entropy isn't a limited resource that
can be exhausted after the prng is seeded at boot time.

When asking our prng gurus for advice on writing a device driver for an
on-chip entropy source, the advice I got was basically: there's no need
to feed in more entropy on an ongoing basis, but no harm in doing so
either, within reason. The recommendation was to feed at or below an
average rate of about 128 bits/second. Pushing in more isn't harmful,
just wasteful of system resources because it doesn't make anything
better.

-- Ian

[Dirk-Willem van Gulik]

Post by Ian Lepore

Post by Dirk-Willem van Gulik
When feeding /dev/random from hardware USB devices like Bill
https://13-37.org/de/shop/infinite-noise-trng/
Are there any caveats with regard to volume or speed of doing so ? Or
is it always a plus ?
Actual code at https://github.com/dirkx/infnoise/blob/master/software
if ((devRandomFD = open("/dev/random",O_WRONLY)) <0)
.. error handling
if (write(devRandomFD, bytes, length) != length)
.. error handling
And is there any case where length would not return the length
written — it seems that the driver traps/ignores EINT, EAGAIN and
short writes ?
Or should one check the entropy available in /dev/random (how?) and
hold off feeding it until it is low enough (this is what the
infinite-trng seems to do on linux).
With kind regards,
Dw

There is no way to check the entropy available in /dev/random because
the whole concept doesn't apply. Entropy isn't a limited resource that
can be exhausted after the prng is seeded at boot time.
When asking our prng gurus for advice on writing a device driver for an
on-chip entropy source, the advice I got was basically: there's no need
to feed in more entropy on an ongoing basis, but no harm in doing so
either, within reason.

Understood. But it is not a bad thing if one also takes into account all sort of governance stuff around certification & what not (and we got bitten at least twice by a virtualised environment producing something identical - which with hindsight was obvious but still).

Post by Ian Lepore
The recommendation was to feed at or below an
average rate of about 128 bits/second. Pushing in more isn't harmful,
just wasteful of system resources because it doesn't make anything
better.

Ok - so these units spit out around a 90-100 K bit/second (that is after it has been whitewashed by a Keccack-1600).

So I guess I should not pass all of that on blindly — but quell that stream by a factor thousand.

Dw.

[RW via freebsd-hackers]

On Wed, 11 Jul 2018 07:58:35 -0600

Post by Ian Lepore
When asking our prng gurus for advice on writing a device driver for
an on-chip entropy source, the advice I got was basically: there's no
need to feed in more entropy on an ongoing basis, but no harm in
doing so either, within reason. The recommendation was to feed at or
below an average rate of about 128 bits/second. Pushing in more isn't
harmful, just wasteful of system resources because it doesn't make
anything better.

This is a bit simplistic because it ignores the way that fortuna
stripes entropy across 32 pools.

In order to fully secure the prng at boot time you need to get 256 bits
of entropy into it, and to guarantee that you need to have 256 bits in
pool[0], which means you need to write 256*32=8192 bits into the random
device. This should be done as early in the rc.d boot process as
possible. Once the pools are primed you could trickle entropy in in
smaller amounts if you wish.

[Dirk-Willem van Gulik]

Post by RW via freebsd-hackers

Post by Ian Lepore
When asking our prng gurus for advice on writing a device driver for
an on-chip entropy source, the advice I got was basically: there's no
need to feed in more entropy on an ongoing basis, but no harm in
doing so either, within reason. The recommendation was to feed at or
below an average rate of about 128 bits/second. Pushing in more isn't
harmful, just wasteful of system resources because it doesn't make
anything better.

This is a bit simplistic because it ignores the way that fortuna
stripes entropy across 32 pools.
In order to fully secure the prng at boot time you need to get 256 bits
of entropy into it, and to guarantee that you need to have 256 bits in
pool[0], which means you need to write 256*32=8192 bits into the random
device. This should be done as early in the rc.d boot process as
possible. Once the pools are primed you could trickle entropy in in
smaller amounts if you wish.

So these HW devices [1] give us a raw feed — which one usually whitewashes [2] in order to use.

It is fairly well defined how many bits of entropy we get ‘raw’.

During boot - can I feed the right number of bits without whitewashing - letting the kernel do the trick (much like random_harvest_queue() does in for example the mouse driver) ?

Or should it be properly whitened first ?

Our goal is to get to a point where a very stripped down BSD can be booted up (sans network or much in terms of attached devices but for a printer and chipcard reader) — yet is know to have a solid seeded RNG.

With kind regards,

Dw.

1: https://13-37.org/en/shop/infinite-noise-trng/
2: https://github.com/manuel-domke/infnoise

[Conrad Meyer]

On Thu, Jul 12, 2018 at 9:03 AM, Dirk-Willem van Gulik

Post by Dirk-Willem van Gulik

Post by RW via freebsd-hackers

Post by Ian Lepore
When asking our prng gurus for advice on writing a device driver for
an on-chip entropy source, the advice I got was basically: there's no
need to feed in more entropy on an ongoing basis, but no harm in
doing so either, within reason. The recommendation was to feed at or
below an average rate of about 128 bits/second. Pushing in more isn't
harmful, just wasteful of system resources because it doesn't make
anything better.

This is a bit simplistic because it ignores the way that fortuna
stripes entropy across 32 pools.

RW, you and Ian are talking about different things. Ian is talking
about post-seed, additional entropy from a hardware device. You are
talking about initial seeding. You're both right, but talking past
each other :-).

Post by Dirk-Willem van Gulik

Post by RW via freebsd-hackers
In order to fully secure the prng at boot time you need to get 256 bits
of entropy into it, and to guarantee that you need to have 256 bits in
pool[0], which means you need to write 256*32=8192 bits into the random
device. This should be done as early in the rc.d boot process as
possible.

For example, it is done by the loader, as well as the /etc/rc.d/random
script using entropy saved from the RNG at shutdown on any FreeBSD
with a writable /, /var, or /boot.

Post by Dirk-Willem van Gulik

Post by RW via freebsd-hackers
Once the pools are primed you could trickle entropy in in
smaller amounts if you wish.

Right — that's what Ian is suggesting.

Post by Dirk-Willem van Gulik
So these HW devices [1] give us a raw feed — which one usually whitewashes [2] in order to use.

Don't feed the raw data — use the washed bits.

Post by Dirk-Willem van Gulik
It is fairly well defined how many bits of entropy we get ‘raw’.
During boot - can I feed the right number of bits without whitewashing - letting the kernel do the trick (much like random_harvest_queue() does in for example the mouse driver) ?

Why feed less random data to the kernel when you have a relatively
high throughput random device available? Your device generates 90
kbps after washing — it'll take at most 90 ms to fully seed
/dev/random at that rate, even with a readonly filesystem
embedded-type device.

Post by Dirk-Willem van Gulik
Or should it be properly whitened first ?

Yes :-).

Post by Dirk-Willem van Gulik
Our goal is to get to a point where a very stripped down BSD can be booted up (sans network or much in terms of attached devices but for a printer and chipcard reader) — yet is know to have a solid seeded RNG.

/dev/u?random never produces unseeded results. If it is not seeded,
reads will just block indefinitely, until it is seeded.

To seed the device without a writable filesystem, write 1kB+ of
whitened random from your device into /dev/random early in boot, and
you will be good to go. You can do the ongoing trickle after that if
you want, but it is not necessary. On FreeBSD 12-CURRENT, you can
verify /dev/random is seeded when getrandom(..., GRND_NONBLOCK) no
longer returns -1 with EAGAIN errno. If you need to use a FreeBSD
prior to 12, you'll know random is seeded when reads no longer block.

All the best,
Conrad

[Dirk-Willem van Gulik]

Post by Conrad Meyer
On Thu, Jul 12, 2018 at 9:03 AM, Dirk-Willem van Gulik

Post by Dirk-Willem van Gulik

Post by RW via freebsd-hackers

Post by Ian Lepore
When asking our prng gurus for advice on writing a device driver for
an on-chip entropy source, the advice I got was basically: there's no
need to feed in more entropy on an ongoing basis, but no harm in
doing so either, within reason. The recommendation was to feed at or
below an average rate of about 128 bits/second. Pushing in more isn't
harmful, just wasteful of system resources because it doesn't make
anything better.

This is a bit simplistic because it ignores the way that fortuna
stripes entropy across 32 pools.

RW, you and Ian are talking about different things. Ian is talking
about post-seed, additional entropy from a hardware device. You are
talking about initial seeding. You're both right, but talking past
each other :-).

Clear thanks ! And it is indeed that space I care about. As it is there where we see hang/wait for entropy and the very occasional identical result in CI testing.

Post by Conrad Meyer

Post by Dirk-Willem van Gulik

Post by RW via freebsd-hackers
In order to fully secure the prng at boot time you need to get 256 bits
of entropy into it, and to guarantee that you need to have 256 bits in
pool[0], which means you need to write 256*32=8192 bits into the random
device. This should be done as early in the rc.d boot process as
possible.

For example, it is done by the loader, as well as the /etc/rc.d/random
script using entropy saved from the RNG at shutdown on any FreeBSD
with a writable /, /var, or /boot.

Post by Dirk-Willem van Gulik

Post by RW via freebsd-hackers
Once the pools are primed you could trickle entropy in in
smaller amounts if you wish.

Right — that's what Ian is suggesting.

Post by Dirk-Willem van Gulik
So these HW devices [1] give us a raw feed — which one usually whitewashes [2] in order to use.

Don't feed the raw data — use the washed bits.

Post by Dirk-Willem van Gulik
It is fairly well defined how many bits of entropy we get ‘raw’.
During boot - can I feed the right number of bits without whitewashing - letting the kernel do the trick (much like random_harvest_queue() does in for example the mouse driver) ?

Why feed less random data to the kernel when you have a relatively
high throughput random device available? Your device generates 90
kbps after washing — it'll take at most 90 ms to fully seed
/dev/random at that rate, even with a readonly filesystem
embedded-type device.

Post by Dirk-Willem van Gulik
Or should it be properly whitened first ?

Yes :-).

Excellent - I have my marching orders. Much appreciated !

Is there any point - much later post boot, in a non-network, read-only situation with essentially just 3 or 4 user processes running with no IO or interaction - to send some entropy (withewashed (or raw with random_harvest_queue()) to wards the PRNG ?

Or is that pointless from thereon.

Post by Conrad Meyer

Post by Dirk-Willem van Gulik
Our goal is to get to a point where a very stripped down BSD can be booted up (sans network or much in terms of attached devices but for a printer and chipcard reader) — yet is know to have a solid seeded RNG.

/dev/u?random never produces unseeded results. If it is not seeded,
reads will just block indefinitely, until it is seeded.

As we’ve found out the hard way (although we are not sure it is indefinitely).

Post by Conrad Meyer
To seed the device without a writable filesystem, write 1kB+ of
whitened random from your device into /dev/random early in boot, and
you will be good to go. You can do the ongoing trickle after that if
you want, but it is not necessary. On FreeBSD 12-CURRENT, you can
verify /dev/random is seeded when getrandom(..., GRND_NONBLOCK) no
longer returns -1 with EAGAIN errno. If you need to use a FreeBSD
prior to 12, you'll know random is seeded when reads no longer block.

Thanks for that. Unfortunately we’re in a read-only situation. And we’ve had CI testing yield identical results a few times now.

Dw.

[Conrad Meyer]

On Thu, Jul 12, 2018 at 10:42 AM, Dirk-Willem van Gulik

Post by Dirk-Willem van Gulik
Is there any point - much later post boot, in a non-network, read-only situation with essentially just 3 or 4 user processes running with no IO or interaction - to send some entropy (withewashed (or raw with random_harvest_queue()) to wards the PRNG ?
Or is that pointless from thereon.

It isn't needed, but it doesn't hurt either (barring elevated CPU use
from excessive feeding).

Post by Dirk-Willem van Gulik

Post by Conrad Meyer
/dev/u?random never produces unseeded results. If it is not seeded,
reads will just block indefinitely, until it is seeded.

As we’ve found out the hard way (although we are not sure it is indefinitely).

It is indefinite, until seeding. Maybe signals can interrupt the
wait, but you should be checking the return value of read(2) of
/dev/random.

Post by Dirk-Willem van Gulik

Post by Conrad Meyer
To seed the device without a writable filesystem, write 1kB+ of
whitened random from your device into /dev/random early in boot, and
you will be good to go. You can do the ongoing trickle after that if
you want, but it is not necessary. On FreeBSD 12-CURRENT, you can
verify /dev/random is seeded when getrandom(..., GRND_NONBLOCK) no
longer returns -1 with EAGAIN errno. If you need to use a FreeBSD
prior to 12, you'll know random is seeded when reads no longer block.

Thanks for that. Unfortunately we’re in a read-only situation. And we’ve had CI testing yield identical results a few times now.

Identical results are very troubling. Maybe your readonly filesystems
contain a static "entropy" file that is being fed in every boot (with
identical contents)? If so, you definitely want to remove that during
image generation. That, in tandem with few other sources of entropy,
could explain identical results.

Another thing I would suggest is taking samples directly from your
random device and running them through
https://github.com/usnistgov/SP800-90B_EntropyAssessment to sanity
check their randomness. W. Dean Freeman did some great work
evaluating random sources in FreeBSD within the last couple years; you
can check out his work here:
https://github.com/badfilemagic/fbsd-entropy

Best,
Conrad

[Ian Lepore]

Identical results are very troubling.  Maybe your readonly
filesystems
contain a static "entropy" file that is being fed in every boot (with
identical contents)?  If so, you definitely want to remove that
during
image generation.  That, in tandem with few other sources of entropy,
could explain identical results.

I have been reporting for years that certain kinds of embedded systems
lead to zero entropy available at boot, including the fact that the
kernel's attempt to harvest entropy from things such as device attach
timings and so forth are, in some situations, completely ineffective
and yield numbers that are identical from one boot to the next. I even
posted logs of it happening years ago. Still, people just find the
whole idea of this sort of reproducibility so gut-level counter-
intuitive that they dismiss and deny it.

It happens. Embedded systems are a different world, and if entropy is
important, sometimes we have to go out of our way to provide some.

-- Ian

[Dirk-Willem van Gulik]

Post by Ian Lepore

Post by Conrad Meyer
Identical results are very troubling. Maybe your readonly
filesystems
contain a static "entropy" file that is being fed in every boot (with

Most certainly not.

Post by Ian Lepore

Post by Conrad Meyer
identical contents)? If so, you definitely want to remove that during
image generation. That, in tandem with few other sources of entropy,
could explain identical results.

I suspect this to be the issue.

Post by Ian Lepore
I have been reporting for years that certain kinds of embedded systems
lead to zero entropy available at boot, including the fact that the

..

Post by Ian Lepore
It happens. Embedded systems are a different world, and if entropy is
important, sometimes we have to go out of our way to provide some.

In our case it is merely a low end machine - but diskless, read-only and with hardly any perifials.

Dw.