TRIM utility

[Eugene Grosbein]

Hi!

I found we have no utility capable to perform TRIM for the whole SSD device
or arbitrary part of it, so I wrote simple one. I can't think of nice name
for it, so proposal are welcome! Draft name is "erase".

I ask for pre-commit code review, too.
The code is tested with TRIM-capable SSD and non-capable other devices.

Currently it has four options, all of them are, hmm, optional:

-b: to specify offset from the beginning of the device for trimmed region instead of default 0;
-l: to specify offset from the "-b" margin - length - for trimmed region instead of whole device;
-r rfile: for alternative way to specify length as length of referenced file;
-v: for verbose mode that shows final values for the beginning offset and length.

Later options override previous ones. Then it expects a list of device names as arguments:

erase ada0
erase /dev/ada0s1
erase -b 4096 -r file.img -v /dev/da0 /dev/da1 /dev/da2

The code:

http://www.grosbein.net/freebsd/erase.c
http://www.grosbein.net/freebsd/Makefile.erase

My "mandoc" skills are very poor and English is not my native language,
so any help with manual page creation will be appreciated.

Eugene Grosbein

P.S. I realized that our kernel-level TRIM support has no connection to cam(4) nor to geom(4),
so distinct utility instead of addition to camcontrol(8) or geom(8).

[Warner Losh]

Post by Eugene Grosbein
Hi!
I found we have no utility capable to perform TRIM for the whole SSD device
or arbitrary part of it, so I wrote simple one. I can't think of nice name
for it, so proposal are welcome! Draft name is "erase".
I ask for pre-commit code review, too.
The code is tested with TRIM-capable SSD and non-capable other devices.
-b: to specify offset from the beginning of the device for trimmed region
instead of default 0;
-l: to specify offset from the "-b" margin - length - for trimmed region
instead of whole device;
-r rfile: for alternative way to specify length as length of referenced file;

This seems really obscure and would be better handled by a stat command.

-v: for verbose mode that shows final values for the beginning offset and

Post by Eugene Grosbein
length.
erase ada0
erase /dev/ada0s1
erase -b 4096 -r file.img -v /dev/da0 /dev/da1 /dev/da2
http://www.grosbein.net/freebsd/erase.c
http://www.grosbein.net/freebsd/Makefile.erase
My "mandoc" skills are very poor and English is not my native language,
so any help with manual page creation will be appreciated.
Eugene Grosbein
P.S. I realized that our kernel-level TRIM support has no connection to
cam(4) nor to geom(4),
so distinct utility instead of addition to camcontrol(8) or geom(8).

"erase" is a really bad name. It's fraught with too many overloaded
meanings. "trim" is likely the least bad name we can use. "delete" is
similarly bad, even though that name is in the ioctl and in BIO_DELETE.
"unmap" is the SCSI term, so it wouldn't be terrible, though the term is
less widely used than "trim" is for this stuff. None of the other command
names for more obscure technologies are general enough or widely enough
known. "discard" is the Linux mount option, which seems less descriptive.

Linux has a fstrim command, which does something kinda similar (it's a lot
like fsck -E to erase unused parts of the filesystem), so there is some
overlap. I couldn't find a dedicated command to do that, but if it does, we
should follow that convention to reimplement.

It feels to me to be a dd conv= option, but that isn't reflected in any
implementation I could find, so that suggests I'm nuts (though sparse is a
weak match).

Warner

[Eugene Grosbein]

Post by Warner Losh

Post by Eugene Grosbein
I found we have no utility capable to perform TRIM for the whole SSD device
or arbitrary part of it, so I wrote simple one. I can't think of nice name
for it, so proposal are welcome! Draft name is "erase".
I ask for pre-commit code review, too.
The code is tested with TRIM-capable SSD and non-capable other devices.
-b: to specify offset from the beginning of the device for trimmed region
instead of default 0;
-l: to specify offset from the "-b" margin - length - for trimmed region
instead of whole device;
-r rfile: for alternative way to specify length as length of referenced file;

This seems really obscure and would be better handled by a stat command.

This is inspired by truncate(1) having same option that saves extra call to stat.

Forgot to note, that options -b and -l allow suffixes [K|k|M|m|G|g|T|t]
just like truncate's option -s does.

Post by Warner Losh
"erase" is a really bad name. It's fraught with too many overloaded
meanings. "trim" is likely the least bad name we can use.

I'm fine with this, renamed.

Post by Warner Losh
Linux has a fstrim command, which does something kinda similar (it's a lot
like fsck -E to erase unused parts of the filesystem), so there is some
overlap. I couldn't find a dedicated command to do that, but if it does, we
should follow that convention to reimplement.

Well, they have http://man7.org/linux/man-pages/man8/blkdiscard.8.html

I don't like the name, though. It's too complicated to pronounce and too long same time.

I can rename -b option to -o to match blkdiscard's, if this matters.

[Eugene Grosbein]

Would "hdtrim" or "ssdtrim" be better names?

Just "trim" sounds better to me :-)

[Warner Losh]

Post by Eugene Grosbein

Would "hdtrim" or "ssdtrim" be better names?

Just "trim" sounds better to me :-)

Trim applies to more than add. And never to hdd. Virtualization uses it. SD
cards use it. NVMe drives use it (I'd argue the aren't really SSDs but
something else).

Warner

[Wojciech Puchar]

thanks. very useful tool

Post by Eugene Grosbein
Hi!
I found we have no utility capable to perform TRIM for the whole SSD device
or arbitrary part of it, so I wrote simple one. I can't think of nice name
for it, so proposal are welcome! Draft name is "erase".
I ask for pre-commit code review, too.
The code is tested with TRIM-capable SSD and non-capable other devices.
-b: to specify offset from the beginning of the device for trimmed region instead of default 0;
-l: to specify offset from the "-b" margin - length - for trimmed region instead of whole device;
-r rfile: for alternative way to specify length as length of referenced file;
-v: for verbose mode that shows final values for the beginning offset and length.
erase ada0
erase /dev/ada0s1
erase -b 4096 -r file.img -v /dev/da0 /dev/da1 /dev/da2
http://www.grosbein.net/freebsd/erase.c
http://www.grosbein.net/freebsd/Makefile.erase
My "mandoc" skills are very poor and English is not my native language,
so any help with manual page creation will be appreciated.
Eugene Grosbein
P.S. I realized that our kernel-level TRIM support has no connection to cam(4) nor to geom(4),
so distinct utility instead of addition to camcontrol(8) or geom(8).

[Lev Serebryakov]

Post by Eugene Grosbein
-b: to specify offset from the beginning of the device for trimmed region instead of default 0;
-l: to specify offset from the "-b" margin - length - for trimmed region instead of whole device;
-r rfile: for alternative way to specify length as length of referenced file;
-v: for verbose mode that shows final values for the beginning offset and length.

Please, add '-n' for dry-run, which could be useful together with '-v'.

[Poul-Henning Kamp]

--------
Isn't this the kind of thing that dd(1) should learn about instead ?

[Lev Serebryakov]

Post by Poul-Henning Kamp
Isn't this the kind of thing that dd(1) should learn about instead ?

One utility to done one thing very well? :-)

dd(1) is way overloaded, IMHO.

[Enji Cooper]

Post by Lev Serebryakov

Post by Poul-Henning Kamp
Isn't this the kind of thing that dd(1) should learn about instead ?

One utility to done one thing very well? :-)
dd(1) is way overloaded, IMHO.

I agree that dd is super complex; while we could add conv=trim, that might not be the right way.

Would it make sense to have pushed down into a simple library (leveraging the logic used by dd), and adopt in several tools?

Thanks so much for bringing this up Eugene :).

-Enji

[Gary Jennejohn]

On Fri, 23 Nov 2018 14:12:47 +0300

Post by Lev Serebryakov

Post by Eugene Grosbein
-b: to specify offset from the beginning of the device for trimmed region instead of default 0;
-l: to specify offset from the "-b" margin - length - for trimmed region instead of whole device;
-r rfile: for alternative way to specify length as length of referenced file;
-v: for verbose mode that shows final values for the beginning offset and length.

Please, add '-n' for dry-run, which could be useful together with '-v'.

+1

[Rodney W. Grimes]

Post by Lev Serebryakov

Post by Poul-Henning Kamp
Isn't this the kind of thing that dd(1) should learn about instead ?

One utility to done one thing very well? :-)
dd(1) is way overloaded, IMHO.

I agree here, we do too much of trying to shoe horn things
into existing utilities then we end up with a command parser
that only a mother could love.

trim, hdtrim, blktrim, camtrim, any of them
are fine, fstrim is bad, this is not a filesystem op,
too bad the next thing that comes
along that is "trim" like well have to pick
something other than trim.

I might ask would it be horribly hard to access the
"secure erase" feature from this utility? Or do we
have another that can easily get at that function,
that is usually the prefered vendor specific method
to "trim" the complete drive, often restoring badly
leveled SSD's to a performant and usable state.

[Warner Losh]

On Fri, Nov 23, 2018, 9:04 AM Rodney W. Grimes <

Post by Rodney W. Grimes

Post by Lev Serebryakov

Post by Poul-Henning Kamp
Isn't this the kind of thing that dd(1) should learn about instead ?

One utility to done one thing very well? :-)
dd(1) is way overloaded, IMHO.

I agree here, we do too much of trying to shoe horn things
into existing utilities then we end up with a command parser
that only a mother could love.
trim, hdtrim, blktrim, camtrim, any of them
are fine, fstrim is bad, this is not a filesystem op,
too bad the next thing that comes
along that is "trim" like well have to pick
something other than trim.

Actually, you can now do the disk delete ioctl on a file range, and the
putative trim program does that... but we've settled on trim I think.

I might ask would it be horribly hard to access the

Post by Rodney W. Grimes
"secure erase" feature from this utility?

Yes. It would. That's hard with the current storage stack to do via the
disk interface. And often the underlying protocols do not support partial
ranges. There is no good way to do this with buf/bio interface we have. So
it is a really bad match all the way around.

Or do we

Post by Rodney W. Grimes
have another that can easily get at that function,
that is usually the prefered vendor specific method
to "trim" the complete drive, often restoring badly
leveled SSD's to a performant and usable state.

Camcontrol already supports secure erase for both SCSI and ATA drives. And
sanitize for SCSI (an alternative way to do the same thing to reset the
ssd's FLT). It bypasses the disk interface and sends raw protocol commands
via the pass interface. I do this all the time to rehab drives, do
diagnosis of vendor issues or scrub ssds I'm sending to third parties.

Warner

[Wojciech Puchar]

Post by Warner Losh
Yes. It would. That's hard with the current storage stack to do via the
disk interface. And often the underlying protocols do not support partial
ranges. There is no good way to do this with buf/bio interface we have. So

what is an actual difference between "secure erase" and trimming whole
disk?

Post by Warner Losh
it is a really bad match all the way around.
Or do we

Post by Rodney W. Grimes
have another that can easily get at that function,
that is usually the prefered vendor specific method
to "trim" the complete drive, often restoring badly
leveled SSD's to a performant and usable state.

Camcontrol already supports secure erase for both SCSI and ATA drives. And
sanitize for SCSI (an alternative way to do the same thing to reset the
ssd's FLT). It bypasses the disk interface and sends raw protocol commands
via the pass interface. I do this all the time to rehab drives, do
diagnosis of vendor issues or scrub ssds I'm sending to third parties.
Warner
--

Post by Rodney W. Grimes
Rod Grimes
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers

_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers

[Dimitry Andric]

Post by Warner Losh
Yes. It would. That's hard with the current storage stack to do via the
disk interface. And often the underlying protocols do not support partial
ranges. There is no good way to do this with buf/bio interface we have. So

what is an actual difference between "secure erase" and trimming whole disk?

It depends a lot on the device. Some devices encrypt all blocks
automatically, and a Secure Erase command might just throw away the key,
not go over all the data and actively wipe it.

Most SSDs will likely also trim all the blocks to be erased, since users
have come to expect the SSD performance to go back to the 'out of box'
level, after such an operation.

If you are lucky, the manufacturer's documentation will explain the
specifics, but don't count on it. :)

-Dimitry

[Warner Losh]

Post by Warner Losh

Post by Wojciech Puchar

Post by Warner Losh
Yes. It would. That's hard with the current storage stack to do via the
disk interface. And often the underlying protocols do not support

partial

Post by Wojciech Puchar

Post by Warner Losh
ranges. There is no good way to do this with buf/bio interface we have.

So

Post by Wojciech Puchar
what is an actual difference between "secure erase" and trimming whole

disk?
It depends a lot on the device. Some devices encrypt all blocks
automatically, and a Secure Erase command might just throw away the key,
not go over all the data and actively wipe it.

When there's a difference, sanitize is usually the variant that does this.
In all the SSDs I've had problems with, however, both Secure Erase and
Sanitize reset the NAND pool (which effectively causes all the NAND erase
blocks to be erased).

TRIM is a 'logical' command, not a physical one. It doesn't tell the drive
to erase anything. It just marks the data as being free. Most drives will
then garbage collect things because certain low-water threshold marks are
hit causing the blocks to be erased, but TRIM is definitely not a command
to erase.

Also, Secure Erase and Sanitize both are more destructive an thorough than
a simple TRIM in that they potentially remove extra metadata that you'd
need to keep intact if you wanted to keep using the drive (such as the
current encryption keys).

Post by Warner Losh
Most SSDs will likely also trim all the blocks to be erased, since userave
come to expect the SSD performance to go back to the 'out of box'
level, after such an operation.

Correct. I've not found any that don't, even though they are allowed to. It
also maximizes future performance because there's no need to stop and wait
for erase blocks to go through and an erase cycle before programming them.
And there's some advantage, in some of the NAND technology nodes, to
keeping things in the erased state a long as possible.

Post by Warner Losh
If you are lucky, the manufacturer's documentation will explain the
specifics, but don't count on it. :)

Very lucky :)

Warner

[Rodney W. Grimes]

Post by Dimitry Andric

Post by Warner Losh
Yes. It would. That's hard with the current storage stack to do via the
disk interface. And often the underlying protocols do not support partial
ranges. There is no good way to do this with buf/bio interface we have. So

what is an actual difference between "secure erase" and trimming whole disk?

I wonder if any drive recognises a single trim command
that covers the whole drive, or if that is even possible
to do. Or if it recognizes that all blocks are infact
in a free/trimmed state (should be easy if they have
a total block inuse counter, just watch for it to hit 0).

Post by Dimitry Andric
It depends a lot on the device. Some devices encrypt all blocks
automatically, and a Secure Erase command might just throw away the key,
not go over all the data and actively wipe it.
Most SSDs will likely also trim all the blocks to be erased, since users
have come to expect the SSD performance to go back to the 'out of box'
level, after such an operation.

I do not think they actually "trim" anything, I am pretty
sure they just do a full FDT re-initializse and keep the
block use counters for future allocations, in effect this
looks like you trimmed the whole drive but usually completed
in just a fex seconds or less.

Some manufactures actually recommend this procedure to
revive a drives performance that has degraded over time.

Post by Dimitry Andric
If you are lucky, the manufacturer's documentation will explain the
specifics, but don't count on it. :)

And there is that problem :-(

[Warner Losh]

On Sat, Nov 24, 2018 at 10:05 AM Rodney W. Grimes <

Post by Warner Losh

Post by Dimitry Andric

Post by Wojciech Puchar

Post by Warner Losh
Yes. It would. That's hard with the current storage stack to do via

the

Post by Dimitry Andric

Post by Wojciech Puchar

Post by Warner Losh
disk interface. And often the underlying protocols do not support

partial

Post by Dimitry Andric

Post by Wojciech Puchar

Post by Warner Losh
ranges. There is no good way to do this with buf/bio interface we

have. So

Post by Dimitry Andric

Post by Wojciech Puchar
what is an actual difference between "secure erase" and trimming whole

disk?
I wonder if any drive recognises a single trim command
that covers the whole drive, or if that is even possible
to do. Or if it recognizes that all blocks are infact
in a free/trimmed state (should be easy if they have
a total block inuse counter, just watch for it to hit 0).

TRIM is a logical operation. And it's limited to ~32GB in ATA.

Lots of TRIMs will trigger garbage collection, which is what causes the
erase blocks to be physically erased so they can be used for future writes.

Post by Warner Losh

Post by Dimitry Andric
It depends a lot on the device. Some devices encrypt all blocks
automatically, and a Secure Erase command might just throw away the key,
not go over all the data and actively wipe it.
Most SSDs will likely also trim all the blocks to be erased, since users
have come to expect the SSD performance to go back to the 'out of box'
level, after such an operation.

I do not think they actually "trim" anything, I am pretty
sure they just do a full FDT re-initializse and keep the
block use counters for future allocations, in effect this
looks like you trimmed the whole drive but usually completed
in just a fex seconds or less.

TRIM and Secure Erase are different beasts. TRIM is a logical operation
that unmaps a LBA to physical mapping and does nothing more (though that
unmapping may cause other things to happen indirectly).

TRIMs also have to conform to certain behavior characteristics spelled out
in the standards, so at the very least trimming the whole drive requires
that, for drives that report back 0's or 1's for trimmed sectors, it write
out records that say the blocks were unmapped in case there's a power
outage before it can get to erasing things.

Such SSDs are somewhat rare these days. Most take more than a few seconds
to trim the entire drive. At least so my experience has shown.

Post by Warner Losh
Some manufactures actually recommend this procedure to
revive a drives performance that has degraded over time.

p
Most recommend secure erase to restore performance because it knows it can
do certain operations cheaply than lots of incremental trims might make
harder to know.

Post by Warner Losh

Post by Dimitry Andric
If you are lucky, the manufacturer's documentation will explain the
specifics, but don't count on it. :)

And there is that problem :-(

Not really. It depends on what you are using the tool for. TRIM is just an
operation that un-does mappings. Secure Erase and Sanitize make the drive's
contents unavailable to the host (though maybe not determined adversaries
with access to the NAND chips). The reason the specs are nailed down 100%
is because there's a number of NAND cell / block failures that make
guarantees impossible. And the vendor may not implement any feature to
'restore factory new speed' in its drives, either because it believes its
FTL never slows down (super high end) or because they don't care (super low
end).

Warner

Warner