Why is there no bus_dmamap_load_sg()?

[Pratyush Yadav]

Hi all,

I am working on Xen this summer [0], and I have to use the bus_dma(9)
interface extensively during the course of the project.

I was looking at the man page and I notice the function
bus_dmamap_load_mbuf_sg() which directly returns the segment array to
the client. But a similar variant of bus_dmamap_load() is missing.
This variant would be useful to me and help me in my work.

Is there a reason the sg variant was only implemented for mbufs?

I was looking at the code in sys/kern/subr_bus_dma.c and a
bus_dmamap_load_sg() does not look too difficult to implement. Is it
ok if I submit a patch implementing it?

[Lakhan Shiva]

Hi,

I think nobody wanted this before. As you have the need for it in your
project you should go ahead and create the service.
Again, I am not 100% sure, correct me if i am wrong.

Thanks,
Lakhan

Post by Pratyush Yadav
Hi all,
I am working on Xen this summer [0], and I have to use the bus_dma(9)
interface extensively during the course of the project.
I was looking at the man page and I notice the function
bus_dmamap_load_mbuf_sg() which directly returns the segment array to
the client. But a similar variant of bus_dmamap_load() is missing.
This variant would be useful to me and help me in my work.
Is there a reason the sg variant was only implemented for mbufs?
I was looking at the code in sys/kern/subr_bus_dma.c and a
bus_dmamap_load_sg() does not look too difficult to implement. Is it
ok if I submit a patch implementing it?
--
Regards,
Pratyush Yadav
[0]
https://wiki.freebsd.org/SummerOfCode2018Projects/ImportXenbus_dmaFromOpenBSD
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers

[Warner Losh]

Post by Pratyush Yadav
I was looking at the man page and I notice the function
bus_dmamap_load_mbuf_sg() which directly returns the segment array to
the client. But a similar variant of bus_dmamap_load() is missing.
This variant would be useful to me and help me in my work.
Is there a reason the sg variant was only implemented for mbufs?
I was looking at the code in sys/kern/subr_bus_dma.c and a
bus_dmamap_load_sg() does not look too difficult to implement. Is it
ok if I submit a patch implementing it?

The biggest reason is because of the failure modes. If there's not
sufficient memory or other resources to honor the request, we defer until
there is such memory / resources and then call the callback. There's no
convenient hook the drivers can connect to so they can retry the allocation
when memory becomes available (busdma has one since eventually a request
will complete and unload). The original port was for CAM SIMs which had
stringent requirements to always make progress on I/Os to avoid deadlock.
Rather than put this tricky code in all the CAM SIMs, it's centralized to
retry as soon and as robustly as it can.

Network drivers, however, don't have such requirements and often don't
care. If they can't load a map, they can put the packet back on the
transmit queue (or whatever), set a flag and return without putting it into
the NIC's buffer rings. Eventually, a packet will finish transmitting and
there will be a chance to send it again (or one of a number of other
events, I'm glossing heavily here). It's a better fit for network drivers
than storage drivers. Since bus_dmamap_load_mbuf* is used only for network
drivers, it makes sense.

So, it would only make sense to add if you can cope with BUS_DMA_NOWAIT
always, and that error handling is simpler than letting busdma call you
back when the shortage is over. Is that the case for your port? Or is it
just because OpenBSD doesn't have this sort of interface (the callback
difference dates back to the initial port to FreeBSD by Justin Gibbs
because it never got folded back into NetBSD for reasons too long, too old
and only half remembered by me for this post)? Can the code you are porting
cope with error returns and retry again later?

Warner

[Pratyush Yadav]

Hi Warner,

Post by Warner Losh

Post by Pratyush Yadav
I was looking at the man page and I notice the function
bus_dmamap_load_mbuf_sg() which directly returns the segment array to
the client. But a similar variant of bus_dmamap_load() is missing.
This variant would be useful to me and help me in my work.
Is there a reason the sg variant was only implemented for mbufs?
I was looking at the code in sys/kern/subr_bus_dma.c and a
bus_dmamap_load_sg() does not look too difficult to implement. Is it
ok if I submit a patch implementing it?

The biggest reason is because of the failure modes. If there's not
sufficient memory or other resources to honor the request, we defer until
there is such memory / resources and then call the callback. There's no
convenient hook the drivers can connect to so they can retry the allocation
when memory becomes available (busdma has one since eventually a request
will complete and unload). The original port was for CAM SIMs which had
stringent requirements to always make progress on I/Os to avoid deadlock.
Rather than put this tricky code in all the CAM SIMs, it's centralized to
retry as soon and as robustly as it can.
Network drivers, however, don't have such requirements and often don't
care. If they can't load a map, they can put the packet back on the
transmit queue (or whatever), set a flag and return without putting it into
the NIC's buffer rings. Eventually, a packet will finish transmitting and
there will be a chance to send it again (or one of a number of other
events, I'm glossing heavily here). It's a better fit for network drivers
than storage drivers. Since bus_dmamap_load_mbuf* is used only for network
drivers, it makes sense.
So, it would only make sense to add if you can cope with BUS_DMA_NOWAIT
always, and that error handling is simpler than letting busdma call you
back when the shortage is over. Is that the case for your port?

I'm not sure. I looked at the OpenBSD code and they always call with a
BUS_DMA_NOWAIT flag, but I'm not sure what the answer is for FreeBSD.

Or is it just because OpenBSD doesn't have this sort of interface (the

Post by Warner Losh
callback difference dates back to the initial port to FreeBSD by Justin
Gibbs because it never got folded back into NetBSD for reasons too long,
too old and only half remembered by me for this post)? Can the code you are
porting cope with error returns and retry again later?

The other problem with using callbacks is that because I pass the
xen-specific callback to the dma load, the driver using the xen specific
dma operation can't pass it's own callback. I thought using the _sg()
variant would allow the driver to use a callback. But the way you put it,
that kind of defeats the purpose of the callback, which is to use it when
allocation is defered.

The bottom line is, I need to re-evaluate. I will spend some more time on
this and discuss it with my mentors. I'll let you (and everyone else) know
if I do decide we need it.

Thanks for the enlightening answer.

Regards,
Pratyush Yadav