Discussion:
Realtek re(4) driver
Dieter BSD
2018-04-23 05:57:07 UTC
Permalink
With several days more data, the Realtek driver is slightly different
than the stock FreeBSD 10.3 driver, but it still fails a lot, with both
TCP and UDP.
Has anyone tried similar testing?
Suggestions of knobs to turn, or other things to try?
I need to receive data via UDP without dropping packets.
Closed source sucks, but I'm stuck with it, and thus with UDP.
Everyone blames re, but AFAICT
no-one switched to a non-Realtek chip to run tests and prove that,
yes, re is really the cause of all the network problems.
I have a box with nfe(4) and bge(4) (BCM5750) that can receive data
with UDP or TCP without error. This has been working for several years.
But a newer, faster machine with re(4) 8111E & 8111F cannot. The nfe is
in the chipset, and the BCM5750 is onboard. I had google look for an
expansion card with BCM5750 but found nothing. I'm looking into
a Broadcom 5719 card, but there is at least one open PR against it, so...
surely it's not a NFS issue
NFS is a bug. By design. "Not a File System"
My machines are NFS-free.
The only thing that link all bug reports
(and there are tons) is the use of re card sustaining gigabit (duplex)
transfers.
My re run (1000baseT <full-duplex>) but the pauses happen even with
light traffic. You don't have to run them full blast with rcp/ftp/whatever
to get failures.
Well, but you know that the U in UDP means unreliable?
Something like a rcp(1) with another
FreeBSD box merely runs slower due to the stalls, but if the other end
has buggy network code and/or if the transfer is forced to use UDP
(Unreliable Data Protocol) data is lost. :-(
With reasonably decent network software and TCP, this is only a minor
problem. However with buggy network software (like a "black box"
with closed source firmware and maybe a transmit buffer that is *way*
too small, and/or true real time requirements), and/or with a brain dead
protocol like UDP, you can lose data.
rcp(1) to 8111F using Realtek driver
Both machines basically idle.
mtu=9000 on both ends but appears to be using 1500 anyway?
netstat -w 1 -d -I re0

input re0 output
packets errs idrops bytes packets errs bytes colls drops
5 0 0 330 6 0 755 0 0
4 0 0 264 4 0 564 0 0
5 0 0 331 5 0 630 0 0
17454 0 0 26176844 17453 0 1152264 0 0
67883 0 0 101023187 67880 0 4481248 0 0
52133 0 0 77141014 52140 0 3444580 0 0
72410 0 0 107980584 72409 0 4780384 0 0
80736 0 0 120645177 80735 0 5328810 0 0
71898 0 0 107298972 71899 0 4745634 0 0
14452 0 0 21510681 14475 0 955577 0 0
5 0 0 330 6 0 755 0 0
4 0 0 264 4 0 564 0 0
5 0 0 331 5 0 630 0 0
4 0 0 264 4 0 564 0 0
65323 0 0 97533279 65321 0 4311552 0 0
28383 0 0 42398766 28385 0 1873769 0 0
80759 0 0 120315366 80756 0 5330394 0 0
80992 0 0 120693553 80995 0 5345772 0 0
80932 0 0 120692488 80930 0 5341746 0 0
78905 0 0 117912275 78922 0 5209152 0 0
80158 0 0 119592452 80159 0 5290728 0 0
input re0 output
packets errs idrops bytes packets errs bytes colls drops
80851 0 0 120590446 80852 0 5336591 0 0
80849 0 0 120589243 80847 0 5336334 0 0
80858 0 0 120598524 80860 0 5336928 0 0
15085 0 0 22529747 15107 0 997296 0 0
4 0 0 264 4 0 564 0 0
4 0 0 264 4 0 564 0 0
5 0 0 331 5 0 630 0 0
4 0 0 264 4 0 564 0 0
63944 0 0 95447185 63943 0 4220604 0 0
80588 0 0 120276192 80589 0 5319233 0 0
80721 0 0 120480130 80722 0 5328011 0 0
79870 0 0 119139685 79870 0 5271720 0 0
79425 0 0 118587402 79442 0 5243472 0 0
80609 0 0 120333747 80609 0 5320494 0 0
80852 0 0 120624944 80850 0 5336532 0 0
80827 0 0 120555558 80831 0 5335073 0 0
79185 0 0 118137499 79185 0 5226510 0 0
76417 0 0 114252562 76431 0 5045010 0 0
79896 0 0 119330393 79900 0 5273436 0 0
79455 0 0 118627478 79456 0 5244330 0 0
77667 0 0 116033902 77662 0 5126256 0 0
input re0 output
packets errs idrops bytes packets errs bytes colls drops
79517 0 0 118735507 79515 0 5248547 0 0
78557 0 0 117304898 78578 0 5186118 0 0
79607 0 0 118742143 79625 0 5255616 0 0
79776 0 0 119140720 79775 0 5265450 0 0
75105 0 0 112725906 75107 0 4957230 0 0
79214 0 0 118774181 79214 0 5228424 0 0
13090 0 0 19520508 13113 0 865692 0 0

Same as above except to 2nd 8111F port.

input re1 output
packets errs idrops bytes packets errs bytes colls drops
75170 0 0 112288950 75165 0 4960956 0 0
74149 0 0 110808698 74150 0 4893834 0 0
75088 0 0 112276960 75088 0 4955808 0 0
74567 0 0 111313160 74566 0 4921356 0 0
77064 0 0 115370280 77065 0 5086290 0 0
71606 0 0 107343414 71622 0 4727052 0 0
70831 0 0 105956278 70832 0 4674849 0 0
70727 0 0 106127094 70725 0 4667916 0 0
73427 0 0 110111478 73425 0 4846182 0 0
72519 0 0 108763678 72522 0 4786320 0 0
5661 0 0 8423884 5681 0 374880 0 0
0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
2 0 0 196 0 0 0 0 0
1 0 0 98 0 0 0 0 0
563 0 0 838984 561 0 37026 0 0
29144 0 0 43754296 29142 0 1923438 0 0
73986 0 0 110909436 73987 0 4883142 0 0
39601 0 0 59343834 39621 0 2614920 0 0
0 0 0 0 0 0 0 0 0
3 0 0 201 0 0 0 0 0
input re1 output
packets errs idrops bytes packets errs bytes colls drops
0 0 0 0 0 0 0 0 0
1 0 0 98 0 0 0 0 0
567 0 0 839958 567 0 37422 0 0
1 0 0 98 0 0 0 0 0
38003 0 0 56780696 38000 0 2508066 0 0
75439 0 0 112523550 75438 0 4978908 0 0
77058 0 0 114890108 77059 0 5085894 0 0
77471 0 0 115473190 77470 0 5113020 0 0
76814 0 0 114644092 76815 0 5069790 0 0
78120 0 0 116633730 78118 0 5155788 0 0
74853 0 0 111727082 74854 0 4940298 0 0
77626 0 0 115697756 77623 0 5123250 0 0
70804 0 0 105582200 70804 0 4672998 0 0
76722 0 0 114492148 76722 0 5063652 0 0
29562 0 0 44157654 29582 0 1952346 0 0
0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
568 0 0 840298 566 0 37356 0 0
0 0 0 0 0 0 0 0 0
input re1 output
packets errs idrops bytes packets errs bytes colls drops
43130 0 0 64590180 43128 0 2846514 0 0
29010 0 0 43374204 29032 0 1916046 0 0
0 0 0 0 0 0 0 0 0
2 0 0 134 0 0 0 0 0
0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
515 0 0 770942 515 0 33990 0 0
17903 0 0 26853958 17902 0 1181598 0 0
74854 0 0 111699988 74851 0 4940166 0 0
77120 0 0 115061536 77117 0 5089722 0 0
76857 0 0 114664362 76855 0 5072562 0 0
75817 0 0 113174146 75820 0 5003988 0 0
76737 0 0 114435914 76737 0 5064642 0 0
78495 0 0 117128814 78493 0 5180604 0 0
75723 0 0 112970948 75724 0 4997718 0 0
76501 0 0 114196604 76500 0 5049066 0 0
76187 0 0 113779382 76189 0 5028408 0 0
77236 0 0 115269384 77235 0 5097510 0 0
75300 0 0 112389128 75300 0 4969800 0 0
76492 0 0 114455568 76492 0 5048472 0 0
75573 0 0 113011418 75572 0 4987752 0 0
input re1 output
packets errs idrops bytes packets errs bytes colls drops
73331 0 0 109541752 73330 0 4839780 0 0
76732 0 0 114631816 76733 0 5064378 0 0
74400 0 0 111067456 74399 0 4910334 0 0
75974 0 0 113463012 75975 0 5014350 0 0
76312 0 0 113999448 76311 0 5036526 0 0
43806 0 0 65415884 43844 0 2893638 0 0
1 0 0 66 0 0 0 0 0
0 0 0 0 0 0 0 0 0

Same as above except to onboard 8111E.

input re2 output
packets errs idrops bytes packets errs bytes colls drops
53122 0 0 80345556 0 0 0 0 0
53984 0 0 81332624 0 0 0 0 0
61394 0 0 92785548 0 0 0 0 0
53421 0 0 80696090 0 0 0 0 0
30857 0 0 46597698 0 0 0 0 0
76124 0 0 115239504 0 0 0 0 0
69172 0 0 104706904 0 0 0 0 0
76094 0 0 115191956 0 0 0 0 0
76849 0 0 116329874 0 0 0 0 0
68788 0 0 104120096 0 0 0 0 0
69233 0 0 104802146 0 0 0 0 0
50527 0 0 76465702 0 0 0 0 0
75916 0 0 114920192 0 0 0 0 0
66062 0 0 99990092 0 0 0 0 0
76257 0 0 115436194 0 0 0 0 0
47074 0 0 71233028 0 0 0 0 0
51788 0 0 78359104 0 0 0 0 0
44486 0 0 67316276 0 0 0 0 0
76159 0 0 115288566 0 0 0 0 0
64766 0 0 98037852 0 0 0 0 0
76786 0 0 116237604 0 0 0 0 0
input re2 output
packets errs idrops bytes packets errs bytes colls drops
50676 0 0 76688424 0 0 0 0 0
76918 0 0 116440932 0 0 0 0 0
65723 0 0 99488694 0 0 0 0 0
73756 0 0 111653288 0 0 0 0 0
71393 0 0 108070674 0 0 0 0 0
63089 0 0 95485346 0 0 0 0 0
69588 0 0 105333248 0 0 0 0 0
59215 0 0 89617670 0 0 0 0 0
57339 0 0 86752950 0 0 0 0 0
75739 0 0 114651942 0 0 0 0 0
68442 0 0 103594692 0 0 0 0 0
49990 0 0 75649100 0 0 0 0 0
27710 0 0 41940348 0 0 0 0 0
50005 0 0 75694058 0 0 0 0 0
76748 0 0 116183384 0 0 0 0 0
52088 0 0 78847168 0 0 0 0 0
76816 0 0 116287048 0 0 0 0 0
70232 0 0 106301624 0 0 0 0 0
76467 0 0 115754502 0 0 0 0 0
68979 0 0 104412918 0 0 0 0 0
76694 0 0 116101068 0 0 0 0 0
input re2 output
packets errs idrops bytes packets errs bytes colls drops
68857 0 0 104224114 0 0 0 0 0
76545 0 0 115871154 0 0 0 0 0
65996 0 0 99843912 0 0 0 0 0
76674 0 0 116070772 0 0 0 0 0
68095 0 0 103070230 0 0 0 0 0
74877 0 0 113319642 0 0 0 0 0
39228 0 0 59366824 0 0 0 0 0
53026 0 0 80203068 0 0 0 0 0

...

77799 0 0 117779926 0 0 0 0 0
72666 0 0 110007220 0 0 0 0 0
69173 0 0 104681578 0 0 0 0 0
55918 0 0 84169172 0 0 0 0 0
24127 0 0 36195390 0 0 0 0 0
input re2 output
packets errs idrops bytes packets errs bytes colls drops
0 0 0 0 0 0 0 0 0
19706 0 0 29578468 0 0 0 0 0
53262 0 0 79974084 0 0 0 0 0
59033 0 0 88835122 0 0 0 0 0
54469 0 0 81830514 0 0 0 0 0
55005 0 0 82842706 0 0 0 0 0
47867 0 0 72331934 0 0 0 0 0
65468 0 0 99099624 0 0 0 0 0
80572 0 0 121980632 0 0 0 0 0
58201 0 0 88091802 0 0 0 0 0
51948 0 0 78614144 0 0 0 0 0
80053 0 0 121196242 0 0 0 0 0
52331 0 0 79200470 0 0 0 0 0
80123 0 0 121302582 0 0 0 0 0
43414 0 0 65698001 0 0 0 0 0
77961 0 0 117995130 0 0 0 0 0
62287 0 0 94226926 0 0 0 0 0
75558 0 0 114290892 0 0 0 0 0
64228 0 0 97158320 0 0 0 0 0
59981 0 0 90654186 0 0 0 0 0
43520 0 0 65739096 0 0 0 0 0
input re2 output
packets errs idrops bytes packets errs bytes colls drops
56570 0 0 85335700 0 0 0 0 0
50066 0 0 75384196 0 0 0 0 0
55959 0 0 84239934 0 0 0 0 0
53192 0 0 80101232 0 0 0 0 0
49002 0 0 73982980 0 0 0 0 0
49713 0 0 74993370 0 0 0 0 0
43984 0 0 66011800 0 0 0 0 0
54324 0 0 81935448 0 0 0 0 0
48502 0 0 72734828 0 0 0 0 0
26191 0 0 39411806 0 0 0 0 0
0 0 0 0 0 0 0 0 0
42582 0 0 64021036 0 0 0 0 0
47216 0 0 70880104 0 0 0 0 0
52239 0 0 78498398 0 0 0 0 0
52517 0 0 79279690 0 0 0 0 0
44212 0 0 66384032 0 0 0 0 0
57225 0 0 86126642 0 0 0 0 0
47902 0 0 72238300 0 0 0 0 0
44933 0 0 67508656 0 0 0 0 0
54967 0 0 82411976 0 0 0 0 0
50724 0 0 76288600 0 0 0 0 0

...

input re2 output
packets errs idrops bytes packets errs bytes colls drops
56426 0 0 84726988 0 0 0 0 0
53332 0 0 79840392 0 0 0 0 0
52763 0 0 78943630 0 0 0 0 0
52190 0 0 78016892 0 0 0 0 0
56889 0 0 85597554 0 0 0 0 0
55265 0 0 82915642 0 0 0 0 0
52162 0 0 78061690 0 0 0 0 0
58041 0 0 87326532 0 0 0 0 0
53244 0 0 79740088 0 0 0 0 0
51387 0 0 76791862 0 0 0 0 0
54187 0 0 81106878 0 0 0 0 0
51846 0 0 77541604 0 0 0 0 0
42316 0 0 63282584 0 0 0 0 0
0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
590 0 0 890908 0 0 0 0 0
40253 0 0 60411290 0 0 0 0 0
56653 0 0 85264434 0 0 0 0 0
53499 0 0 80125838 0 0 0 0 0
[ ... ]
input re2 output
packets errs idrops bytes packets errs bytes colls drops
52839 0 0 78935238 0 0 0 0 0
53098 0 0 79307900 0 0 0 0 0
49559 0 0 74231830 0 0 0 0 0
79905 0 0 120972890 0 0 0 0 0
66836 0 0 101174008 0 0 0 0 0
75308 0 0 113998016 0 0 0 0 0
47258 0 0 71543932 0 0 0 0 0
32 0 0 48448 0 0 0 0 0
70026 0 0 106002764 0 0 0 0 0
63937 0 0 96604786 0 0 0 0 0
60947 0 0 92062662 0 0 0 0 0
63772 0 0 96410208 0 0 0 0 0
58129 0 0 87582154 0 0 0 0 0
59204 0 0 89190072 0 0 0 0 0
61375 0 0 92741230 0 0 0 0 0
69159 0 0 104622734 0 0 0 0 0
75748 0 0 114661800 0 0 0 0 0
75478 0 0 114253980 0 0 0 0 0
76814 0 0 116282228 0 0 0 0 0
76931 0 0 116458670 0 0 0 0 0
75299 0 0 113979718 0 0 0 0 0
YongHyeon PYUN
2018-04-24 05:45:03 UTC
Permalink
Post by Dieter BSD
With several days more data, the Realtek driver is slightly different
than the stock FreeBSD 10.3 driver, but it still fails a lot, with both
TCP and UDP.
[...]
Post by Dieter BSD
My re run (1000baseT <full-duplex>) but the pauses happen even with
light traffic. You don't have to run them full blast with rcp/ftp/whatever
to get failures.
I guess benchmarks/netperf or benchmarks/iperf is better tool to
measure performance of ethernet driver. rcp/ftp involves other IOs
with network IO so you may measure performance of other subsystem.
If you see intermittent pauses during the test, try enabling or
disabling ethernet flow control of the controller.

# Disable flowcontrol
#ifconfig re0 media auto -mediaopt flow


# Enable flowcontrol
#ifconfig re0 media auto mediaopt flow

[...]
Post by Dieter BSD
rcp(1) to 8111F using Realtek driver
Both machines basically idle.
mtu=9000 on both ends but appears to be using 1500 anyway?
Cached route may cache the MTU of the interface. You may have to
down and re-up the interface to reflect the change.
Post by Dieter BSD
netstat -w 1 -d -I re0
input re0 output
packets errs idrops bytes packets errs bytes colls drops
5 0 0 330 6 0 755 0 0
4 0 0 264 4 0 564 0 0
5 0 0 331 5 0 630 0 0
17454 0 0 26176844 17453 0 1152264 0 0
67883 0 0 101023187 67880 0 4481248 0 0
52133 0 0 77141014 52140 0 3444580 0 0
72410 0 0 107980584 72409 0 4780384 0 0
80736 0 0 120645177 80735 0 5328810 0 0
71898 0 0 107298972 71899 0 4745634 0 0
14452 0 0 21510681 14475 0 955577 0 0
5 0 0 330 6 0 755 0 0
4 0 0 264 4 0 564 0 0
5 0 0 331 5 0 630 0 0
4 0 0 264 4 0 564 0 0
65323 0 0 97533279 65321 0 4311552 0 0
28383 0 0 42398766 28385 0 1873769 0 0
80759 0 0 120315366 80756 0 5330394 0 0
80992 0 0 120693553 80995 0 5345772 0 0
80932 0 0 120692488 80930 0 5341746 0 0
78905 0 0 117912275 78922 0 5209152 0 0
80158 0 0 119592452 80159 0 5290728 0 0
These counters are maintained in driver but it's not from MAC of
the controller. So you can't tell how much RX packets were dropped
before driver sees them. There is an undocumented sysctl variable
which reads H/W MAC counters and output them on console.

#sysctl dev.re.0.stats=1

Loading...