How to Test Microsoft Server 2012 R2 Upload Test
Windows server upload speed issues
Hello, so we have some problem that i cant find any solution..
We accept a server with windows server 2019 (latest updates) connected 10G network card straight to 10G router and next Internet access provider router.
nosotros get 9Gb/s download and ~1,5Gb/s upload.
when downloading over cyberspace from that server we get only 10MB/s speed
When we change network card speed to 1GB, download from same server to aforementioned computer go up to ~50MB/south
I was trying to modify network adapter settings, merely nothing helped.
This problem just occurs on windows server, on linux everything become normal and we get fifty-fifty more than speed.
Only is very strange that if i connect my network carte du jour to 1G i get the speed that clients want.. with 10G its as well boring..
I installed windows server 2022 and the speed a scrap increased
but even so not so fast when with 1G
How-do-you-do @AurimasKurauskas-5863,
The observed behaviour may be counter-intuitive, only information technology would be necessary to compare traces of the 10G and 1G cards earlier last that the behaviour is inexplicable.
Some "auto-tuning" of TCP behaviour is based on estimates of the bandwidth and latency product. It may exist the case that this car-tuning is pushing the behaviour into a region where (when combined with the congestion control mechanism) the increased bandwidth results in increased packet losses and more severe congestion control steps.
If you are prepared to create traces for the two configurations and share them, and then permit me know - I tin can then suggest what test steps would be useful.
Gary
I am prepared for anything that would ostend that is not our trouble.
I tin make many test but at this moment i dont know what to examination.
Hello @AurimasKurauskas-5863,
Here is another thread on Microsoft Q&A describing substantially the aforementioned problem: https://docs.microsoft.com/en-us/answers/questions/89768/dull-wired-upload-speed-vs-linux-on-aforementioned-hardware.html
The control that I would propose to start a trace is pktmon start --capture --comp nics --flags 0x10 --trace --provider Microsoft-Windows-TCPIP --keywords 0x200700000000 --level 16 --file-name why.etl and the command to stop the trace is pktmon stop
The trace needs to be run on the sending organization (the system where the congestion control is taking place). In your instance this is the server. The trace but needs to capture a few seconds of activeness during the "upload" (from the server to the client). The trace will capture all network activity of the server during the trace. The trace volition abound in size quite quickly, then it is worthwhile trying to construct a curt test scenario that "captures"/exhibits the problem.
Most (actually all) of the performance issues that I have helped with in this forum occurred when the client was sending data (and the traces were made on the client system); in that location are probably more "confidentiality" issues when tracing on a server. If you are still willing to share the captured data, I would suggest making information technology available via a file sharing service such that readers take to "request" access. Alternatively you can try to contact me directly (perhaps via LinkedIn - I am the but "Gary Nebbett" at that place) to hash out other transfer mechanisms.
Gary
Yeh i readed that thread but nothing helped and i didint notice out why on 1G downloading is good but with 10G not.
this server is for testing so no problems about confidentiality.
I made 2 identical tests.
why.etl file:
10G connection.
1G.etl
1G connection.
Link to files: https://1drv.ms/u/south!Asq_YVUBXCNBkxDpQw8yHJsDCILB?e=Ai6pya
Hello Aurimas,
Hither is some quick feedback, simply in that location is still more analysis and thinking to be done. This is a "visualisation" of the 1G trace:
The blueish line is the transmit sequence number and the dark-green line (not to scale) is the size of the congestion window. As can be seen, the congestion window size goes up and down a lot, but it is generally in the range from 150,000 to 250,000 bytes. The trace shows that LSO (Large Transport Offload) is in use and the trace does non contain any information well-nigh received (acknowledgement) packets; the trace information is too decoupled from the actual on-the-wire behaviour (dues to LSO) to let many conclusions to exist drawn.
This is a visualization of the 10G trace:
The outset half of this trace is similar to the 1G trace (LSO in utilize, no received/acknowledgement packets in the trace data) and so suddenly acknowledgements start to appear in the raw trace information and other TCPIP events appear in the trace. The congestion window ranges in size from 35,000 to 60,000 bytes. In both traces, the send window is stable at 4194304 bytes (this value is chosen past the receiver and is likewise the limit on the congestion window size).
There are some "spurious" retransmissions in the 10G trace (the receiver reports receiving the retransmitted data twice via DSACKs) and this might be an indication of the problem that I mention in https://gary-nebbett.blogspot.com/2022/01/windows-11-tcpip-congestion-control.html.
I volition spend more time looking at the department of the 10G trace that contains acknowledgements (the second half of the trace).
One thing that you lot could endeavour is to disable LSO in both adapters and repeat the traces. I think that the main purpose of LSO is to reduce the computational load on the server (rather than to increase transfer speed) and since this is a examination system, obtaining a clearer view of what is actually happening on-the-wire and the expense of disabling LSO would probably be worthwhile.
Gary
Give thanks you for feedback,
I made test with turned off LSO, speed is better as i can see.
Link to the test results: https://1drv.ms/u/s!Asq_YVUBXCNBkxJvd_l-xx9WFmok?e=pZ62y3
On the network card i enabled these property too:
Jumbo Packet: 9014 Bytes
Receive Buffers: 4096
Transmit Buffers: 16384
Worth to mention that i am using same network card, just change the Speed & Duplex from 10G to 1G.
Will be waiting youre analysis.
Thank y'all
Hi Aurimas,
Thanks for the new data.
As well in these traces, there is no "receive" tracing in the 1G trace and only "receive" events in the last 10% of the 10G trace. I don't nevertheless know how this is possible.
Our top-level goal is to understand the network performance (and what can be done to influence information technology), but to practise that nosotros need reliable and (nearly) complete trace data - and currently we are not getting that. So we at present have an intermediate goal of agreement why the tracing is not working as expected. I will perform a few tests to encounter if I can observe any style of tracing the tracing!
Gary
0 Votes 0 ·
Hello Aurimas,
I am yet working on understanding how the tracing could possibly fail, only here is a trivial bit more information about what is happening when the tracing is working. This is just an excerpt from one of the images previously posted:
The greenish line is the congestion window size; the first row of dots (nearest the top) are the moments at which a "fast retransmit" occurs (triggered by analysis of the received SACKs); the next row of (red) dots (near looks similar a line) are the moments when SACKs are received and the bottom row of dots are DSACKs (RFC 2883).
For every (fast) retransmission there is a corresponding DSACK (which means that the retransmission was unnecessary - the original information somewhen arrived, simply just took so long to do so (and the SACKs sent in the acting triggered the retransmit). These "spurious" (unnecessary) retransmits cause the congestion window to shut a bit. For every plough downward in the green line, there is a fast retransmission and a DSACK. This seems to be happening 10 to 15 times per second in the 10G trace.
In the 1G trace, only the congestion window size is bachelor and this seems to have about 5 hits backwards per 2d. I would estimate that the behaviour is the same every bit the 10G trace: fast retransmit followed by DSACK. Because it happens less often, the congestion window is able to stay wider open up.
Gary
Hello Aurimas,
Upon reflection, I remember that at that place is a very proficient adventure that this is an example of the Windows 11 (and Windows Server 2022) problems that I described in my weblog (referenced in an earlier message in this thread).
Since there are many SACKs but no indication of any genuine packet loss, the data packets are very probably arriving slightly out-of-social club at the receiver.
As the web log entry describes, the issues is that the auto-tuning of the RACK mechanism does not happen because the bespeak to car-melody is the reception of a DSACK, but DSACKs are non being properly recognized. The 10G behaviour is probably requires more tuning of the original RACK parameters than the 1G behaviour and, since the required tuning is not taking identify, the 10G performance ultimately turns out to be worse than the 1G performance.
Gary
0 Votes 0 ·
Hi Aurimas,
Here is a physical example of a "unnecessary" retransmission:
15:28:51.097963 server.50076 > client.60345: . 62673456:62674916(1460) ack 81 win 8192 (DF) fifteen:28:51.101547 client.60345 > server.50076: . ack 62673456 win 32768 (DF) 15:28:51.101548 client.60345 > server.50076: . ack 62673456 win 32768 <nop,nop,sack 62676376:62677836> (DF) 15:28:51.101549 client.60345 > server.50076: . ack 62673456 win 32768 <nop,nop,sack 62679296:62682216 62676376:62677836> (DF) 15:28:51.101632 client.60345 > server.50076: . ack 62673456 win 32768 <nop,nop,sack 62685136:62686596 62679296:62682216 62676376:62677836> (DF) xv:28:51.101633 customer.60345 > server.50076: . ack 62674916 win 32768 <nop,nop,sack 62685136:62686596 62679296:62682216 62676376:62677836> (DF) 15:28:51.101645 server.50076 > client.60345: . 62673456:62674916(1460) ack 81 win 8192 (DF) 15:28:51.105159 client.60345 > server.50076: . ack 62709956 win 32768 <nop,nop,sack 62673456:62674916> (DF) These are just "selected" packets from the trace, all containing a reference to the sequence number 62673456. This is how I translate that:
-
The packet is starting time sent.
-
All prior packets are acknowledged up to the package.
-
one packet, sent later, is selectively acknowledged.
-
2 packets, sent afterwards, are selectively acknowledged.
-
3 packets, sent afterward, are selectively acknowledged.
-
The bundle is acknowledged, along with some selective acknowledgements.
-
The bundle is retransmitted.
-
A indistinguishable selective acknowledgement (DSACK) for the bundle is received.
The ordering of steps 6 and vii may seem a bit odd, but this is just due to the asynchrony of the processing and logging. After step five (iii reports that the packet is missing), a retransmission is initiated (queued); before the retransmission is actually sent, the "missing" acknowledgement arrives but it is now likewise late to stop the queued retransmission.
If the acknowledgement for the packet had arrived 1 microsecond earlier (before the packet was reported "missing" for a third fourth dimension) then there would have been no problem. These small intervals of time are probably an indication that the "reordering" of packets is happening as the packets pass through some intermediate network device that processes packets in parallel.
The "big" event that these pocket-sized glitches take is on the size of the sender's congestion window. It won't be practicable to eliminate the small amount of reordering, so the congestion control mechanism needs to exist cleverer.
Gary
Source: https://docs.microsoft.com/answers/questions/710439/windows-server-upload-speed-problems.html
Hello @AurimasKurauskas-5863,
Did you finish the traces (
pktmon cease) before copying them? Both files just contain three.6 megabytes of the binary value 0. The cause might be that in-memory information was non flushed to disk (which would happen at the latest when a trace was stopped).Gary
0 Votes 0 ·
yes i stopped the traces.
0 Votes 0 ·
How-do-you-do @AurimasKurauskas-5863,
Perhaps there was some trouble with the upload. If you download one of the files from your link and try the tracerpt control over again, I would expect you to see:
Input
File(south):
\Users\Gary\Downloads\why.etl
0.00%
Error:
The file or directory is corrupted and unreadable.
Gary
0 Votes 0 ·
Show more comments