-2

Recently I encounter a very puzzuling problem. We are doing AI traingings based on some type of NPU device. The training usually read/write data from/to some NFS directory, and during the training we found the network device became down status.

We tried executing ip link set dev xxxx up, only to find that it doesn't take any effect in the real status of this device. It keeps down status until we reboots the machine.

Via the system log (ubuntu 18.04) and the NIC driver log, we know something is wrong either with the card or the driver upon the card. The driver keeps complaining something like "update mac stats fail, get mac pkt stats fail" etc.

The most interesting and difficult thing is to figure out what application behavior triggers this issue and how that makes it happens.

How can a user-space application cause the NIC to go down? Is it possible to figure out what happens there without having the source code of the NIC driver?

Please help me to explain this, or give me some suggestions about how to figure it out, or list some documentations on it.

Improve this question
4
  • 4
    Guess: bug in firmware or driver. You can start with editing the question and including details about hardware, firmware and driver used. Some vendors are notorious for bad drivers or firmware. Unless you become intimately familiar with the details of your hardware, chances of fixing it are low.
    – dirkt
    Sep 27, 2020 at 6:01
  • 4
    Adding actual error messages and logs would be helpful.
    – ThatOneDude
    Oct 6, 2020 at 15:19
  • A shot in the dark, give it a look 1 and 2 maybe you can find some hints.
    – Hastur
    Oct 7, 2020 at 11:51
  • How does it make sense to start a bounty rather than simply include the requested info in the question?
    – Alex M
    Oct 9, 2020 at 22:41

1 Answer 1

Reset to default
0

I had a similar issue, my Solarflare cards' interfaces went down during some workloads. I've noticed that it's enough to unload and load sfc kernel module responsible for driving the card.

I don't know what hardware you're running or what are the actual errors, but see whether unloading and loading the kernel module will help (watch out for losing network connectivity). You can also try upgrading the firmware on the card.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .