vRouer w/DPDK and Mellanox NIC

[nirhenn]

Hi All,

I am trying to setup vRouter w/DPDK and ConnectX-3-Pro.
Is there any documentation how to?

The problem is that I cannot bring vif to recognize the DPPK enabled interface.
The current state is that install script identifies correctly and interface is binded to the DPDK.

root@Setup-H-C0:/opt/contrail/bin# ./dpdk_nic_bind.py -s

Network devices using DPDK-compatible driver

0000:05:00.0 'MT27520 Family [ConnectX-3 Pro]' drv=igb_uio unused=

Network devices using kernel driver

0000:03:00.0 'I350 Gigabit Network Connection' if=em0 drv=igb unused=igb_uio Active
0000:03:00.3 'I350 Gigabit Network Connection' if=eth1 drv=igb unused=igb_uio

Other network devices

Since OFED + PMD drivers are taking over it is not in control of the OS any more and does not show up in ifconfig.

in - /etc/contrail/contrail-vrouter-agent.conf

DPDK or legacy work mode

platform=dpdk

Physical address of PCI used by dpdk

physical_interface_address=0000:05:00.0

MAC address of device used by dpdk

physical_interface_mac=7c:fe:90:b1:ea:50

in - /etc/contrail/agent_param

LOG=/var/log/contrail.log
CONFIG=/etc/contrail/contrail-vrouter-agent.conf
prog=/usr/bin/contrail-vrouter-agent
kmod=vrouter
pname=contrail-vrouter-agent
LIBDIR=/usr/lib64
DEVICE=vhost0
dev=p785p1
vgw_subnet_ip=VGW_SUBNET_IP
vgw_intf=VGW_INTF_LIST
LOGFILE=--log-file=/var/log/contrail/vrouter.log

[nirhenn]

Adding some more finding...

Continue our earlier call, from contrail point of view it looks all pointing to the right config, pointing to the right PCI bus/slot and MAC address.
When we exec:
root@Setup-H-C0:~# /opt/contrail/bin/dpdk_nic_bind.py -s

Network devices using DPDK-compatible driver

0000:05:00.0 'MT27520 Family [ConnectX-3 Pro]' drv=igb_uio unused=

Network devices using kernel driver

0000:03:00.0 'I350 Gigabit Network Connection' if=em0 drv=igb unused=igb_uio Active
0000:03:00.3 'I350 Gigabit Network Connection' if=eth1 drv=igb unused=igb_uio

Other network devices

root@Setup-H-C0:~#

What is the meaning of the Active ????
From the vrouter log we can see that instead of working with 05:00.0 PCI bus/slot as defined in the setup/config it takes 03:00.0 as show in the dpdk_nic_bind.

root@Setup-H-C0:/var/log/contrail# less contrail-vrouter-dpdk-stdout.log

EAL: PCI device 0000:03:00.0 on NUMA socket 0
EAL: probe driver: 8086:1521 rte_igb_pmd
EAL: Not managed by a supported kernel driver, skipped

Nir.

[srajag]

“Active” might mean that it is in use by the kernel driver (I didn’t confirm by looking at the dpdk_nic_bind script,
but we see it on our setup too). Do you see an vif add messages in contrail-vrouter-dpdk-stdout.log? Does
“contrail-status” report any errors? Also, please look in the agent log for errors (/var/log/contrail/contrail-vrouter-agent.log)

Raja

From: Nir Henn notifications@github.com
Reply-To: Juniper/contrail-vrouter reply@reply.github.com
Date: Sunday, September 18, 2016 at 7:34 AM
To: Juniper/contrail-vrouter contrail-vrouter@noreply.github.com
Subject: Re: [Juniper/contrail-vrouter] vRouer w/DPDK and Mellanox NIC (#94)

Adding some more finding...

Continue our earlier call, from contrail point of view it looks all pointing to the right config, pointing to the right PCI bus/slot and MAC address.
When we exec:
root@Setup-H-C0:~# /opt/contrail/bin/dpdk_nic_bind.py -s

Network devices using DPDK-compatible driver

0000:05:00.0 'MT27520 Family [ConnectX-3 Pro]' drv=igb_uio unused=

Network devices using kernel driver

0000:03:00.0 'I350 Gigabit Network Connection' if=em0 drv=igb unused=igb_uio Active
0000:03:00.3 'I350 Gigabit Network Connection' if=eth1 drv=igb unused=igb_uio

Other network devices

root@Setup-H-C0:~#

What is the meaning of the Active ????
From the vrouter log we can see that instead of working with 05:00.0 PCI bus/slot as defined in the setup/config it takes 03:00.0 as show in the dpdk_nic_bind.

root@Setup-H-C0:/var/log/contrail# less contrail-vrouter-dpdk-stdout.log

EAL: PCI device 0000:03:00.0 on NUMA socket 0
EAL: probe driver: 8086:1521 rte_igb_pmd
EAL: Not managed by a supported kernel driver, skipped

Nir.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://github.com//issues/94#issuecomment-247851185, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AFni16RBg6CpWbCEMSerS6pK8IQLOM_Iks5qrUwQgaJpZM4J_27s.

[nirhenn]

The two compute nodes show:
== Contrail vRouter ==
supervisor-vrouter: active
contrail-vrouter-agent initializing (Collector, Discovery:Collector, Discovery:dns-server, Discovery:xmpp-server connection down)
contrail-vrouter-dpdk active
contrail-vrouter-nodemgr active

root@Setup-H-C1:~# contrail-status
== Contrail vRouter ==
supervisor-vrouter: active
contrail-vrouter-agent initializing (Collector, Discovery:Collector, Discovery:dns-server, Discovery:xmpp-server connection down)
contrail-vrouter-dpdk active
contrail-vrouter-nodemgr active

The controller shows:
root@Setup-H-Master:~# contrail-status
== Contrail Control ==
supervisor-control: active
contrail-control active
contrail-control-nodemgr active
contrail-dns active
contrail-named active

== Contrail Analytics ==
supervisor-analytics: active
contrail-alarm-gen active
contrail-analytics-api active
contrail-analytics-nodemgr active
contrail-collector active
contrail-query-engine active
contrail-snmp-collector active
contrail-topology active

== Contrail Config ==
supervisor-config: active
contrail-api:0 active
contrail-config-nodemgr active
contrail-device-manager active
contrail-discovery:0 active
contrail-schema active
contrail-svc-monitor active
ifmap active

== Contrail Web UI ==
supervisor-webui: active
contrail-webui active
contrail-webui-middleware active

== Contrail Database ==
contrail-database: active
supervisor-database: active
contrail-database-nodemgr active
kafka active

== Contrail Support Services ==
supervisor-support-service: active
rabbitmq-server active

root@Setup-H-Master:~#

on the vif it shows only one interface 0/2 but no vhost0

Potential problem from the contrail-vrouter-dpdk-stdout.log….
You can see that the DPDK boot binds to the wrong interface even that in then config it is the correct one.

Does the DPDK build is linked with Mellanox OFED drivers correctly?

• echo 'Thu Sep 15 17:35:44 IDT 2016: Creating vhost0 interface with vif utility...'
  Thu Sep 15 17:35:44 IDT 2016: Creating vhost0 interface with vif utility...
• '[' -z 7c:fe:90:b1:ea:50 ']'
• '[' -z 0000:05:00.0 ']'
  ++ date
• echo 'Thu Sep 15 17:35:44 IDT 2016: Adding p785p1 interface with vif utility...'
  Thu Sep 15 17:35:44 IDT 2016: Adding p785p1 interface with vif utility...
• vif --add 0 --mac 7c:fe:90:b1:ea:50 --vrf 0 --vhost-phys --type physical --pmd --id 0
  vRouter(Response): No such file or directory (2)
• '[' 0 '!=' 0 ']'
  ++ date
• echo 'Thu Sep 15 17:35:44 IDT 2016: Adding vhost0 interface with vif utility...'
  Thu Sep 15 17:35:44 IDT 2016: Adding vhost0 interface with vif utility...
• vif --add vhost0 --mac 7c:fe:90:b1:ea:50 --vrf 0 --type vhost --xconnect 0 --pmd --id 1
  vRouter(Response): No such file or directory (2)
• '[' 0 '!=' 0 ']'
  ++ date
• echo 'Thu Sep 15 17:35:44 IDT 2016: Done starting vRouter/DPDK.'
  Thu Sep 15 17:35:44 IDT 2016: Done starting vRouter/DPDK.
• return 0
• return 0
  Thu Sep 15 17:35:44 IDT 2016: Value VGW_SUBNET_IP

[srajag]

We have not tested with Mellanox NICs. We use DPDK 2.1. I am not sure if it has the drivers for Mellanox NICs.

Raja

From: Nir Henn notifications@github.com
Reply-To: Juniper/contrail-vrouter reply@reply.github.com
Date: Monday, September 19, 2016 at 10:30 AM
To: Juniper/contrail-vrouter contrail-vrouter@noreply.github.com
Cc: Raja Sivaramakrishnan raja@juniper.net, Comment comment@noreply.github.com
Subject: Re: [Juniper/contrail-vrouter] vRouer w/DPDK and Mellanox NIC (#94)

The two compute nodes show:
== Contrail vRouter ==
supervisor-vrouter: active
contrail-vrouter-agent initializing (Collector, Discovery:Collector, Discovery:dns-server, Discovery:xmpp-server connection down)
contrail-vrouter-dpdk active

contrail-vrouter-nodemgr active

root@Setup-H-C1:~# contrail-status
== Contrail vRouter ==
supervisor-vrouter: active
contrail-vrouter-agent initializing (Collector, Discovery:Collector, Discovery:dns-server, Discovery:xmpp-server connection down)
contrail-vrouter-dpdk active

contrail-vrouter-nodemgr active

The controller shows:
root@Setup-H-Master:~# contrail-status
== Contrail Control ==
supervisor-control: active
contrail-control active

contrail-control-nodemgr active

contrail-dns active

contrail-named active

== Contrail Analytics ==
supervisor-analytics: active
contrail-alarm-gen active

contrail-analytics-api active

contrail-analytics-nodemgr active

contrail-collector active

contrail-query-engine active

contrail-snmp-collector active

contrail-topology active

== Contrail Config ==
supervisor-config: active
contrail-api:0 active

contrail-config-nodemgr active

contrail-device-manager active

contrail-discovery:0 active

contrail-schema active

contrail-svc-monitor active

ifmap active

== Contrail Web UI ==
supervisor-webui: active
contrail-webui active

contrail-webui-middleware active

== Contrail Database ==
contrail-database: active
supervisor-database: active
contrail-database-nodemgr active

kafka active

== Contrail Support Services ==
supervisor-support-service: active
rabbitmq-server active

root@Setup-H-Master:~#

on the vif it shows only one interface 0/2 but no vhost0

Potential problem from the contrail-vrouter-dpdk-stdout.log….
You can see that the DPDK boot binds to the wrong interface even that in then config it is the correct one.

Does the DPDK build is linked with Mellanox OFED drivers correctly?

• echo 'Thu Sep 15 17:35:44 IDT 2016: Creating vhost0 interface with vif utility...' Thu Sep 15 17:35:44 IDT 2016: Creating vhost0 interface with vif utility...
• '[' -z 7c:fe:90:b1:ea:50 ']'
• '[' -z 0000:05:00.0 ']' ++ date
• echo 'Thu Sep 15 17:35:44 IDT 2016: Adding p785p1 interface with vif utility...' Thu Sep 15 17:35:44 IDT 2016: Adding p785p1 interface with vif utility...
• vif --add 0 --mac 7c:fe:90:b1:ea:50 --vrf 0 --vhost-phys --type physical --pmd --id 0 vRouter(Response): No such file or directory (2)
• '[' 0 '!=' 0 ']' ++ date
• echo 'Thu Sep 15 17:35:44 IDT 2016: Adding vhost0 interface with vif utility...' Thu Sep 15 17:35:44 IDT 2016: Adding vhost0 interface with vif utility...
• vif --add vhost0 --mac 7c:fe:90:b1:ea:50 --vrf 0 --type vhost --xconnect 0 --pmd --id 1 vRouter(Response): No such file or directory (2)
• '[' 0 '!=' 0 ']' ++ date
• echo 'Thu Sep 15 17:35:44 IDT 2016: Done starting vRouter/DPDK.' Thu Sep 15 17:35:44 IDT 2016: Done starting vRouter/DPDK.
• return 0
• return 0 Thu Sep 15 17:35:44 IDT 2016: Value VGW_SUBNET_IP

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com//issues/94#issuecomment-248061798, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AFni13wYKG6f9pm_AIqr8hxcGPEx11wvks5qrsbAgaJpZM4J_27s.

[nirhenn]

I have also looked in a Juniper Contrail install instance we have and it looks like it uses DPDK 2.1 which do have reference to mlnx4 driver. There is a procedure from Mellanox how to compile it (DODK) with mlnx but it is not clear what will be the Contrail proceedure to update the node with the new setup. Is there dependency with the DKMS ? What is the proceedure to upadate NIC type regardless the Mellanox support?
I can try to follow it and give it a try and contribute the output to the community.

[srajag]

You should be able to build the contrail-vrouter-dpdk binary (with any DPDK changes if required) and
replace the image in /usr/bin with the new one, then run “service supervisor-vrouter restart”.

Raja

From: Nir Henn notifications@github.com
Reply-To: Juniper/contrail-vrouter reply@reply.github.com
Date: Monday, September 19, 2016 at 12:39 PM
To: Juniper/contrail-vrouter contrail-vrouter@noreply.github.com
Cc: Raja Sivaramakrishnan raja@juniper.net, Comment comment@noreply.github.com
Subject: Re: [Juniper/contrail-vrouter] vRouer w/DPDK and Mellanox NIC (#94)

I have also looked in a Juniper Contrail install instance we have and it looks like it uses DPDK 2.1 which do have reference to mlnx4 driver. There is a procedure from Mellanox how to compile it (DODK) with mlnx but it is not clear what will be the Contrail proceedure to update the node with the new setup. Is there dependency with the DKMS ? What is the proceedure to upadate NIC type regardless the Mellanox support?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com//issues/94#issuecomment-248100171, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AFni13dDNBwlHYHGMW_M0D-tqPWCGPwzks5qruUGgaJpZM4J_27s.

[jjgrinwis]

Did anyone get this to work?
We're seeing the same issues and looks like the vRouter is not detecting our Mellanox cards:

EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: probe driver: 8086:1521 rte_igb_pmd
EAL: Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL: probe driver: 8086:1521 rte_igb_pmd
EAL: Not managed by a supported kernel driver, skipped
....

Looks like it’s only looking for Intel cards, as Mellanox is being ignored and not added.

root@node-110:# lspci -s 0000:02:00.0
02:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
root@node-110:# lspci -s 0000:01:00.0
01:00.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]

in an Intel only setup it's working fine:
EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL: probe driver: 8086:1521 rte_igb_pmd
EAL: Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:01:00.1 on NUMA socket 0
EAL: probe driver: 8086:1521 rte_igb_pmd
EAL: Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:0e:00.0 on NUMA socket 0
EAL: probe driver: 8086:10fb rte_ixgbe_pmd
EAL: PCI memory mapped at 0x7f5540000000
EAL: PCI memory mapped at 0x7f5540080000
PMD: eth_ixgbe_dev_init(): MAC: 2, PHY: 12, SFP+: 3
PMD: eth_ixgbe_dev_init(): port 1 vendorID=0x8086 deviceID=0x10fb
EAL: PCI device 0000:0e:00.1 on NUMA socket 0
EAL: probe driver: 8086:10fb rte_ixgbe_pmd
EAL: PCI memory mapped at 0x7f5540084000
EAL: PCI memory mapped at 0x7f5540104000
PMD: eth_ixgbe_dev_init(): MAC: 2, PHY: 12, SFP+: 4
PMD: eth_ixgbe_dev_init(): port 2 vendorID=0x8086 deviceID=0x10fb

[nirhenn]

I also gave it a try in the past and after speaking with Mellanox I learned it is not supported. They might have it in future releases of Contrail. Nir.

…

-- Nir Henn Innovation Coach, AT&T Foundry Desk +97297765298 Mobile +972547606867 AMDOCS | EMBRACE CHALLENGE EXPERIENCE SUCCESS This email was sent using 100% recycled electrons! STOP texting while driving. Take the pledge at itcanwait.com<http://www.itcanwait.com/> From: jjgrinwis <notifications@github.com> Reply-To: Juniper/contrail-vrouter <reply@reply.github.com> Date: Saturday, February 18, 2017 at 11:37 AM To: Juniper/contrail-vrouter <contrail-vrouter@noreply.github.com> Cc: Nir Henn <nir.henn@amdocs.com>, State change <state_change@noreply.github.com> Subject: Re: [Juniper/contrail-vrouter] vRouer w/DPDK and Mellanox NIC (#94) Did anyone get this to work? We're seeing the same issues and looks like the vRouter is not detecting our Mellanox cards: EAL: PCI device 0000:02:00.0 on NUMA socket -1 EAL: probe driver: 8086:1521 rte_igb_pmd EAL: Not managed by a supported kernel driver, skipped EAL: PCI device 0000:02:00.1 on NUMA socket -1 EAL: probe driver: 8086:1521 rte_igb_pmd EAL: Not managed by a supported kernel driver, skipped .... Looks like it’s only looking for Intel cards, as Mellanox is being ignored and not added. root@node-110:# lspci -s 0000:02:00.0 02:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) root@node-110:# lspci -s 0000:01:00.0 01:00.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4] in an Intel only setup it's working fine: EAL: PCI device 0000:01:00.0 on NUMA socket 0 EAL: probe driver: 8086:1521 rte_igb_pmd EAL: Not managed by a supported kernel driver, skipped EAL: PCI device 0000:01:00.1 on NUMA socket 0 EAL: probe driver: 8086:1521 rte_igb_pmd EAL: Not managed by a supported kernel driver, skipped EAL: PCI device 0000:0e:00.0 on NUMA socket 0 EAL: probe driver: 8086:10fb rte_ixgbe_pmd EAL: PCI memory mapped at 0x7f5540000000 EAL: PCI memory mapped at 0x7f5540080000 PMD: eth_ixgbe_dev_init(): MAC: 2, PHY: 12, SFP+: 3 PMD: eth_ixgbe_dev_init(): port 1 vendorID=0x8086 deviceID=0x10fb EAL: PCI device 0000:0e:00.1 on NUMA socket 0 EAL: probe driver: 8086:10fb rte_ixgbe_pmd EAL: PCI memory mapped at 0x7f5540084000 EAL: PCI memory mapped at 0x7f5540104000 PMD: eth_ixgbe_dev_init(): MAC: 2, PHY: 12, SFP+: 4 PMD: eth_ixgbe_dev_init(): port 2 vendorID=0x8086 deviceID=0x10fb — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub<#94 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AKB1nrABRshU3o8sSrBU1sTddycrb_Kbks5rdrvJgaJpZM4J_27s>. This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement, you may review at http://www.amdocs.com/email_disclaimer.asp

[srajag]

We have not tested this with Mellanox NICs yet. Raja From: jjgrinwis <notifications@github.com> Reply-To: Juniper/contrail-vrouter <reply@reply.github.com> Date: Saturday, February 18, 2017 at 1:37 AM To: Juniper/contrail-vrouter <contrail-vrouter@noreply.github.com> Cc: Rajagopalan Sivaramakrishnan <raja@juniper.net>, Comment <comment@noreply.github.com> Subject: Re: [Juniper/contrail-vrouter] vRouer w/DPDK and Mellanox NIC (#94) Did anyone get this to work? We're seeing the same issues and looks like the vRouter is not detecting our Mellanox cards: EAL: PCI device 0000:02:00.0 on NUMA socket -1 EAL: probe driver: 8086:1521 rte_igb_pmd EAL: Not managed by a supported kernel driver, skipped EAL: PCI device 0000:02:00.1 on NUMA socket -1 EAL: probe driver: 8086:1521 rte_igb_pmd EAL: Not managed by a supported kernel driver, skipped .... Looks like it’s only looking for Intel cards, as Mellanox is being ignored and not added. root@node-110:# lspci -s 0000:02:00.0 02:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) root@node-110:# lspci -s 0000:01:00.0 01:00.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4] in an Intel only setup it's working fine: EAL: PCI device 0000:01:00.0 on NUMA socket 0 EAL: probe driver: 8086:1521 rte_igb_pmd EAL: Not managed by a supported kernel driver, skipped EAL: PCI device 0000:01:00.1 on NUMA socket 0 EAL: probe driver: 8086:1521 rte_igb_pmd EAL: Not managed by a supported kernel driver, skipped EAL: PCI device 0000:0e:00.0 on NUMA socket 0 EAL: probe driver: 8086:10fb rte_ixgbe_pmd EAL: PCI memory mapped at 0x7f5540000000 EAL: PCI memory mapped at 0x7f5540080000 PMD: eth_ixgbe_dev_init(): MAC: 2, PHY: 12, SFP+: 3 PMD: eth_ixgbe_dev_init(): port 1 vendorID=0x8086 deviceID=0x10fb EAL: PCI device 0000:0e:00.1 on NUMA socket 0 EAL: probe driver: 8086:10fb rte_ixgbe_pmd EAL: PCI memory mapped at 0x7f5540084000 EAL: PCI memory mapped at 0x7f5540104000 PMD: eth_ixgbe_dev_init(): MAC: 2, PHY: 12, SFP+: 4 PMD: eth_ixgbe_dev_init(): port 2 vendorID=0x8086 deviceID=0x10fb — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#94 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AFni19MiSXn9aYLz59KiXEoiDCxp39fMks5rdrvFgaJpZM4J_27s>.